Mount s3 bucket to linux

How to Mount S3 Bucket on CentOS/RHEL and Ubuntu using S3FS

S3FS is FUSE (File System in User Space) based solution to mount an Amazon S3 buckets, We can use system commands with this drive just like as another Hard Disk in the system. On s3fs mounted files systems we can simply use cp, mv and ls the basic Unix commands similar to run on locally attached disks.

If you like to access S3 buckets without mounting on system, use s3cmd command line utility to manage s3 buckets. s3cmd is also provides faster speed for data upload and download rather than s3fs. To work with s3cmd use next articles to install s3cmd in Linux systems and Windows systems.

This article will help you to install S3FS and Fuse by compiling from source, and also help you to mount S3 bucket on your CentOS/RHEL and Ubuntu systems.

Step 1 – Remove Existing Packages

First, check if you have any existing s3fs or fuse package installed on your system. If installed it already remove it to avoid any file conflicts.

Step 2: Install Required Packages

After removing packages. First, we will install all the dependencies for fuse and s3cmd. Install the required packages to system use following command.

Step 3 – Download and Compile Fuse

Download and compile latest version of fuse source code. For this article, we are using fuse version 3.5. Following the set of command will compile fuse and add fuse module in the kernel.

Step 4 – Download and Compile Latest S3FS

Download and compile latest version of s3fs source code. For this article we are using s3fs version 1.74. After downloading extract the archive and compile source code in system.

Step 5 – Setup Access Key

Also In order to configure s3fs, we would require Access Key and Secret Key of your S3 Amazon account. Get these security keys from Here.

Note: Change AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with your actual key values.

Step 6 – Mount S3 Bucket

Finally mount your s3 bucket using following set of commands. For this example, we are using s3 bucket name as mydbbackup and mount point as /s3mnt.

Related Posts

How to Remove CloudFront Cache

How To Delete Application Versions from AWS Beanstalk

How to Install and Use AWS CLI Tools on Linux

39 Comments

cd fuse-3.1.0 Not exist ( just fuse-3.5.0)
[[email protected] fuse-3.5.0]# ./configure –prefix=/usr/local
-bash: ./configure: No such file or directory

any idea how to mount local S3 (not on AWS) created on PURE storage. I got the bucket name, Bucket Endpoint, Bucket Access Key and bucket Secret Access Key .

They use meson/ninja now, not make && make install nor ./configure

s3fs: HTTP: 403 Forbidden – it is likely that your credentials are invalid

are you have solved ??

Can be compatible with the cheap wasabi.com instead of amazon s3?

Do you have a solution for this problem? Run into the same problem.
regards Ronald

On Ubuntu steps 1-4 can be replaced with one command:
sudo apt-get install s3fs
Do this instead of steps 1-4 and continue from step 5. It works – tested.

Do NOT use fuse3* but stick to a fuse 2*, otherwise you will not be able to install s3fs.

I am getting below error.

fuse: warning: library too old, some operations may not not work

# rpm -qa | grep fuse
libconfuse-2.7-4.el6.x86_64
fuse-libs-2.8.3-5.el6.x86_64
#

though latest package is not available in Yum repo. please suggest

Hello, i followed your guide (on Ubuntu 14.04, Bitnami – EC2 ) , i am getting following error at “make”
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/libcurl.so: undefined reference to `[email protected]_2.4_2′
collect2: error: ld returned 1 exit status
make[2]: *** [s3fs] Error 1
make[2]: Leaving directory `/tmp/s3fs-fuse-1.80/src’
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tmp/s3fs-fuse-1.80′
make: *** [all] Error 2

use this if you get “bash: s3fs: command not found”
sudo /usr/local/bin/s3fs -o use_cache=/tmp/cache mydbbackup /s3mnt

Those of you having an error on CentOS: s3fs: error while loading shared libraries: libfuse.so.2: cannot open shared object file: No such file or directory

yum install fuse-libs

Great article! Worked a treat….

only issue was my library path required updating as when I ran s3fs it couldn’t find the fuse dependancies.

LD_LIBRARY_PATH=/usr/local/lib
export LD_LIBRARY_PATH

That did the trick 🙂

Hi..
i have mounted s3 sucessfully , but when i try to “cd ” to mounted dirs it says “operation not permitted”

cd: app_logs/: Operation not permitted

below is permission for dir

d———. 1 root root 1 May 6 2015 app_logs

below command used
s3fs -o use_cache=/tmp/cache s3bukcket /s3mnt

what permission i need to set for bucket or bucket folder ?
am i writing wrong command for mount?

Can you please share the steps to mount Amazon S3 bucket on windows OS for both 2008 & 2012?

Thanks & Regards,
Mehul

Thanks! This was quite helpful.
You can take a look at the docker image which I built, with S3FS and S3 bucket mounting capabilities:
https://registry.hub.docker.com/u/ihealthtechnologies/s3-mount/

Hi all, I’m receiving the following error after trying to mount the bucker:

# s3fs -o user_cache=/tmp/cache lehar-backup /s3mnt
s3fs: /lib/libfuse.so.2: version `FUSE_2.8′ not found (required by s3fs)

I downloaded fuse 2.8 and compiled it per the instructions however if there is something I’m missing I’d love to know what that is. Let me know what information you might need. Thanks for any help you can provide.

My disk is full after mount and not able to use the system. Ideally it should not take disk space.
How can i resolve the issue?

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda1 10321208 10297608 0 100% /
none 847476 0 847476 0% /dev/shm
s3fs 274877906944 0 274877906944 0% /s3mnt

s3fs: HTTP: 403 Forbidden – it is likely that your credentials are invalid

are you have solved ??

I am getting error:
s3fs: unable to access MOUNTPOINT storingfiles1: No such file or directory

Hi Team,
Very good it working fine in my ubuntu system

So far so good but I get this error:

[[email protected] s3fs-1.74]# s3fs -o use_cache=/tmp/cache agarta /etc/httpd/imagestore/
bash: s3fs: command not found

any idea on how to resolve this?

Thanks for the tutorial. All worked fine, except this messsage – Transport endpoint is not connected. Unmounted and mounted again using the link you shared in comments. I am using s3fs to connect to Google Cloud Storage.

Kindly share some pointers or might be issue with the permission level mentioned above.

/usr/local/lib
include ld.so.conf.d/*.conf

# ldconfig
# s3fs -o use_cache=/tmp/cache ****.******.*** /s3mnt

s3fs: error while loading shared libraries: libfuse.so.2: cannot open shared object file: No such file or directory

I have a problem, can you helpme?

[[email protected]*********** s3fs-1.74]# s3fs -o use_cache=/tmp/cache ****.******.*** /s3mnt
s3fs: error while loading shared libraries: libfuse.so.2: cannot open shared object file: No such file or directory

install fuse-libs and it should solve your issue… it did in my case…

sudo yum install fuse-libs

hope that helps

yes, you could add this in the tutorial ! it solved it in my case too. Thanks

Источник

How to Mount S3 bucket on EC2 Linux Instance

A S3 bucket can be mounted in a AWS instance as a file system known as S3fs. S3fs is a FUSE file-system that allows you to mount an Amazon S3 bucket as a local file-system. It behaves like a network attached drive, as it does not store anything on the Amazon EC2, but user can access the data on S3 from EC2 instance.

Filesystem in Userspace (FUSE) is a simple interface for userspace programs to export a virtual file-system to the Linux kernel. It also aims to provide a secure method for non privileged users to create and mount their own file-system implementations.

S3fs-fuse project is written in python backed by Amazons Simple Storage Service. Amazon offers an open API to build applications on top of this service, which several companies have done, using a variety of interfaces (web, rsync, fuse, etc).

Follow the below steps to mount your S3 bucket to your Linux Instance.

This Tutorial assumes that you have a running Linux EC2 instance on AWS with root access and a bucket created in S3 which is to be mounted on your Linux Instance. You will also require Access and Secret key pair with sufficient permissions of S3 or else an IAM access to generate or Create it.

We will perform the steps as a root user. You can also use sudo command if you are a normal user with sudo access. So lets get started.

Step-1:- If you are using a new centos or ubuntu instance. Update the system.

-> For CentOS or Red Hat

Step-2:- Install the dependencies.

-> In CentOS or Red Hat

In Ubuntu or Debian

Step-3:- Clone s3fs source code from git.

Step-4:- Now change to source code directory, and compile and install the code with the following commands:

Step-5:- Use below command to check where s3fs command is placed in O.S. It will also tell you the installation is ok.

Step-6:- Getting the access key and secret key.

You will need AWS Access key and Secret key with appropriate permissions in order to access your s3 bucket from your EC2 instance. You can easily manage your user permissions from IAM (Identity and Access Management) Service provided by AWS. Create an IAM user with S3 full access(or with a role with sufficient permissions) or use root credentials of your Account. Here we will use the root credentials for simplicity.

Go to AWS Menu -> Your AWS Account Name -> My Security Credentials. Here your IAM console will appear. You have to go to Users > Your Account name and under permissions Tab, check whether you have sufficient access on S3 bucket. If not, you can manually assign an existing “S3 Full-Access” policy or create a new policy with sufficient permissions.

Now go to Security Credentials Tab and Create Access Key. A new Access Key and Secret Key pair will be generated. Here you can see access key and secret key (secret key is visible when you click on show tab) which you can also download. Copy these both keys separately.

Note that you can always use an existing access and secret key pair. Alternatively, you can also create a new IAM user and assign it sufficient permissions to generate the access and secret key.

Источник