FUSE on Amazon S3
FUSE on Amazon S3
Join the DZone community and get the full member experience.Join For Free
Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.
FUSE: File System In User Space, hosted on sourceforge, a well known open source project: http://fuse.sourceforge.net/
You either put the files in S3 bucket directly or in the mount point, both will always be in the same hierarchy and in Sync. The best thing is that any arbitrary program can just point to this mount point and perform simple/ normal commands, rather than file system specific commands.
Here is a small documentation about how we can achieve this.
1. Check out the code from google code.
$ svn checkout http://s3fs.googlecode.com/svn/trunk/ s3fs2. Switch to the working directory
$ cd s3fs
AUTHORS autogen.sh ChangeLog configure.ac COPYING doc INSTALL Makefile.am NEWS README src test3. Now same old ritual of configure , make and install.
To run the subsequent command you need autoconf. So make sure you have it by running the following command.
$ sudo apt-get install autoconf$ autoreconf --install
It is silently notifying you that you lack the libraries. Time to get them installed...
$ sudo apt-get install build-essential libfuse-dev fuse-utils libcurl4-openssl-dev libxml2-dev mime-supportGetting back...
$ ./configure --prefix=/usr
$ sudo make install4. Done with the Installation process.
s3fs: missing BUCKET argumentUsage: s3fs BUCKET:[PATH] MOUNTPOINT [OPTION]...5. Add the following line to your ~/.bashrc file and source it.
$source ~/.bashrc$ s3fs s3fs: missing BUCKET argumentUsage: s3fs BUCKET:[PATH] MOUNTPOINT [OPTION]...6. Install s3cmd. Many of you must be using this tool to interact with s3.
$ sudo apt-get install s3cmd$ s3cmd --configureThis will configure with the S3 account using Access and Secret Key.
1. First set use_allow_other for others to use. Uncomment in fuse.conf
$ vi /etc/fuse.conf2. Set the AcessKey:SecretKey in the format in passwd-s3fs file
$ sudo vi /etc/passwd-s3fs
$ sudo chmod 640 /etc/passwd-s3fs3. Created a bucket called "s3dir-sync" for this experiment.
$ s3cmd ls2012-12-18 09:23 s3://s3dir-sync4. Creating a mount point where you want to dump/place the files and keep them in sync with the S3 bucket. Create as root user.
$ sudo mkdir -p /mnt/s3Sync$ sudo chmod 777 /mnt/s3Sync5. With s3fs, as a root user.
$ sudo s3fs s3dir-sync -o default_acl=public-read -o allow_other /mnt/s3Sync/Cross-check:
$ mount -ls3fs on /mnt/s3Sync type fuse.s3fs (rw,nosuid,nodev,allow_other)If you try mounting again, you will get the following Warning
mount: according to mtab, s3fs is already mounted on /mnt/s3Sync6. I created a directory structure of
-> 2012/12/18$ more test.txt
This is a check file to sync with the s3dir-sync.
The same is synced in the bucket "s3dir-sync"
$ s3cmd ls s3://s3dir-sync
2012-12-18 09:57 0 s3://s3dir-sync/2012Happy Learning!
Published at DZone with permission of Swathi Venkatachala , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.