S3, Amazons cloud-based object storage service, is designed primarily for storing data that is used by applications running directly in the cloud. However, there are situations where you may want to access S3 buckets directly from your PC. You might want to do this to upload files from your PC to S3 without using the AWS Console, for example. Or, you may want to be able to monitor changes to S3 data from an application running on your local PC.
For purposes such as these, being able to access S3 data directly from your PC comes in handy. This article explains how to make S3 files available locally on both Windows and Linux using rclone, a free and open source tool for syncing cloud storage to local computers.
There are various other tools available for achieving this goal. I like rclone, however, both because its open source and because it works with any major operating system. Although there are some minor differences in the way you use the tool to access S3 data on Windows as compared to Linux, the basic process is the same regardless of which operating system youre running.
Following are the steps for using rclone to access S3 data from Linux or Windows.
Installing rclone is quite simple. You can download it for Windows or any version of Linux. On most Linux distributions, you also have the option of installing directly from your package manager using a command such as the following (for Ubuntu):
Or, you can download and run a Bash script to install rclone for you:
The latter approach may be desirable if you want a later version of rclone than the one offered in your distributions package repositories; otherwise, its better to use an official package from the repositories, because rclone will then be updated automatically for you whenever a new package version becomes available.
Rclone is a command-line tool, so youll need to open up a command shell or terminal to run it.
Once in the shell, you can run rclone directly with a simple rclone (or rclone.exe on Windows) command if the application is in your path--which it probably is if you installed it on Linux using a package.
If instead you just downloaded rclone as a ZIP file, you will have to unpack it, then use the cd command to navigate to the directory where the rclone files are located.
Once there, a simple ./rclone config (on Linux) or .rclone.exe config (on Windows) will start the program.
Rclone will then ask you a variety of configuration questions, including your AWS credentials for the S3 bucket you want to access. The configuration data will vary depending on how your S3 bucket is set up, but in general the default options should work.
After youve completed configuration, youre ready to use rclone to access S3 buckets.
Rclone offers about a dozen commands that you can use to interact with S3 data. For example, to list the contents of a bucket, use the ls command:
(If youre on Windows, replace rclone with rclone.exe.)
In this example, bucket-name is the name of your S3 bucket.
Likewise, to copy a file, use the copy command:
A full list of rclone commands is available on the rclone website. Keep in mind, however, that not all of them will work with S3 data. For example, you cant use the mkdir command (which would create a new directory) with an S3 bucket because S3 doesnt support directories.
Rclones built-in commands for interacting with data are handy if you just need to copy or access some files manually. But what if you want to automate interaction with your S3 data, or access it using commands that are not supported by rclone?
In that case, you can use the rclone mount command to mount your S3 bucket as a directory. That way, you can interact with your S3 data just as you would any other data stored locally on your computer.
To mount an S3 bucket with rclone on Linux, use a command like:
Note that you may need to run this command as root. Youll also need to make sure the mount point (/mnt/some-dir in the example above) exists before you run the command. (If it doesnt, use mkdir on Linux to create it.)
The process is similar on Windows, with one major difference: You first need to install WinFsp (find the installer here) before you can mount an S3 bucket. Once WinFsp has been installed, you can mount your S3 bucket as a directory with:
In this case, your mount point (C:somedir in the example) should be a directory that does not yet exist.
Whether you use Windows or Linux, rclone offers a free and straightforward way to access S3 buckets from your local computer. However, there are some caveats to keep in mind.
One is that Amazon charges you a fee every time you create or modify a file in an S3 bucket. This means that, if you perform a large number of file operations via rclone on S3 data, you may end up with a substantial cloud bill.
A second consideration to weigh is that the performance of your S3 data when you access it from your PC may be limited due to network latency. Even if you mount your bucket as a local directory, expect a delay when you interact with the data.
For both of these reasons, you may end up shooting yourself in the foot if you try to use S3 buckets as a cheap way to back up all of the data from your PC, or as a personal file-sharing service. In other words, dont try to use the method described above to turn S3 into something like Dropbox or Google Drive, which are better suited to situations where you need fast, cost-efficient integration between your local file system and cloud storage. Even though accessing S3 data from a local computer is relatively easy, the performance and cost implications make it impractical to do this on a large-scale or recurring basis.
Still, if you need a fast and simple way to access S3 data from your computer in order to copy files or use a certain application on a one-off basis, rclone makes it easy to do so on Windows, Linux and virtually every other operating system you can find.
View post:
How to Access S3 Buckets from Windows or Linux - ITPro Today
- CTERA Networks Partners with SYNNEX Corporation to Drive Market Demand for Hybrid Cloud Storage, Collaboration and ... [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Cloud storage exempt from Ninefold's uptime boost [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Virsto Named Finalist of 2012 Storage Virtualization & Cloud Awards [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Innovative Hybrid Cloud Storage Solutions Now Available From PROMISE Technology [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Box Talks Integration with BlackBerry 10 and Cloud Storage for Business - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- AG112's Weekly Technology Tutorials Ep.7 Cloud Storage - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Cloud Storage - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Google Cloud Storage Office Hours - 9/5/2012 - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- IBM Cloud Storage -- Future Directions - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Working with best FREE Cloud storage solution - MediaFire - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Best Cloud Storage | How Nate Made $450 His First Hour... - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Cloud Storage Services: Comparison - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Top 10 Free Cloud Storage Services of 2012 - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Cloud Storage Wars - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Secure and Comprehensive Cloud Storage for Health IT - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Free Cloud Storage! - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Microsoft SkyDrive Cloud Storage - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Top 16 Android Cloud Storage Apps Quick Breakdown - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Up to 48GB of FREE Cloud Storage, 14GB Guaranteed - Video [Last Updated On: October 5th, 2012] [Originally Added On: October 5th, 2012]
- Nasuni's CEO To Speak At Interop On The Secure Use Of Cloud Storage [Last Updated On: October 6th, 2012] [Originally Added On: October 6th, 2012]
- Oracle vs Amazon Cloud Storage: OpenWorld 2012 - Video [Last Updated On: October 6th, 2012] [Originally Added On: October 6th, 2012]
- Apple extends iCloud storage for another year [Last Updated On: October 7th, 2012] [Originally Added On: October 7th, 2012]
- Interush Introduces Convenient Cloud-Based Storage Service with Release of PHYTTER DOCK Application [Last Updated On: October 9th, 2012] [Originally Added On: October 9th, 2012]
- Get a free 15GB cloud-storage account from 4Sync [Last Updated On: October 9th, 2012] [Originally Added On: October 9th, 2012]
- Cloud Solutions Increase Customer Engagement and Retention [Last Updated On: October 9th, 2012] [Originally Added On: October 9th, 2012]
- Pogoplug offering 100GB of cloud storage to UK users for just £19.99 a year [Last Updated On: October 10th, 2012] [Originally Added On: October 10th, 2012]
- New vFoglight Storage 2.0 Provides Integrated Application to Disk Performance Monitoring [Last Updated On: October 10th, 2012] [Originally Added On: October 10th, 2012]
- Lunacloud Deploys Cloudian® To Grow Business, Offer S3 Compatible Cloud Storage [Last Updated On: October 11th, 2012] [Originally Added On: October 11th, 2012]
- New Cloud Storage Company, ZapDrive, Launches Today Offering 100 GB for $19.99/year. [Last Updated On: October 11th, 2012] [Originally Added On: October 11th, 2012]
- Otixo Adds Ubuntu One to Aggregated Cloud Storage Lineup [Last Updated On: October 11th, 2012] [Originally Added On: October 11th, 2012]
- Cloud Storage Reviews Announcement Video - Video [Last Updated On: October 11th, 2012] [Originally Added On: October 11th, 2012]
- Cloud storage outage strikes Macquarie Telecom [Last Updated On: October 11th, 2012] [Originally Added On: October 11th, 2012]
- Online-Storage.com is Now SIO.CO [Last Updated On: October 11th, 2012] [Originally Added On: October 11th, 2012]
- C2C Maximizes eMail Archiving Flexibility and Control With Support for the Hybrid Cloud [Last Updated On: October 11th, 2012] [Originally Added On: October 11th, 2012]
- OwnCloud: Build your own or manage your public cloud storage services [Last Updated On: October 12th, 2012] [Originally Added On: October 12th, 2012]
- Ubuntu's cloud storage service hits Mac in beta, with 5GB free [Last Updated On: October 12th, 2012] [Originally Added On: October 12th, 2012]
- Akitio Cloud Hybrid Review: Convenient NAS and USB Storage in One [Last Updated On: October 13th, 2012] [Originally Added On: October 13th, 2012]
- Symform Hires Senior Sales Executive to Build Global Partnerships as Distributed Cloud Storage Network Surpasses 5.5 ... [Last Updated On: October 15th, 2012] [Originally Added On: October 15th, 2012]
- Get an extra 25GB of storage in the Dropbox Great Space Race [Last Updated On: October 16th, 2012] [Originally Added On: October 16th, 2012]
- Microsoft Acquires StorSimple To Increase Cloud Storage Capabilities [Last Updated On: October 17th, 2012] [Originally Added On: October 17th, 2012]
- Inktank-Metacloud Partnership Enhances Fully Managed Private Cloud Solution With Enterprise-Class Storage [Last Updated On: October 17th, 2012] [Originally Added On: October 17th, 2012]
- Citrix and NetApp Collaborate to Simplify Cloud Storage [Last Updated On: October 17th, 2012] [Originally Added On: October 17th, 2012]
- Microsoft Acquires Leader In Cloud-integrated Storage [Last Updated On: October 17th, 2012] [Originally Added On: October 17th, 2012]
- Microsoft Buys StorSimple for Enterprise Cloud Storage [Last Updated On: October 18th, 2012] [Originally Added On: October 18th, 2012]
- FreedomPACS, Radiology PACS and Cloud Image Storage Provider, Releases Results of County Hospital Case Study ... [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- Nirvanix Selects Brocade as Networking Backbone for Global Cloud Expansion [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- Pogoplug offers unlimited cloud storage for $5 a month [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- NTT Communications Chooses Cloudian® S3 compatible Object Storage Platform for Multi Petabyte Cloud Storage as a Service [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- TwinStrata and Google to Host "Beyond Disaster Recovery: Integrating Cloud Storage into Your IT Strategy" Seminar [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- Cloud Storage Reviews Outlines "How SugarSync Works" In Latest Guide [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- Symform Challenges Users to Think Beyond Centralized Data Centers With Its 'Byte Me' Promotion [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- Avere to tart up FTX with cloud storage gateway, mutterings foretell [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- Deals WD My Book Live Personal Cloud Storage 2 TB Network Attached Best Price 2012 - Video [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- Create and Manage Your Own Cloud Storage Free - Video [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- Free Cloud Space 100GB - Video [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- DuraCloud Brown Bag Series: How DuraCloud is Different From Amazon - Video [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- PocketCloud Explore - Video [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- Free 1TB Cloud storage - Video [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- Store your files on WEB for free - Unlimited and better than dropbox - Video [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- CloudBackupNow - Retention Policy (with audio) - Video [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- CloudBackupNow - Retention Policy - Video [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- CloudBackupNow - Primer II - Video [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- ERP Data Capture animation - Video [Last Updated On: November 1st, 2012] [Originally Added On: November 1st, 2012]
- Cash rains DOWN on the Cloud - Nasuni trousers $20m [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- My PC Backup Review The Cloud Storage Service For You - Video [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- Samsung ATIV S Review - Phones 4u - Video [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- Trust Me mv - Video [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- Product Webinar: Collaborating and Exchanging Large Data at Distance with Faspex 3.0 - Video [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- DT Daily: Facebook takes aim at Craigslist, Halo 4 reviews a - Video [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- 2 MCSE Private Cloud Storage Basics - Video [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- Gladinet Cloud Enterprise Quick Start Guide - Video [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- Installing OfficeDrop Mac File Sync - Video [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- OfficeDrop Mac File Sync - Video [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- Secure Cloud Storage - Video [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- Windows Phone 8: Lenese integrates apps in the camera app - Video [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- Graphic Video on Wuala Secure Cloud Storage from Paula Hansen and Chart Magic - Video [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- Hurricane Sandy Cheat Meal Run to Tastee Diner - Video [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- Cloud Zow Review - Cloudzow Review | Marketing Secret Revealed - Video [Last Updated On: November 3rd, 2012] [Originally Added On: November 3rd, 2012]
- What is Cloud Storage? - Video [Last Updated On: November 4th, 2012] [Originally Added On: November 4th, 2012]
- Perfume - Chocolate Disco [ hide@BSB Battle In Feb. Remix ] - Video [Last Updated On: November 4th, 2012] [Originally Added On: November 4th, 2012]