S3, Amazons cloud-based object storage service, is designed primarily for storing data that is used by applications running directly in the cloud. However, there are situations where you may want to access S3 buckets directly from your PC. You might want to do this to upload files from your PC to S3 without using the AWS Console, for example. Or, you may want to be able to monitor changes to S3 data from an application running on your local PC.
For purposes such as these, being able to access S3 data directly from your PC comes in handy. This article explains how to make S3 files available locally on both Windows and Linux using rclone, a free and open source tool for syncing cloud storage to local computers.
There are various other tools available for achieving this goal. I like rclone, however, both because its open source and because it works with any major operating system. Although there are some minor differences in the way you use the tool to access S3 data on Windows as compared to Linux, the basic process is the same regardless of which operating system youre running.
Following are the steps for using rclone to access S3 data from Linux or Windows.
Installing rclone is quite simple. You can download it for Windows or any version of Linux. On most Linux distributions, you also have the option of installing directly from your package manager using a command such as the following (for Ubuntu):
Or, you can download and run a Bash script to install rclone for you:
The latter approach may be desirable if you want a later version of rclone than the one offered in your distributions package repositories; otherwise, its better to use an official package from the repositories, because rclone will then be updated automatically for you whenever a new package version becomes available.
Rclone is a command-line tool, so youll need to open up a command shell or terminal to run it.
Once in the shell, you can run rclone directly with a simple rclone (or rclone.exe on Windows) command if the application is in your path--which it probably is if you installed it on Linux using a package.
If instead you just downloaded rclone as a ZIP file, you will have to unpack it, then use the cd command to navigate to the directory where the rclone files are located.
Once there, a simple ./rclone config (on Linux) or .rclone.exe config (on Windows) will start the program.
Rclone will then ask you a variety of configuration questions, including your AWS credentials for the S3 bucket you want to access. The configuration data will vary depending on how your S3 bucket is set up, but in general the default options should work.
After youve completed configuration, youre ready to use rclone to access S3 buckets.
Rclone offers about a dozen commands that you can use to interact with S3 data. For example, to list the contents of a bucket, use the ls command:
(If youre on Windows, replace rclone with rclone.exe.)
In this example, bucket-name is the name of your S3 bucket.
Likewise, to copy a file, use the copy command:
A full list of rclone commands is available on the rclone website. Keep in mind, however, that not all of them will work with S3 data. For example, you cant use the mkdir command (which would create a new directory) with an S3 bucket because S3 doesnt support directories.
Rclones built-in commands for interacting with data are handy if you just need to copy or access some files manually. But what if you want to automate interaction with your S3 data, or access it using commands that are not supported by rclone?
In that case, you can use the rclone mount command to mount your S3 bucket as a directory. That way, you can interact with your S3 data just as you would any other data stored locally on your computer.
To mount an S3 bucket with rclone on Linux, use a command like:
Note that you may need to run this command as root. Youll also need to make sure the mount point (/mnt/some-dir in the example above) exists before you run the command. (If it doesnt, use mkdir on Linux to create it.)
The process is similar on Windows, with one major difference: You first need to install WinFsp (find the installer here) before you can mount an S3 bucket. Once WinFsp has been installed, you can mount your S3 bucket as a directory with:
In this case, your mount point (C:somedir in the example) should be a directory that does not yet exist.
Whether you use Windows or Linux, rclone offers a free and straightforward way to access S3 buckets from your local computer. However, there are some caveats to keep in mind.
One is that Amazon charges you a fee every time you create or modify a file in an S3 bucket. This means that, if you perform a large number of file operations via rclone on S3 data, you may end up with a substantial cloud bill.
A second consideration to weigh is that the performance of your S3 data when you access it from your PC may be limited due to network latency. Even if you mount your bucket as a local directory, expect a delay when you interact with the data.
For both of these reasons, you may end up shooting yourself in the foot if you try to use S3 buckets as a cheap way to back up all of the data from your PC, or as a personal file-sharing service. In other words, dont try to use the method described above to turn S3 into something like Dropbox or Google Drive, which are better suited to situations where you need fast, cost-efficient integration between your local file system and cloud storage. Even though accessing S3 data from a local computer is relatively easy, the performance and cost implications make it impractical to do this on a large-scale or recurring basis.
Still, if you need a fast and simple way to access S3 data from your computer in order to copy files or use a certain application on a one-off basis, rclone makes it easy to do so on Windows, Linux and virtually every other operating system you can find.
- Google: We're capping your free cloud storage at 15GB starting next year - TechRepublic - November 16th, 2020
- Google Photos a reminder that cloud storage isn't infinite - TechHQ - November 16th, 2020
- What is the cheapest way to save all those photos in your phone? - Mint - November 16th, 2020
- What is OneDrive? Everything you need to know about Microsoft's cloud storage service - Business Insider - Business Insider - November 13th, 2020
- Samsung Cloud storage shutdown: Everything you need to know - Tom's Guide - November 13th, 2020
- InfluxData adds yet another 2.0 platform and opens a new front to cloud storage - ZDNet - November 13th, 2020
- This week in storage with Kioxia and Elastic Blocks and Files - Blocks and Files - November 13th, 2020
- Global Business Cloud Storage Market Report With in Depth Analysis by Top Key Players- Zoolz OpenDrive JustCloud MozyPro Egnyte CrashPlan Dropbox... - November 13th, 2020
- Global Business Cloud Storage Market 2020 Industry Analysis, Size, Share, Strategies and Forecast to 2026 Wall Street Call - Reported Times - November 13th, 2020
- Cloud Storage Gateway Market Key Players, Product and Production Information analysis and forecast to 2026 - Zenit News - November 13th, 2020
- Cloud Storage Market Consumption Analysis, Business Overview and Upcoming Trends Forecast by 2026 - The Daily Philadelphian - November 13th, 2020
- IMT partners with Dalet to simplify AI and media storage in the cloud - KMWorld Magazine - October 29th, 2020
- NetApp CEO George Kurian: The Digital Economy 'Rewrites The Rules' - CRN: Technology news for channel partners and solution providers - October 29th, 2020
- Catalogic Software Expands its Smart Data Suite to Cloud Storage and Cloud Native Applications - PR Web - October 29th, 2020
- Global Cloud Storage Market Report Forecast to 2025 by Global Market Insights, Key Companies and Driving Trends| Zoolz, OpenDrive, JustCloud, MozyPro,... - October 29th, 2020
- Sycomp launches an unmatched cloud data experience with Sycomp Storage Fueled by IBM Spectrum Scale on Azure - AiThority - October 29th, 2020
- Global Cloud Storage Providers Market Expected To Reach Highest CAGR By 2026: Pertino, Asigra, SoftLayer, StorageCraft, Dropbox etc. - The Think... - October 27th, 2020
- NetApp plays Spot the difference in cloud services build-out Blocks and Files - Blocks and Files - October 27th, 2020
- Private Cloud Storage Market Analysis, Growth Forecast Analysis by Manufacturers, Regions, Type and Application to 2026 - TechnoWeekly - October 27th, 2020
- Global Cloud Storage Software Market Growth, Demand And Threats Analysis 2020 By Regional Overview Of Leading Players, Types, Application and Forecast... - October 27th, 2020
- Tag: Cloud Storage - The Think Curiouser - October 27th, 2020
- Cloud Object Storage Market Analysis 2020-2027: by Key Manufacturers with Countries, Types, Application and Forecast Till 2027 - The Think Curiouser - October 27th, 2020
- Cloud Storage Software Market (2020 to 2025) | Growing Application in Industry, Presents Opportunities and Demand Analysis, Growth, Size and Share -... - October 27th, 2020
- Cloud Storage Market: Qualitative Analysis of the Leading Players and Competitive Industry Scenario, 2023 - The Think Curiouser - October 27th, 2020
- Backblaze B2 Cloud Storage Integrated With Facebook as Photo and Video Transfer Destination - PRNewswire - October 24th, 2020
- Enterprise Data Cloud Storage Software Market is projected to grow at High CAGR during the forecast period, 2026 | MailChimp, Constant Contact,... - October 24th, 2020
- Cloud Storage Software Industry Market 2020: Industry Growth, Competitive Analysis, Future Prospects and Forecast 2026 - PRnews Leader - October 24th, 2020
- 5 top tips for businesses in data backup and recovery - Tom's Guide - October 24th, 2020
- Unified file and object storage: The best of both worlds? - ComputerWeekly.com - October 24th, 2020
- Business Cloud Storage market will continue to boom says analyst - Eurowire - October 24th, 2020
- Veeam Enhances Offerings To Be A 'Perfect' Fit For MSPs - CRN: Technology news for channel partners and solution providers - October 24th, 2020
- Cloud Storage Gateway Market Globally Expected to Drive Growth through 2026 - KYT24 - October 24th, 2020
- BTFS is Poised to Disrupt the Cloud Storage Industry? - ihodl.com - October 15th, 2020
- Reducing cloud storage costs: what you need to know - ITProPortal - October 15th, 2020
- Save an extra 20% on VPNs, password managers, and cloud storage - Mashable - October 15th, 2020
- Cohesity hooks up with AWS to pipe data management-as-a-service at users, starting with backup - Blocks and Files - October 15th, 2020
- The Best Backup and Restore Courses and Online Training for 2020 - Solutions Review - October 15th, 2020
- NetApp Insight 2020 conference news and coverage - TechTarget - October 15th, 2020
- iPad productivity tips: Keyboard tricks, shortcuts, and more - Fast Company - October 15th, 2020
- Cloud At The Edge, GPU Storage And LTO Gen 9 - Forbes - September 25th, 2020
- Zoom is being sued over its cloud storage practices - TechRadar - September 25th, 2020
- Microsofts storage dream: a hard disk drive the size of a wardrobe with Samsung Galaxy S20 parts - TechRadar - September 25th, 2020
- There is a hole in my cloud bucket - Fudzilla - September 25th, 2020
- Red Hat shifts automated data pipeline into OpenShift Blocks and Files - Blocks and Files - September 25th, 2020
- Seagate gets into object storage with new CORTX software Blocks and Files - Blocks and Files - September 25th, 2020
- Kioxia's Ethernet SSD stirs into EBOF life as architects dream - Blocks and Files - September 25th, 2020
- Ceph scales to 10 billion objects Blocks and Files - Blocks and Files - September 25th, 2020
- Google accused of using its online dominance to hold back competitors; to face antitrust lawsuit: Report - Times Now - September 25th, 2020
- Synology DiskStation can keep your digital life organized and safe - The Dallas Morning News - September 25th, 2020
- Looking for a job? The Library is here to help! | Globe Times - swglobetimes.com - September 25th, 2020
- Multi Cloud Storage Market is Projected to Increment at an Eye-Catching CAGR by 2023 | 21.2% CAGR | Know the COVID19 Impact - Verdant News - September 25th, 2020
- Internet of Things (IoT) Cloud Platform Market: Demand Rate with Regional Outlook, Applications, Consumer Profiles & Forecast 2026 - The Daily... - September 25th, 2020
- This tiny CPU firm could play a key role in the future of Apple One - TechRadar - September 25th, 2020
- Could Snowflake Rival Amazon in Cloud Storage and Services? Here's What You Need to Know About the New So - Tech Times - September 15th, 2020
- How Cloud Computing Can Deal With Lightning Strikes and Hackers - Carnegie Endowment for International Peace - September 15th, 2020
- How to approach IT logging in the cloud vs. on premises - TechTarget - September 15th, 2020
- This lifetime web hosting subscription comes with up to 1TB of storage - Mashable - September 15th, 2020
- Keep It in the Cloud! Best Cloud Storage Systems of 2020 - iDrop News - September 6th, 2020
- Impact of COVID-19 on Cloud Storage Software Market 2025 Expected to reach Highest CAGR including major key players Amazon Web Services, Microsoft,... - September 6th, 2020
- Facebook adds cloud storage providers Dropbox and Koofr to its photo and video portability tool - Digital Information World - September 6th, 2020
- Cloud storages you need to know - The Star, Kenya - September 6th, 2020
- How COVID-19 is Impacting the Consumer Cloud Storage Services Market by Industry Analysis, by Type, Application and Top Players:Apple, Google, Box,... - September 6th, 2020
- Cloud Storage Gateway Market to Witness Stunning Growth by 2027; Key Players are Riverbed Technology, SoftNAS, Inc., Oracle, Microsoft, Nasuni... - September 6th, 2020
- COVID-19 Is Driving a Cloud Computing Surge That Will Only Continue | Opinion - Newsweek - September 6th, 2020
- Asia Pacific Personal Cloud Market Industry Analysis and Market Forecast (2019-2026) _ Hosted Types, Revenues, User Type, and Geography. - Galus... - September 6th, 2020
- Amazon's Blink Unveils New Wireless Security Cameras with HD Video, Flexible Storage Options, and New Battery Expansion Pack Cameras Start at $79.99... - September 2nd, 2020
- Cloud Storage Software Market Will Raise Beyond Imagination over Period 2025 | Microsoft, Oracle, Rackspace Hosting, Red Hat, IBM - Scientect - September 2nd, 2020
- Stand Alone Cloud Storage Market Current Industry Size and Future Prospective with Key Players, Drivers and Trends - The Daily Chronicle - September 2nd, 2020
- Media And Entertainment Storage TAM To Exceed $16B By 2025 - Forbes - September 2nd, 2020
- The Launching Ceremony for XnMatrix Wrapped Up, the Next Generation of Cloud Computing Eco-System Sets Sail - PRNewswire - September 2nd, 2020
- Why not open our own Container Registry, muses GitHub as it gives orgs a hand at resource-sharing DEVCLASS - DevClass - September 2nd, 2020
- Sharing responsibility: Why we need to work together to keep the cloud secure - ComputerWeekly.com - September 2nd, 2020
- Data breach exposes tens of thousands of NSW drivers licences online - ABC News - September 2nd, 2020
- 10 Key Takeaways From NetApp CEO George Kurian: Cloud, Coronavirus And Growth - CRN: Technology news for channel partners and solution providers - September 2nd, 2020
- Responding to Cloud Misconfigurations with Security Automation and Common-Sense Tips - Security Boulevard - September 2nd, 2020
- How to Prepare for the Next Time the Cloud Goes Down - Gizmodo - September 2nd, 2020
- Demand for Consumer Cloud Storage Services Market from Major End-use Sectors to Increase in the Near Future - The Scarlet - August 29th, 2020
- Prevent the storage and data security risks of remote work - TechTarget - August 29th, 2020
- Samsung kills Gallery Sync and Drive support in favor of OneDrive - Android Central - August 29th, 2020
- 4 great Android apps to edit the perfect photo - Phandroid - News for Android - August 29th, 2020