Amazon s3 client linux

How to Install s3cmd in Linux and Manage S3 Buckets

s3cmd is a command line utility used for creating s3 buckets, uploading, retrieving and managing data to Amazon s3 storage. This article will help you to how to use install s3cmd on CentOS, RHEL, OpenSUSE, Ubuntu, Debian & LinuxMint systems and manage s3 buckets via command line in easy steps. To install s3cmd on windows servers read article install s3cmd in Windows.

We can also mount s3 bucket as local drive in our system using S3FS with FUSE. To configure it read next article mount s3 bucket on Linux.

Install s3cmd on Linux

s3cmd is available in default package repositories for the Ubuntu, Debian, Fedora, CentOS, and RHEL Linux systems. You can install it using simply executing the following commands on your system.

On CentOS/RHEL and Fedora:

On Ubuntu/Debian:

Install Latest s3cmd using Source

If you are not getting the latest version of s3cmd using package managers, You can install the last s3cmd version on your system using the source code. Visit this url or use below command to download latest version of s3cmd.

Then install it using the below command with source files.

Configure S3cmd Environment

In order to configure s3cmd we would require Access Key and Secret Key of your S3 Amazon account. Get these security keys from aws securityCredentials page. If will prompt to login to your amazon account.

After getting key files, use the below command to configure s3cmd.

Working with the s3cmd Command Line

Once the configuration is successfully completed. Now find below command details to how to manage s3 buckets using commands.

1. List All S3 Bucket

Use the following command to list all s3 buckets in your aws account.

2. Creating New Bucket

To create a new bucket in Amazon s3 use the below command. It will create bucket named tecadmin in S3 account.

3. Uploading file in Bucket

Below command will upload file file.txt to s3 bucket using s3cmd command.

4. Uploading Directory in Bucket

If we need to upload entire directory use -r to upload it recursively as below.

Make sure you are not adding a trailing slash in the upload directory named backup (eg: backup/), else it will upload only content to the backup directory only.

5. List Data of S3 Bucket

List the objects of s3 bucket using ls switch with s3cmd.

6. Download Files from Bucket

Sometimes if we need to download files from the s3 bucket, Use the following commands to download them.

7. Remove Files from S3 Bucket

You can also remove a file or a directory from s3 bucket. Read the below examples to delete a file or a directory from an s3 bucket with s3cmd.

To Remove file from s3 bucket:

Removing directory from s3 bucket:

8. Remove S3 Bucket

If we don’t need s3 bucket anymore, we can simply delete it using the following command. Before removing the bucket make sure it is empty.

Above command failed because of s3 bucket was not empty

To remove the bucket first remove all objects inside the bucket and then use the command again.

Thanks for using this article. If you want to mount s3bucket in your system, you can article Mount S3bucket in Linux using s3fs. Also you can sync data between s3 bucket and local directory using s3cmd.

Источник

Linux Package Repositories

S3cmd 1.5.0 pre-release packages in Fedora, RHEL, CentOS, Debian

In an effort to get additional users testing the 1.5.0 codebase, in preparation for a final 1.5.0 release, new packages have been published for Fedora, RHEL, CentOS, and Debian. You may wish to use these, rather than each distribution’s primary release repositories.

In Fedora, you can find a release slightly newer than 1.5.0-beta1 in the updates-testing repository for Fedora 18, 19, 20, and rawhide.

$ sudo yum —enablerepo updates-testing install s3cmd

In EPEL (Red Hat Enterprise Linux RHEL, CentOS, etc) you can find a release slightly newer than 1.5.0-beta1 in the updates-testing repository for EPEL 6 and beta/7. Please note that EPEL must be installed for this to work.

$ sudo yum —enablerepo epel-testing install s3cmd

In Debian, you can find a release slightly newer than 1.5.0-beta1 in the Experimental repository

S3cmd 1.0.0 packages in CentOS, RHEL, Fedora, SLES, Debian, Ubuntu

Some Linux distributions now provide s3cmd package in their base or add-ons package repositories. Unfortunately these repositories are very often “frozen” in the sense that package versions are never upgraded. From some points of view this is an understandable policy, however it also means that you will never automatically get any new features of future s3cmd releases.

For example – Fedora 8 (FC8) has been released with s3cmd 0.9.8.1 and even if security issues may get fixed in their updates repository it is unlikely that the users of FC8 will ever get any new features from s3cmd 1.0.0, for instance.

Therefore we decided to provide package repositories / RPM repositories / DEB repositories for some of the most popular distributions with the always most recent s3cmd package ready for installation.

Here is a list of currently supported distributions:

Repository	.repo file
RHEL 5 & CentOS 5	s3tools.repo
RHEL 6 & CentOS 6	s3tools.repo	Use also for Amazon Linux AMI and Fedora
SLES 11	s3tools.repo
Debian & Ubuntu

We can’t provide packages for discontinued RPM based distributions like openSUSE 10.3 or Fedora 10 and older. However you can grab the .src.rpm file from one of the repositories above and rebuild it for your system, that should work just fine.

How to add s3cmd 1.0.0 repository to RedHat, CentOS and Fedora

There are probably some graphical package managers in RedHat based systems, but we only use yum 😉

1. As a superuser (root) go to
2. Download s3tools.repo file for your distribution. Links to these .repo files are in the table above. For instance if you’re on CentOS 6.x
3. Run if you don’t have s3cmd rpm package installed yet, or if you already have s3cmd rpm installed and long for a newer version.
4. You will be asked to accept a new GPG key – answer yes (perhaps twice).
5. That’s it. Next time you run you’ll automatically get the very latest s3cmd for your system.

How to add s3cmd 1.0.0 repository to SLES 11

There are two ways to do it. The one described below uses command line package management tool called zypper, the other way is using YaST.

1. Become a superuser (root)
2. Find the s3tools.repo URL in the table above and run for instance: if you’re on OpenSUSE 11.0
3. Install s3cmd with:
4. You will be asked whether you want to trust a new GPG key. Answer yes two times.
5. That’s it. The s3cmd package will now be kept up to date together with all your other installed packages.

Debian & Ubuntu

Our DEB repository has been carefully created in the most compatible way – it should work for Debian 5 (Lenny), Debian 6 (Squeeze), Ubuntu 10.04 LTS (Lucid Lynx) and for all newer and possibly for some older Ubuntu releases. Follow these steps from the command line:

Источник

Ubuntu 18 AWS S3 best client, terminal and setup

Linux Ubuntu 18 and Linux Mint 18.1 offer good and reliable way to work with AWS S3 or Amazon Simple Storage Service.

• Prerequisite
• Install AWSCLI with pip
• Easy Configuration for AWS CLI
• Best client for AWS S3 FileZilla Pro
• Resources

Prerequisite

In order to install AWS client on Linux you will need to have pip a package manager for Python. By default Ubuntu and Linux Mint come with default installation of python. If you want to check the version then type:

Check pip version:

You may need to upgrade pip by:

Note: you can use pip3 for python3.

Check python version (python 2):

By default pip is included in python default packages. If you need to install it manually you can check install pip with easy_install.

Install AWSCLI with pip

If you want to use it you can do install the aws client by(giving user or not):

Once the installation process is finished you can verify the application version by:

Upgrading of the AWS client can be done by:

Depending on are you using user or not.

In case that you are using environments for python you can check your installation folder by:

or for default python:

Easy Configuration for AWS CLI

The simplest possible configuration for AWS client is to setup 4 properties:

• AWS Access Key ID — your personal key ( you need to ask your admin to provide this information for you)
• AWS Secret Access Key — your personal Secret Access Key
• Default region name — you can find your region from here: AWS Regions and Endpoints
• Default output format — allows different formats like: text, xml, json..

Configuration is done by the command:

or if you have different profiles:

AWS Access Key ID [None]: AKIAI44QH8DHBEXAMPLE
AWS Secret Access Key [None]: je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
Default region name [None]: eu-central-1
Default output format [None]: json

Best client for AWS S3 FileZilla Pro

The best client which I was able to use for AWS S3 is FileZilla Pro. It’s good to mention that is paid. In order to get it you can go on this link:

I was able to run it with wine or under virtual machine. In order to setup it you can:

Источник

linux-notes.org

AWS CLI — это консольная утилита которая позволяет работать с AWS S3 через командную строку. Я хотел бы сделать заметку о том как работать AWS S3 в консоле.

Работа с AWS S3 через командную строку в Unix/Linux

Устанавливаем AWS CLI:

Прописываем ключи. Например:

• Your_acc_name — Имя для аккаунта. Потом можно его использовать.
• aws_access_key_id — Ключик.
• aws_secret_access_key — Еще один ключик.

Создание AWS S3 bucket-ов через командную строку в Unix/Linux

Работа с бакетом, т.е с S3 начнется непосредственно, с создания самого бакета.

Чтобы создать, используйте:

Иногда, нужно указывать регион в котором будет создаваться бакет:

Так же, можно через s3api, например:

Ну, создание вроде-бы понятно как создавать уже. Идем дальше.

Просмотр AWS S3 bucket-ов через командную строку в Unix/Linux

Чтобы просмотреть содержимое бакета, используйте:

ИЛИ, просмотреть какие бакеты имеются:

Или, можно вывести список бакетов по конкретному профелю:

Так же, можно просмотреть данные рекурсивно для конкретной корзины:

ИЛИ, вывести сколько занимают данные:

Копирование, удаление, перемещение данных AWS S3 bucket-ов через командную строку в Unix/Linux

В следующем примере объект копируется в bucket (Test_bucket). Ему назначено чтение для всех пользователей и полный доступ (чтение и запись) для учетной записи My_user@linux-notes.org:

Чтобы указать класс хранения(REDUCED_REDUNDANCY или STANDARD_IA) для объектов которые вы загружаете в S3, используйте опцию —storage-class:

Например, давайте скопируем некоторый файл со своего рабочего сервера в бакет:

Еще пример, — например нужно скопировать все *.jpg файлы в s3://Test_bucket/some_path_in_backet бакете в ./MyDirectory:

Если пунктом назначения указан корень bucket-а, то будут скопированы только файлы из указанного каталога:

Если указать каталог, то будет скопировано всё дерево файлов/каталогов:

Можно копировать данные между S3 бакетами:

Чтобы удалить файл в бакете, используйте:

Чтобы удалить весь контент в бакете (рекурсивно), используйте:

Вроде бы понятно все.

Синхронизация AWS S3 bucket-ов через командную строку в Unix/Linux

В следующем примере выполняется синхронизация текущего каталога с бакетом:

Normally, sync only copies missing or outdated files or objects between the source and target. However, you may supply the —delete option to remove files or objects from the target not present in the source.

Если вы удалили в текущем каталоге файл и хотите чтобы он удалился ( засинкался) в бакете, используйте «—delete» опцию:

Используем синхронизацию определенного класса:

Команда sync также принимает параметр «—acl», с помощью которого вы можете установить права доступа для файлов, скопированных в S3. Параметр принимает значения private, public-read и public-read-write:

Удаление AWS S3 bucket-ов через командную строку в Unix/Linux

Чтобы удалить бакет, используйте:

Так же можно использовать принудительное удаление бакета (если он не пустой). Для этого служит «—force» опция:

Вот и все, статья «Работа с AWS S3 через командную строку в Unix/Linux» завершена.

Источник

Amazon S3 Tools: Command Line S3 Client Software and S3 Backup

AWS S3 Command Line Clients for Windows, Linux, Mac. Backup to S3, upload, retrieve, query data on Amazon S3.

S3cmd : Command Line S3 Client and Backup for Linux and Mac

Amazon S3 is a reasonably priced data storage service. Ideal for off-site file backups, file archiving, web hosting and other data storage needs. It is generally more reliable than your regular web hosting for storing your files and images. Check out about Amazon S3 to find out more.

S3cmd is a free command line tool and client for uploading, retrieving and managing data in Amazon S3 and other cloud storage service providers that use the S3 protocol, such as Google Cloud Storage or DreamHost DreamObjects. It is best suited for power users who are familiar with command line programs. It is also ideal for batch scripts and automated backup to S3, triggered from cron, etc.

S3cmd is written in Python. It’s an open source project available under GNU Public License v2 (GPLv2) and is free for both commercial and private use. You will only have to pay Amazon for using their storage.

Lots of features and options have been added to S3cmd, since its very first release in 2008. we recently counted more than 60 command line options, including multipart uploads, encryption, incremental backup, s3 sync, ACL and Metadata management, S3 bucket size, bucket policies, and more!

S3Express : Command Line S3 Client and S3 Backup for Windows

S3Express is a commercial S3 command line tool for Windows. Differently from S3cmd, S3Express is designed to run specifically on Windows, it is self-contained in one executable (s3express.exe) and does not require any additional libraries or software to be installed to run. It’s very compact and has very small footprint: the entire program is less than 5MB. It’s very easy to install on Windows servers, clients, desktops and laptop computers alike.

S3Express is developed and maintained by TGRMN Software, the company behind ViceVersa PRO, professional software for file backup, file synchronization and file copy on Windows.

With S3Express you can list and query S3 objects using conditional filters, manage S3 objects’ metadata and ACLs, upload files to S3 using multipart uploads and multiple concurrent threads, upload only new or changed files to S3 for automated backup (= incremental backup), delete multiple S3 objects, copy S3 objects, etc. using the Windows command line.

All operations in S3Express are multithreaded (fast), automatically retryable (network-failure resistant), and interruptible (= all commands can be stopped and restarted at any time). Connections to Amazon S3 are made using secure http (https), which is an encrypted version of the HTTP protocol, to protect your files while they’re in transit to and from Amazon S3 storage servers.

Источник