Use AWS CLI with PILW.IO StorageVault

Overview

One PILW.IO key offering is its StorageVault. It is REST API S3 based object storage enabling to store files as an objects. You can read more about our object storage in one of our earlier blog posts. Also we have one blog post describing how to use PILW.IO StorageVault with Cyberduck file GUI based file browser. With this blog post we will explain how to use StorageVault with AWS command line interface and how to interact with PILW.IO StorageVault.

AWS CLI is S3 based object storage interaction interface. So to say S3 command line client if you will. It has a support for most of the operating systems and it is being constantly developed. The command line interface can be integrated with scripts or programs. There are also many libraries available to use S3 inside the programs. We probably cover few of these in future blog posts.

Preparation

To explain, how object storage CLI can be used, I have prepared one Ubuntu system in PILW.IO environment. It is just the small system running all necessary components to run the demonstration. The server is with following configuration:

• 1 virtual CPU
• 512MB RAM
• 20GB Disk
• OS is Ubuntu 18.04LTS

Obviously I have network to public and private networks, although for the example here we use just public network address. The operation system I would suggest to have also based on use case. The CLI is supported to run on Windows, Linux or MacOS. Or in fact any Unix system. I am using Ubuntu 18.04LTS, but it works as well with 16.04LTS or any other version.

Depending on a use case, the system can be located anywhere, in cloud or in your own environment. As long as it has access to PILW.IO StorageVault.

You need Python2 (2.6.5+) or Python3 (3.3+) installed in system. Depending on how you install the awscli you might need the package management system installed too. In the example herewith we use Python package manager pip. You could use also operating system package repositories.

The StorageVault access address is https://s3.pilw.io:8080. You could access it directly with your web browser, but of course you need to have proper credentials.

AWS CLI installation

As described above, I will install AWS CLI with Python package manager. The installation can be started as follows:

~$ pip install awscli --upgrade

The --upgrade in the end of the command will tell to pip just to update all existing packages so these would work together with installed package. Once installed, you can check if everything went as planned, by checking the version of awscli. Here is a command and result:

~$ aws --version
aws-cli/1.16.24 Python/3.6.6 Linux/4.15.0-34-generic botocore/1.12.14

To update the existing AWS CLI installation, you can just run installation command. To uninstall AWS CLI, run a command:

awscli:~$ pip uninstall awscli

Configuration

Once installed, we need to configure settings for AWS CLI. We need to provide access information to PILW.IO storage vault. It can be done with following command:

~$ aws configure
AWS Access Key ID [None]: LY6PSWJKNZPPFAT8QVPP
AWS Secret Access Key [None]: pfJuhreDHPNrQhqCiyaGoDvWT3RmTHmuh8XLomLL
Default region name [None]: us-east-1
Default output format [None]: json

You can get the Access Key and Secret Access Key in PILW.IO portal In API tab. If you do not have keys there, you can generate keys easily. We have more about these in one of our earlier blog posts. In example herewith I am using my example Access Key ID and Secret Access Key here.

The region is default storage region. And output format means, how AWS CLI responds back when you issue command. There are other formats, and we can take these later on. We also are able to say the preferred output format in command line.

The aws configure command creates .aws subdirectory to your home catalog, and two files in it (config and credentials).

~$ cat .aws/config
[default]
region = us-east-1
output = json

~$ cat .aws/credentials
[default]
aws_access_key_id = LY6PSWJKNZPPFAT8QVPP
aws_secret_access_key = pfJuhreDHPNrQhqCiyaGoDvWT3RmTHmuh8XLomLL

The files can contain also several profiles (like [default] in files), when you need to have different access to PILW.IO StorageVault or you would like to access some other storage solution. Profiles can be added manually or to run a command:

~$ aws configure --profile <profile name>

You can use aws configure command once again to make changes to these files or you can also manually edit these files.

Using aws cli S3

AWS CLI itself is tool to manage pretty much all AWS services. We do offer here S3 compatible object storage and focus on accessing it here in this article. The command syntax is:

aws <command> <subcommand> [options and parameters]

There are two commands to access files in PILW.IO StorageVaut s3 and s3api command. We go through the s3 command first. So s3 would be the command we use here for now. To warm up lets test it with listing buckets in our StorageVault:

~$ aws s3 ls --endpoint=https://s3.pilw.io:8080
2018-08-20 04:57:10 nextcloud-demo
2018-09-08 07:25:29 orchesto
2018-08-15 15:25:01 prodbucket
2018-06-15 08:01:47 s3fsmount

The result are various buckets in StorageVault. The s3 is a command and ls is a subcommand that runs relative to the command. --endpoint=https://s3.pilw.io:8080 in the command arguments is a parameter. The order itself is not that important so we can actually change the order of parameter and command in the command line. With that we can simplify command a bit by creating alias for the command:

~$ alias pilwios3="aws --endpoint=https://s3.pilw.io:8080"

Now just run the command:

~$ pilwios3 s3 ls

and you should get same result as above. You can store the alias in .bash_aliases file or .bashrc file to be loaded every time you log in.

Now since we have now basic understanding how the command works. First lets create one new bucket to our storage. Bucket details are given as parameters with s3://bucket-name. Bucket names can contain lowercase letters, numbers, hyphens and periods. Bucket names can only start and end with a letter or number, and cannot contain a period next to a hyphen or another period. Also bucket names must be unique and DNS compliant.

~$ pilwios3 s3 mb s3://aws.clitest.s3
make_bucket: aws.clitest.s3

~$ pilwios3 s3 ls
2018-09-30 14:56:38 aws.clitest.s3
2018-08-20 04:57:10 nextcloud-demo
2018-09-08 07:25:29 orchesto
2018-08-15 15:25:01 prodbucket
2018-06-15 08:01:47 s3fsmount

As yo usee the bucket is nicely listed. Lets put something in it now. I have directory called testset where I have some smaller files and one large file. With commands herewith below, I will copy these files to StorageVault.

I copy testset/lfile_1G.txt to StorageVault:

~$ pilwios3 s3 cp testset/lfile_1G.txt s3://aws.clitest.s3
upload: testset/lfile_1G.txt to s3://aws.clitest.s3/lfile_1G.txt

~$ pilwios3 s3 ls s3://aws.clitest.s3
2018-09-30 15:14:25 1073741824 lfile_1G.txt

As we can create buckets, we can also remove buckets:

~$ pilwios3 s3 rb s3://aws.toremove
remove_bucket: aws.toremove

We can copy whole directory to StorageVault, with all subdirectories

~$ pilwios3 s3 cp testset s3://aws.clitest.s3 --recursive

You can remove files with rm command.

~$ pilwios3 s3 rm s3://aws.clitest.s3/sfile_1.txt
delete: s3://aws.clitest.s3/sfile_1.txt

If you use --recursive flag, all files relative to provided bucket or directory in bucket, will be deleted.

~$ pilwios3 s3 rm s3://aws.clitest.s3 --recursive
delete: s3://aws.clitest.s3/sfile_2.txt
delete: s3://aws.clitest.s3/sfile_3.txt
delete: s3://aws.clitest.s3/lfile_1G.txt

You can sync directory to StorageVault:

~$ pilwios3 s3 sync testset s3://aws.clitest.s3/testset
upload: testset/sfile_1.txt to s3://aws.clitest.s3/testset/sfile_1.txt
upload: testset/sfile_2.txt to s3://aws.clitest.s3/testset/sfile_2.txt
upload: testset/sfile_3.txt to s3://aws.clitest.s3/testset/sfile_3.txt

~$ pilwios3 s3 ls s3://aws.clitest.s3/testset/
2018-09-30 15:46:14       4717 sfile_1.txt
2018-09-30 15:46:14      11700 sfile_2.txt
2018-09-30 15:46:14       6564 sfile_3.txt

Now lets remove file from testset directory and run sync again with --delete flag:

~$ rm testset/sfile_3.txt

~$ pilwios3 s3 sync testset s3://aws.clitest.s3/testset --delete
delete: s3://aws.clitest.s3/testset/sfile_3.txt

~$ pilwios3 s3 ls s3://aws.clitest.s3/testset/
2018-09-30 15:46:14       4717 sfile_1.txt
2018-09-30 15:46:14      11700 sfile_2.txt

Without --delete flag, file will not be deleted from bucket. We can also sync files to StorageVault and make these publicly available:

~$ pilwios3 s3 sync testset2 s3://aws.clitest.s3/testset2 --acl public-read
upload: testset2/sfile_2.txt to s3://aws.clitest.s3/testset2/sfile_2.txt
upload: testset2/sfile_3.txt to s3://aws.clitest.s3/testset2/sfile_3.txt
upload: testset2/sfile_1.txt to s3://aws.clitest.s3/testset2/sfile_1.txt

Now pointing our web browser to https://s3.pilw.io:8080/aws.clitest.s3/testset2/sfile_1.txt, we can see contents of this file. With --acl flag you can control also whether other people or the rest of the world will have write access to the file. The possible options can be private, public-read, public-read-write, authenticated-read, aws-exec-read, bucket-owner-read, bucket-owner-full-control and log-delivery-write.

You can enable temporary access to the file with:

~$ pilwios3 s3 presign s3://aws.clitest.s3/testset2/sfile_3.txt --expires-in 120
https://s3.pilw.io:8080/aws.clitest.s3/testset2/sfile_3.txt?AWSAccessKeyId=LY6PSWJKNZPPFAT8QVPP&Signature=yndFtAB9Oad8OpQykgh%2FE5W2kJU%3D&Expires=1538323752

This will create temporary link to the file, which is available number of seconds defined with --expires-in flag. In example here it made link available for 120 seconds only. Without --expires-in flag, the default is 3600 seconds. It is useful if you want to share a link with co-workers or customers to presentation that might be valid for a short time. Or share photos with friends after last trip together.

Here we had few commands that you can run with AWS CLI. The commands which you can run with S3 storage access are:

• cp – copy file or directory to/from S3 storage
• ls – list bucket or directory contents
• mb – make new bucket
• mv – move file or directory
• presign – generate pre-singed URL for object assign access
• rb – remove bucket
• rm – remove file or directory from bucket
• sync – sync folders or buckets to S3 bucket
• website – configure bucket as a static website

You can read more about the commands and subcommands with aws s3 help command. Besides moving files or directories to/from bucket, you can also move files around between buckets or even storage solutions.

This is far from all that you can do with AWS CLI. There is a bit more comprehensive approach to the S3 storage, which we will cover with some of the next blog articles.