Object storace consumption should be easy even with API’s. The simple AWS API command line tool is
aws s3 and it is very easy to use. With this blog article we will introduce AWS CLI S3 command and show how to use it with PILW.IO StorageVault object storage solution.
One PILW.IO key offering is its StorageVault. It is REST API S3 based object storage enabling to store files as an objects. You can read more about our object storage in one of our earlier blog posts. Also we have one blog post describing how to use PILW.IO StorageVault with Cyberduck file GUI based file browser. With this blog post we will explain how to use StorageVault with AWS command line interface and how to interact with PILW.IO StorageVault.
AWS CLI is S3 based object storage interaction interface. So to say S3 command line client if you will. It has a support for most of the operating systems and it is being constantly developed. The command line interface can be integrated with scripts or programs. There are also many libraries available to use S3 inside the programs. We probably cover few of these in future blog posts.
To explain, how object storage CLI can be used, I have prepared one Ubuntu system in PILW.IO environment. It is just the small system running all necessary components to run the demonstration. The server is with following configuration:
- 1 virtual CPU
- 512MB RAM
- 20GB Disk
- OS is Ubuntu 18.04LTS
Obviously I have network to public and private networks, although for the example here we use just public network address. The operation system I would suggest to have also based on use case. The CLI is supported to run on Windows, Linux or MacOS. Or in fact any Unix system. I am using Ubuntu 18.04LTS, but it works as well with 16.04LTS or any other version.
Depending on a use case, the system can be located anywhere, in cloud or in your own environment. As long as it has access to PILW.IO StorageVault.
You need Python2 (2.6.5+) or Python3 (3.3+) installed in system. Depending on how you install the awscli you might need the package management system installed too. In the example herewith we use Python package manager
pip. You could use also operating system package repositories.
The StorageVault access address is https://s3.pilw.io:8080. You could access it directly with your web browser, but of course you need to have proper credentials.
AWS CLI installation
As described above, I will install AWS CLI with Python package manager. The installation can be started as follows:
~$ pip install awscli --upgrade
--upgrade in the end of the command will tell to pip just to update all existing packages so these would work together with installed package. Once installed, you can check if everything went as planned, by checking the version of awscli. Here is a command and result:
~$ aws --version aws-cli/1.16.24 Python/3.6.6 Linux/4.15.0-34-generic botocore/1.12.14
To update the existing AWS CLI installation, you can just run installation command. To uninstall AWS CLI, run a command:
awscli:~$ pip uninstall awscli
Once installed, we need to configure settings for AWS CLI. We need to provide access information to PILW.IO storage vault. It can be done with following command:
~$ aws configure AWS Access Key ID [None]: LY6PSWJKNZPPFAT8QVPP AWS Secret Access Key [None]: pfJuhreDHPNrQhqCiyaGoDvWT3RmTHmuh8XLomLL Default region name [None]: us-east-1 Default output format [None]: json
You can get the Access Key and Secret Access Key in PILW.IO portal In API tab. If you do not have keys there, you can generate keys easily. We have more about these in one of our earlier blog posts. In example herewith I am using my example Access Key ID and Secret Access Key here.
The region is default storage region. And output format means, how AWS CLI responds back when you issue command. There are other formats, and we can take these later on. We also are able to say the preferred output format in command line.
aws configure command creates
.aws subdirectory to your home catalog, and two files in it (
~$ cat .aws/config [default] region = us-east-1 output = json ~$ cat .aws/credentials [default] aws_access_key_id = LY6PSWJKNZPPFAT8QVPP aws_secret_access_key = pfJuhreDHPNrQhqCiyaGoDvWT3RmTHmuh8XLomLL
The files can contain also several profiles (like
[default] in files), when you need to have different access to PILW.IO StorageVault or you would like to access some other storage solution. Profiles can be added manually or to run a command:
~$ aws configure --profile <profile name>
You can use
aws configure command once again to make changes to these files or you can also manually edit these files.
Using aws cli S3
AWS CLI itself is tool to manage pretty much all AWS services. We do offer here S3 compatible object storage and focus on accessing it here in this article. The command syntax is:
aws <command> <subcommand> [options and parameters]
There are two commands to access files in PILW.IO StorageVaut s3 and s3api command. We go through the s3 command first. So s3 would be the command we use here for now. To warm up lets test it with listing buckets in our StorageVault:
~$ aws s3 ls --endpoint=https://s3.pilw.io:8080 2018-08-20 04:57:10 nextcloud-demo 2018-09-08 07:25:29 orchesto 2018-08-15 15:25:01 prodbucket 2018-06-15 08:01:47 s3fsmount
The result are various buckets in StorageVault. The
s3 is a command and
ls is a subcommand that runs relative to the command.
--endpoint=https://s3.pilw.io:8080 in the command arguments is a parameter. The order itself is not that important so we can actually change the order of parameter and command in the command line. With that we can simplify command a bit by creating alias for the command:
~$ alias pilwios3="aws --endpoint=https://s3.pilw.io:8080"
Now just run the command:
~$ pilwios3 s3 ls
and you should get same result as above. You can store the alias in .bash_aliases file or .bashrc file to be loaded every time you log in.
Now since we have now basic understanding how the command works. First lets create one new bucket to our storage. Bucket details are given as parameters with
s3://bucket-name. Bucket names can contain lowercase letters, numbers, hyphens and periods. Bucket names can only start and end with a letter or number, and cannot contain a period next to a hyphen or another period. Also bucket names must be unique and DNS compliant.
~$ pilwios3 s3 mb s3://aws.clitest.s3 make_bucket: aws.clitest.s3 ~$ pilwios3 s3 ls 2018-09-30 14:56:38 aws.clitest.s3 2018-08-20 04:57:10 nextcloud-demo 2018-09-08 07:25:29 orchesto 2018-08-15 15:25:01 prodbucket 2018-06-15 08:01:47 s3fsmount
As yo usee the bucket is nicely listed. Lets put something in it now. I have directory called testset where I have some smaller files and one large file. With commands herewith below, I will copy these files to StorageVault.
testset/lfile_1G.txt to StorageVault:
~$ pilwios3 s3 cp testset/lfile_1G.txt s3://aws.clitest.s3 upload: testset/lfile_1G.txt to s3://aws.clitest.s3/lfile_1G.txt ~$ pilwios3 s3 ls s3://aws.clitest.s3 2018-09-30 15:14:25 1073741824 lfile_1G.txt
As we can create buckets, we can also remove buckets:
~$ pilwios3 s3 rb s3://aws.toremove remove_bucket: aws.toremove
We can copy whole directory to StorageVault, with all subdirectories
~$ pilwios3 s3 cp testset s3://aws.clitest.s3 --recursive
You can remove files with
~$ pilwios3 s3 rm s3://aws.clitest.s3/sfile_1.txt delete: s3://aws.clitest.s3/sfile_1.txt
If you use
--recursive flag, all files relative to provided bucket or directory in bucket, will be deleted.
~$ pilwios3 s3 rm s3://aws.clitest.s3 --recursive delete: s3://aws.clitest.s3/sfile_2.txt delete: s3://aws.clitest.s3/sfile_3.txt delete: s3://aws.clitest.s3/lfile_1G.txt
You can sync directory to StorageVault:
~$ pilwios3 s3 sync testset s3://aws.clitest.s3/testset upload: testset/sfile_1.txt to s3://aws.clitest.s3/testset/sfile_1.txt upload: testset/sfile_2.txt to s3://aws.clitest.s3/testset/sfile_2.txt upload: testset/sfile_3.txt to s3://aws.clitest.s3/testset/sfile_3.txt ~$ pilwios3 s3 ls s3://aws.clitest.s3/testset/ 2018-09-30 15:46:14 4717 sfile_1.txt 2018-09-30 15:46:14 11700 sfile_2.txt 2018-09-30 15:46:14 6564 sfile_3.txt
Now lets remove file from
testset directory and run sync again with
~$ rm testset/sfile_3.txt ~$ pilwios3 s3 sync testset s3://aws.clitest.s3/testset --delete delete: s3://aws.clitest.s3/testset/sfile_3.txt ~$ pilwios3 s3 ls s3://aws.clitest.s3/testset/ 2018-09-30 15:46:14 4717 sfile_1.txt 2018-09-30 15:46:14 11700 sfile_2.txt
--delete flag, file will not be deleted from bucket. We can also sync files to StorageVault and make these publicly available:
~$ pilwios3 s3 sync testset2 s3://aws.clitest.s3/testset2 --acl public-read upload: testset2/sfile_2.txt to s3://aws.clitest.s3/testset2/sfile_2.txt upload: testset2/sfile_3.txt to s3://aws.clitest.s3/testset2/sfile_3.txt upload: testset2/sfile_1.txt to s3://aws.clitest.s3/testset2/sfile_1.txt
Now pointing our web browser to https://s3.pilw.io:8080/aws.clitest.s3/testset2/sfile_1.txt, we can see contents of this file. With
--acl flag you can control also whether other people or the rest of the world will have write access to the file. The possible options can be
You can enable temporary access to the file with:
~$ pilwios3 s3 presign s3://aws.clitest.s3/testset2/sfile_3.txt --expires-in 120 https://s3.pilw.io:8080/aws.clitest.s3/testset2/sfile_3.txt?AWSAccessKeyId=LY6PSWJKNZPPFAT8QVPP&Signature=yndFtAB9Oad8OpQykgh%2FE5W2kJU%3D&Expires=1538323752
This will create temporary link to the file, which is available number of seconds defined with
--expires-in flag. In example here it made link available for 120 seconds only. Without
--expires-in flag, the default is 3600 seconds. It is useful if you want to share a link with co-workers or customers to presentation that might be valid for a short time. Or share photos with friends after last trip together.
Here we had few commands that you can run with AWS CLI. The commands which you can run with S3 storage access are:
- cp – copy file or directory to/from S3 storage
- ls – list bucket or directory contents
- mb – make new bucket
- mv – move file or directory
- presign – generate pre-singed URL for object assign access
- rb – remove bucket
- rm – remove file or directory from bucket
- sync – sync folders or buckets to S3 bucket
- website – configure bucket as a static website
You can read more about the commands and subcommands with
aws s3 help command. Besides moving files or directories to/from bucket, you can also move files around between buckets or even storage solutions.
This is far from all that you can do with AWS CLI. There is a bit more comprehensive approach to the S3 storage, which we will cover with some of the next blog articles.