Cloud Storage using Amazon S3 - Direct file access using AWSCLI

Nyquest Consulting Amazon S3 Storage

This week, let's take a look at using Amazon S3.

In laymen's terms, Amazon S3 is an online cloud "cold storage service". It's usefulness comes from the fact that files stored here can be accessed through a variety of methods such as:

  • Web Requests (HTTP directives)
  • Fuse Filesystems
  • Bittorrent
  • Amazon's own AWSCLI

However, please note that S3 storage is only useful for cold storage, meaning storing files whose contents do not constantly change. This means it would be good for things such as, for example, RMAN backups and/or web page images. It WOULD NOT BE good for things such as Oracle data files/redo log files since these files are constantly in a state of flux.

Before we get into the tech details, make sure you have the following already setup.

  1. You have set up your AWS "Access Key" and "Access ID". (Click here if you have not done so)
  2. You have downloaded the AWSCLI and installed it on your machine. Its currently available for Macs/Linux and Windows. (Click here if you have not done so.)

Step 1: Create S3 Container (or Bucket)

Once you have finished with the items above, login into the AWS console and perform the following:

 

Step 2: Configure AWSCLI

Once you are done with the S3 Bucket creation, you need to configure your AWSCLI installation. Perform the following in Terminal or a DOS SHELL.


Make sure you enter the correct region code which is defined in the S3 setup of the previous step. It is not always obvious what the AWS Region Code is with respect to its name, so here is a quick list that is accurate as of 8.2018. The region code that you will need to enter in the AWSCLI setup is in the first column.

Region Code Region Name Availability Zones
1us-east-1 N. Virginia us-east-1a us-east-1b us-east-1c us-east-1d us-east-1e us-east-1f
us-east-2 Ohio us-east-2a us-east-2b us-east-2c
1us-west-1 N. California us-west-1a us-west-1b *us-west-1c
us-west-2 Oregon us-west-2a us-west-2b us-west-2c
2us-gov-west-1 US GovCloud West us-gov-west-1a us-gov-west-1b us-gov-west-1c
ca-central-1 Canada ca-central-1a ca-central-1b
eu-west-1 Ireland eu-west-1a eu-west-1b eu-west-1c
eu-west-2 London eu-west-2a eu-west-2b eu-west-2c
eu-west-3 Paris eu-west-3a eu-west-3b eu-west-3c
eu-central-1 Frankfurt eu-central-1a eu-central-1b eu-central-1c
1ap-northeast-1 Tokyo ap-northeast-1a ap-northeast-1b ap-northeast-1c *ap-northeast-1d
ap-northeast-2 Seoul ap-northeast-2a ap-northeast-2c
4ap-northeast-3 Osaka ap-northeast-3a
ap-southeast-1 Singapore ap-southeast-1a ap-southeast-1b ap-southeast-1c
ap-southeast-2 Sydney ap-southeast-2a ap-southeast-2b ap-southeast-2c
ap-south-1 Mumbai ap-south-1a ap-south-1b
1sa-east-1 Sao Paulo sa-east-1a sa-east-1b *sa-east-1c
3cn-north-1 Bejing cn-north-1a cn-north-1b
3cn-northwest-1 Ningxia cn-northwest-1a cn-northwest-1b cn-northwest-1c

Step 3: Test AWS S3 Commands

Once you have correctly configured the AWSCLI, you should be all set to transfer files to/from S3. A brief overview of S3 commands is shown below.

CommandDescriptionSample Usage
lsList contents of a Bucket (prefix bucket with "s3://")
Using this command w/o arguments will list all buckets
aws s3 ls s3://mybucket
OR
aws s3 ls
cpCp file/files to/from Bucket (prefix bucket with "s3://")aws s3 cp myfile.txt s3://mybucket/myfile.txt
OR
aws s3 cp s3://mybucket/myfile.txt myfile.txt
mvMove file/files to/from Bucket (prefix bucket with "s3://").
After the move is complete the file in the original location is removed
aws s3 mv myfile.txt s3://mybucket/myfile.txt
OR
aws s3 mv s3://mybucket/myfile.txt myfile.txt
rmRemove file/files to/from Bucket (prefix bucket with "s3://")aws s3 rm myfile.txt s3://mybucket/myfile.txt

More information on S3 commands is available here. Some examples are shown below:


That's basically it! While there are numerous cloud storage options available now, the ease of setup and use of S3 makes it particularly attractive to System Administrators whose free time is usually at a minimum. In addition the availability of "*nix like commands makes it extremely easy to integrate with automated processes.


Contact Nyquest Consulting, LLC if you have questions on S3 or any other AWS related components