Cloud Storage using Amazon S3 - Direct file access using AWSCLI
This week, let's take a look at using Amazon S3.
In laymen's terms, Amazon S3 is an online cloud "cold storage service". It's usefulness comes from the fact that files stored here can be accessed through a variety of methods such as:
- Web Requests (HTTP directives)
- Fuse Filesystems
- Bittorrent
- Amazon's own AWSCLI
However, please note that S3 storage is only useful for cold storage, meaning storing files whose contents do not constantly change. This means it would be good for things such as, for example, RMAN backups and/or web page images. It WOULD NOT BE good for things such as Oracle data files/redo log files since these files are constantly in a state of flux.
Before we get into the tech details, make sure you have the following already setup.
- You have set up your AWS "Access Key" and "Access ID". (Click here if you have not done so)
- You have downloaded the AWSCLI and installed it on your machine. Its currently available for Macs/Linux and Windows. (Click here if you have not done so.)
Step 1: Create S3 Container (or Bucket)
Once you have finished with the items above, login into the AWS console and perform the following:
Step 2: Configure AWSCLI
Once you are done with the S3 Bucket creation, you need to configure your AWSCLI installation. Perform the following in Terminal or a DOS SHELL.
Make sure you enter the correct region code which is defined in the S3 setup of the previous step. It is not always obvious what the AWS Region Code is with respect to its name, so here is a quick list that is accurate as of 8.2018. The region code that you will need to enter in the AWSCLI setup is in the first column.
| Region Code | Region Name | Availability Zones |
|---|---|---|
| 1us-east-1 | N. Virginia | us-east-1a us-east-1b us-east-1c us-east-1d us-east-1e us-east-1f |
| us-east-2 | Ohio | us-east-2a us-east-2b us-east-2c |
| 1us-west-1 | N. California | us-west-1a us-west-1b *us-west-1c |
| us-west-2 | Oregon | us-west-2a us-west-2b us-west-2c |
| 2us-gov-west-1 | US GovCloud West | us-gov-west-1a us-gov-west-1b us-gov-west-1c |
| ca-central-1 | Canada | ca-central-1a ca-central-1b |
| eu-west-1 | Ireland | eu-west-1a eu-west-1b eu-west-1c |
| eu-west-2 | London | eu-west-2a eu-west-2b eu-west-2c |
| eu-west-3 | Paris | eu-west-3a eu-west-3b eu-west-3c |
| eu-central-1 | Frankfurt | eu-central-1a eu-central-1b eu-central-1c |
| 1ap-northeast-1 | Tokyo | ap-northeast-1a ap-northeast-1b ap-northeast-1c *ap-northeast-1d |
| ap-northeast-2 | Seoul | ap-northeast-2a ap-northeast-2c |
| 4ap-northeast-3 | Osaka | ap-northeast-3a |
| ap-southeast-1 | Singapore | ap-southeast-1a ap-southeast-1b ap-southeast-1c |
| ap-southeast-2 | Sydney | ap-southeast-2a ap-southeast-2b ap-southeast-2c |
| ap-south-1 | Mumbai | ap-south-1a ap-south-1b |
| 1sa-east-1 | Sao Paulo | sa-east-1a sa-east-1b *sa-east-1c |
| 3cn-north-1 | Bejing | cn-north-1a cn-north-1b |
| 3cn-northwest-1 | Ningxia | cn-northwest-1a cn-northwest-1b cn-northwest-1c |
Step 3: Test AWS S3 Commands
Once you have correctly configured the AWSCLI, you should be all set to transfer files to/from S3. A brief overview of S3 commands is shown below.| Command | Description | Sample Usage |
|---|---|---|
| ls | List contents of a Bucket (prefix bucket with "s3://") Using this command w/o arguments will list all buckets | aws s3 ls s3://mybucket OR aws s3 ls |
| cp | Cp file/files to/from Bucket (prefix bucket with "s3://") | aws s3 cp myfile.txt s3://mybucket/myfile.txt OR aws s3 cp s3://mybucket/myfile.txt myfile.txt |
| mv | Move file/files to/from Bucket (prefix bucket with "s3://"). After the move is complete the file in the original location is removed | aws s3 mv myfile.txt s3://mybucket/myfile.txt OR aws s3 mv s3://mybucket/myfile.txt myfile.txt |
| rm | Remove file/files to/from Bucket (prefix bucket with "s3://") | aws s3 rm myfile.txt s3://mybucket/myfile.txt |
