No description

Find a file

Aidan Feldman e3a3e47c9a minor readme tweaks mostly to trigger docker build		2016-03-23 14:55:36 -04:00
assets	`check` returns the most recent LastModified timestamp	2016-03-23 10:22:47 -06:00
test	Update docs for Docker install, Development use, and Tests	2016-03-18 08:32:10 -06:00
.gitignore	pass the AWS credentials through the Concourse source data	2016-03-10 10:57:12 -05:00
config.example.json	Address Feedback	2016-03-21 16:38:36 -06:00
CONTRIBUTING.md	add boilerplate 18F files	2016-03-10 00:45:03 -05:00
Dockerfile	finish up the `out` script	2016-03-10 03:01:02 -05:00
LICENSE.md	add boilerplate 18F files	2016-03-10 00:45:03 -05:00
README.md	minor readme tweaks	2016-03-23 14:55:36 -04:00

README.md

Simple S3 Resource for Concourse CI

Resource to upload files to S3. Unlike the the official S3 Resource, this Resource can upload or download multiple files.

Usage

Include the following in your Pipeline YAML file, replacing the values in the angle brackets (< >):

resource_types:
- name: <resource type name>
  type: docker-image
  source:
    repository: 18fgsa/s3-resource-simple
resources:
- name: <resource name>
  type: <resource type name>
  source:
    access_key_id: {{aws-access-key}}
    secret_access_key: {{aws-secret-key}}
    bucket: {{aws-bucket}}
    options: [<optional, see note below>]
jobs:
- name: <job name>
  plan:
  - <some Resource or Task that outputs files>
  - put: <resource name>

Options

The options parameter is synonymous with the options that aws cli accepts for sync. Please see S3 Sync Options and pay special attention to the Use of Exclude and Include Filters.

Given the following directory test:

test
├── results
│   ├── 1.json
│   └── 2.json
└── scripts
    └── bad.sh

we can upload only the results subdirectory by using the following options in our task configuration:

options:
  - "--exclude: '*'",
  - "--include: 'results/*'"

Development

Requires Docker.

Building, Uploading, and Using the Docker Image

Download Docker Toolbox.
Get a Docker Hub account
Launch the Docker Terminal and cd to this directory.
docker login -e <email> -p <password> -u <username>
docker build -t <username>/s3-resource-simple .
verify with docker images
docker push <username>/s3-resource-simple
Now you can test your local Concourse pipelines using /s3-resource-simple.

Tests

Run cp config.example.json config.json.
Modify config.json.

See the instructions for getting your AWS credentials.
Exclude the s3:// prefix/protocol for bucket.

Run ./test/out </full/path/to/dir>.
Run ./test/in </full/path/to/dir>.
Run ./test/check.