Simple S3 Resource for Concourse CI

Resource to upload files to S3.

Usage

Include the following in your Pipeline YAML file, replacing the values in the angle brackets (< >):

resource_types:
- name: <resource type name>
  type: docker-image
  source:
    repository: 18fgsa/s3-resource-simple
resources:
- name: <resource name>
  type: <resource type name>
  source:
    access_key_id: {{aws-access-key}}
    secret_access_key: {{aws-secret-key}}
    bucket: {{aws-bucket}}
    options: [<optional, see note below>]
jobs:
- name: <job name>
  plan:
  - <some Resource or Task that outputs files>
  - put: <resource name>

Options

The options parameter is synonymous with the options that aws cli accepts for sync. Please see S3 Sync Options And pay special attention to the Use of Exclude and Include Filters

Given the following directory test we can upload only the results directory by using the options JSON below.

test
├── results
│   ├── 1.json
│   └── 2.json
└── scripts
    └── bad.sh

options: [
  "--exclude: '*'",
  "--include: 'results/*'"
]

Development

Requires Docker.

Building, Uploading, and Using the Docker Image

1. Download Docker Toolbox.
2. Get a Docker Hub account
3. Launch the Docker Terminal and cd to this directory.
4. docker login -e <email> -p <password> -u <username>
5. docker build -t <username>/s3-resource-simple .
6. verify with docker images
7. docker push <username>/s3-resource-simple
8. Now you can test your local Concourse pipelines using /s3-resource-simple.

Tests

1. Run cp config.example.json config.json.
2. Modify config.json.
   • Recommended S3 setup is via cloud.gov's s3 service and a 1-off app.
   • Exclude the s3:// prefix/protocol for bucket.
3. Run ./test/out </full/path/to/dir>.