Upload/download files to/from Google Storage

Upload and download a JAR from Google Storage from within a pipeline

Prerequisites

• A Codefresh account
• A Google Storage Bucket with public read access
• A private key downloaded for the existing service account associated with your bucket (for this example, we base64 encoded the key for ease of use in a pipeline variable using base64 key_file.json > key_file.b64)

Example Project

The example project is at GitHub. The application is a simple Scala Hello World application contained in a jar, with a dependency on a scala-library jar which we will download from the bucket and package into a Docker image.

Our project contains two pipelines, one to upload the dependency JAR to our bucket, and the other to download the JAR from the bucket.

Create the first pipeline

The first pipeline contains one stage/step, to upload the JAR to the Google Storage Bucket.

You need to define a pipeline variable, KEY_FILE, in the pipeline settings:

Here is the first pipeline:

codefresh-upload.yml

version: "1.0"

stages:
  - "upload"

steps:
  upload:
    title: "Uploading library jar to GS..."
    type: "freestyle"
    stage: "upload"
    arguments:
      image: "google/cloud-sdk:slim"
      commands:
        - echo $KEY_FILE | base64 --decode > key_file.json
        - gcloud auth activate-service-account --key-file=key_file.json
        - curl https://repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.2/scala-library-2.12.2.jar | gsutil cp - gs://anna-demo-bucket/scala-library-2.12.2.jar

This pipeline:

1. Uploads a JAR from Maven into our Google Storage bucket through a freestyle step.

Create the second pipeline

Our second pipeline has four stages:

• A stage for cloning the repository
• A stage for downloading the jar from the bucket
• A stage for building the image
• A stage for pushing the image to the repository

Here is the YAML for the second pipeline:

codefresh-download.yml

version: "1.0"

stages:
  - "clone"
  - "download"
  - "build"
  - "push"

steps:
  clone:
    title: "Cloning main repository..."
    type: "git-clone"
    stage: "clone"
    arguments:
      repo: "codefresh-contrib/gcloud-storage-sample-app"
      git: "github"
      revision: "master"
  download:
    title: "Downloading dependency lib from GS..."
    type: "freestyle"
    stage: "download"
    working_directory: ${{clone}}
    arguments:
      image: "google/cloud-sdk:slim"
      commands:
        - gsutil cp gs://anna-demo-bucket/scala-library-2.12.2.jar .
  build:
    title: "Building docker image..."
    type: "build"
    stage: "build"
    working_directory: ${{clone}}
    arguments:
      image_name: "annabaker/gcloud-storage-sample-app"
      tag: "master"
  push_to_my_registry:
    stage: "push"
    type: "push"
    title: "Pushing to external registry..."
    arguments:
      candidate: ${{build}}
      tag: '1.0.0'
      registry: "dockerhub"  

This pipeline does the following:

1. Clones the source code through a Git clone step.
2. Downloads the dependency JAR from our publicly-accessible Google Storage bucket through a freestyle step.
3. Builds a docker image through a build step.
4. Pushes the Docker image to the DockerHub registry you have integrated with Codefresh through a push step.

Related articles

CI/CD pipeline examples
Codefresh YAML for pipeline definitions