Securing Your Container Images Using the Codefresh OIDC Provider and Keyless Signing

Deploying software to your internal systems or sending releases to external customers is a process that ideally follows strong security requirements in an end-to-end manner.

In a perfect world, every software release should come with at least the following guarantees

• That it was indeed created by the claimed author/organization 
• That it wasn’t tampered after its initial creation

In the world of CI/CD we are mostly interested in binary artifacts and how they were created all the way from the initial release up  until they reach production.  

We are glad to announce that Codefresh is now an official OIDC provider for the Sigstore project and can be now used as an identity provider for artifact signing. This means that you now have access to a much easier, more transparent and secure process for signing and verifying artifacts built in Codefresh pipelines.

In this article we will briefly discuss the concepts behind keyless signing and the advantages keyless signing has over signing artifacts with traditional self managed long lived keys. Then we’ll dive into signing artifacts with just a few lines of yaml in Codefresh pipelines and learn how to securely verify the artifacts and the identity of the signing pipeline.

The traditional way of securing release artifacts

Signing artifacts is not a new item on the agenda for software delivery teams. Whether software is used internally or vendored, it’s crucial to sign executable artifacts in order to verify their origin and that they have not been tampered with by potentially malicious actors. 

The cosign binary by the sigstore project has long been the goto solution for signing and verifying artifacts, especially when it comes to container images. Before the introduction of keyless signing, asymmetric cryptographic keys were used for signing. 

The vendor or owner of the software delivery pipeline would generate a keypair, sign the artifacts with the private key and publish the public key to be used for verification. 

This is how signing of a container image would look with pre-provisioned keys:

The basic security principles behind this technique are sound and battle-tested. However the management of this solution comes with  several limitations.

The most important point in this traditional solution is the  requirement to self-manage the keypair. This poses a series of security risks and operational challenges:

• Securely storing the private key while providing access to various pipelines used for signing is a delicate balance to get right. 
• Key rotation is a challenge as artifacts that were signed by the old key cannot be verified by the new key. Having a permanent key which is not rotated for prolonged periods of time greatly increases the risk of leaking the key.
• Key distribution to end users is not standardized – each software vendor has their way of distributing public keys to customers – some publish it on their website, some put it in a README file in a Git repository etc. This makes automatic verification extremely difficult as a user must locate the relevant public key for each artifact.
• Signer identity granularity is challenging to achieve – if for example we want to make sure a specific pipeline has signed an artifact we’d have to maintain a separate keypair for said pipeline. Given all the challenges stated above this would be very hard to maintain.

All the challenges have forced several organizations to treat artifact security as a secondary requirement or even just an afterthought.

The new way –  Keyless signing

The sigstore project has recognized all the challenges mentioned in the previous section and decided to tackle them with the keyless signing procedure. As the name implies there are no keypairs any more in the process – or more truthfully no keys must be created and managed manually.

The new process is implemented with the OpenID connect protocol. In this process anybody who wants to secure an artifact does the following:

1. The organization/author must first become an OIDC provider
2. An ID token is created for the provider
3. A short-lived cryptographic keypair is created based on the ID token
4. The artifact is signed as usual
5. The certificate (public key) along with signature/digest are packaged and sent to an append-only transparency log
6. Any external organization can verify the artifact using the information provided in the transparency log.

The key point here is that there are no long-lived keypairs. And nobody actually creates key-pairs by hand anymore. The transparency log also presents a unified interface to all clients that want to verify the artifact.

• Cosign remains the client used for executing signing and verification. It is still possible to sign and verify artifacts with self-provisioned and maintained keypairs, but the keyless procedure is now the default.
• Fulcio is the sigstore certificate authority that is responsible for verifying id tokens against OIDC providers and issuing the short-lived certificates based on those tokens.
• Rekor is the append-only transparency log storing the signatures and certificates.  
• Fulcio and Rekor are hosted publicly by the Sigstore project; they are open source and can also be self hosted if needed

What we announce today is that Codefresh is now an official OIDC provider. This means that anybody can now verify container images as created by Codefresh pipelines, using a standardized way.

This flow chart illustrates the keyless signature process with the Codefresh OIDC provider and the Sigstore project:  

Now that we have explained the overview of the whole architecture, let’s see how we can apply it to Codefresh pipelines.

Securing container images with Codefresh pipelines

Let’s take the simplest Codefresh pipeline possible. The classic workflow where we checkout the source code and create a container image:

The first two steps are the familiar clone and build steps. We have enhanced this pipeline with a dedicated stage for signing the created container. 

Here is the YAML definition of the pipeline.

version: "1.0"
stages:
  - "main-clone"
  - "build"
  - "cosign-sign"
steps:
  main-clone:
      title: Cloning main repository...
      stage: main-clone
      type: git-clone
      repo: 'codefresh-contrib/python-flask-sample-app'
      revision: 'master'
  build:
    title: Building Docker Image
    type: build
    stage: build
    image_name: ilmedcodefresh/python-flask-sample-app
    working_directory: ./python-flask-sample-app
    tag: 'cosign-keyless-demo'
    dockerfile: Dockerfile 
 
  obtain_id_token:
    stage: "cosign-sign"
    title: Obtain ID Token
    type: obtain-oidc-id-token
    arguments:
      AUDIENCE: 'sigstore'

  cosign-sign:
    title: "keyless signing"
    type: "freestyle
    image: "alpine:latest"
    working_directory: "/tmp"
    stage: "cosign-sign"
    commands:
      - |
        apk add --update cosign
        export SIGSTORE_ID_TOKEN=$ID_TOKEN
        cosign sign --oidc-issuer "https://oidc.codefresh.io" --yes=true --registry-username ${DOCKERHUB_USERNAME} --registry-password ${DOCKERHUB_PASSWORD} docker.io/ilmedcodefresh/python-flask-sample-app:cosign-demo

The signature process consists of the last 2 steps:

Step 1 – Obtaining the identity token from the OIDC provider and setting the audience of the token to “sigstore”. This step exports the identity token to an environment variable named “ID_TOKEN”

obtain_id_token:
    stage: "cosign-sign"
    title: Obtain ID Token
    type: obtain-oidc-id-token
    arguments:
      AUDIENCE: 'sigstore'

Step 2 – Executing cosign with the OIDC issuer set to “https://oidc.codefresh.io” and the environment variable SIGSTORE_ID_TOKEN containing the identity token.

This part also requires authentication to the docker registry as the signature is pushed to the registry.

cosign-sign:
    title: "keyless signing"
    type: "freestyle
    image: "alpine:latest"
    working_directory: "/tmp"
    stage: "cosign-sign"
    commands:
      - |
        apk add --update cosign
        export SIGSTORE_ID_TOKEN=$ID_TOKEN
        cosign sign --oidc-issuer "https://oidc.codefresh.io" --yes=true --registry-username ${DOCKERHUB_USERNAME} --registry-password ${DOCKERHUB_PASSWORD} docker.io/ilmedcodefresh/python-flask-sample-app:cosign-keyless-demo

This is all!

Behind the scenes cosign is orchestrating the whole signature process – it requests a certificate and id token verification from Fulcio (the certificate authority), pushes the signature to the registry and pushes the signature, digest and certificate to Rekor (the append only transparency log).

Verifying artifacts signed with Codefresh pipelines

We have seen how a Codefresh pipeline can sign and secure a container image it creates in the previous section.

Let’s see now the rest of the process from the point of view of the consumer which is typically an organization that wants to deploy this container image.

During the artifact verification process, we seek answers to two fundamental questions: firstly, whether the artifact bears a signature, and secondly, the identity of the signer.

The identity part is the interesting bit here, as it highly depends on the security requirements. Some users may trust all artifacts that are signed by any Codefresh pipeline, others may trust artifacts that are signed by a specific account in Codefresh while others may only trust artifacts that are signed by a specific pipeline.

The command structure to verify a container image built in Codefresh utilizing the Coefresh OIDC provider is as follows – 

cosign verify --certificate-oidc-issuer "https://oidc.codefresh.io" --certificate-identity/--certificate-identity-regexp "<certificate identity>" <image tag>

We can use the certificate identity to verify the origin of the artifact. The certificate identity for artifacts that originate in Codefresh pipelines is constructed as follows:

<Codefresh platform URL (https://g.codefresh.io)>/<account name>/<project name>/<pipeline name>:<account id>/<pipeline id>

The reason we include the account id and the pipeline id is because account names and pipeline names in Codefresh are mutable, and stricter security requirements may state that an immutable identity is required for successful verification.

Either the full identity can be used for verifications (--certificate-identity) or regular expressions (--certificate-identity–regexp) to flexibly mix and match various parts of the identity.

Let’s look at the example pipeline:

The pipeline was executed from an account named `ilia-codefresh` , it is part of `sigstore-blog` project and it’s named `keyless-signature`

The pipeline id is: `6612ad6384ef42f18019b7e9`

And the account id is `628a80b693a15c0f9c13ab75` (To get your account id go to https://g.codefresh.io/2.0/account-settings/account-information):

We can now use that information to verify the identity of the artifact signer:

To verify on the full identity we would run the following command:

cosign verify --certificate-oidc-issuer "https://oidc.codefresh.io" --certificate-identity "https://g.codefresh.io/ilia-codefresh/sigstore-blog/keyless-signature:628a80b693a15c0f9c13ab75/6612ad6384ef42f18019b7e9" ilmedcodefresh/python-flask-sample-app:cosign-keyless-demo

To verify the artifact was signed by a any pipelines in a specific immutable account in Codefresh we would use the account-id with a regexp:

cosign verify --certificate-oidc-issuer "https://oidc.codefresh.io" --certificate-identity-regexp "https://g.codefresh.io/.*:628a80b693a15c0f9c13ab75/.*" ilmedcodefresh/python-flask-sample-app:cosign-keyless-demo

Similarly we can use the a regular expression to verify the artifact was signed by a specific account name:

cosign verify --certificate-oidc-issuer "https://oidc.codefresh.io" --certificate-identity-regexp "https://g.codefresh.io/ilia-codefresh/.*" ilmedcodefresh/python-flask-sample-app:cosign-keyless-demo

When the identity doesn’t match the signing certificate identity cosign will exit with an error – let’s try to set a different identity –

cosign verify --certificate-oidc-issuer "https://oidc.codefresh.io" --certificate-identity-regexp "https://g.codefresh.io/the-wrong-account/.*" ilmedcodefresh/python-flask-sample-app:cosign-keyless-demo

On successful verification cosign will display some useful information about the signature

Such as the digest and the base64 encoded content of the signing certificate.

Summary

We hope that with this article you have seen how easy you can sign and verify container images using Codefresh pipelines. The hard work is already taken care of by the Sigstore project and all that remains is for you to enhance your Codefresh pipelines with OIDC steps. 

Even if your organization doesn’t have strict legal requirements, securing your artifacts (and also knowing where they came from) is a fundamental process that should be treated with the appropriate attention.

Learn more:

• How keyless signing works – the official Sigstore docs: https://www.sigstore.dev/how-it-works 
• Codefresh OIDC provider usage in pipelines: https://codefresh.io/docs/docs/integrations/oidc-pipelines