Run a Docker container with its dependencies inside a pipeline
The composition step runs a Docker Composition as a means to execute finite commands in a more complex interaction of services.
Note that while composition steps are still supported, the recommended way to run integrations tests going forward is with service containers.
Motivation for Compositions
The primary purpose of compositions is to run integration tests or any kind of tests that require multiple services for their execution.
The syntax offered by Codefresh closely follows the syntax for Docker-compose files, so if you already know how Docker compose works, you will be immediately familiar with Codefresh compositions.
The big difference between the two, is that Codefresh is distinguishing between two kinds of services:
- Composition Services
- Composition Candidates
Composition services are helper services that are needed for the tests to run. These can be a database, a queue, a cache, or the backend docker image of your application.
Composition candidates are special services that will execute the tests. Codefresh will monitor their execution and fail the build if they do not succeed. Almost always composition candidates are Docker images that contain unit/integration tests or other kinds of tests (e.g. performance)
You need at least one composition service and one candidate for the composition step.
Here is an example of a composition. There is one composition service (PostgreSQL database) and one candidate (tests executed with gulp)
The most important part is the
command line that executes the tests. If it fails, then the whole composition step will fail.
||The free-text display name of the step.||Optional|
||A basic, free-text description of the step.||Optional|
||Parent group of this step. See using stages for more information.||Optional|
||The directory in which to search for the composition file. It can be an explicit path in the container’s file system, or a variable that references another step. The default is
||The composition you want to run. It can be an inline YAML definition, a path to a composition file on the file system, or the logical name of a composition stored in the Codefresh system. We support most features of Docker compose version 2.0 and 3.0||Required|
||Version for docker compose. Use
||The definition of the service to monitor. Each candidate has a single
||environment that will be accessible to the container||Optional|
||defines the working directory that will be used in a service before running a command. By default it is defined by the docker image that is used by the service.||Optional|
||Extra volumes for individual services. Used for transferring information between your steps. Explained in detail later in this page.||Optional|
||A set of environment variables to substitute in the composition. Notice that these variables are docker-compose variables and NOT environment variables||Optional|
||If a step fails, and the process is halted. The default value is
||Define a set of conditions which need to be satisfied in order to execute this step.
You can find more information in the Conditional Execution of Steps article.
||Define operations to perform upon step completion using a set of predefined Post-Step Operations.||Optional|
||Define retry behavior as described in Retrying a step.||Optional|
Composition versus Composition Candidates
For Codefresh to determine if the step and operations were successfully executed, you must specify at least one
composition_candidate is a single service component of the normal Docker composition that is monitored for a successful exit code and determines the outcome of the step. During runtime, the
composition_candidate is merged into the specified
compositionand is monitored for successful execution.
The critical part of each candidate is the
command parameter. This takes a single command that will
be executed inside the Docker container of the candidate and will decide if the whole composition is successful or not. Only one command is allowed (similar to Docker compose). If you wish to test multiple commands you need to connect them with
&& like this.
Working directories in a composition
By default, all services that take part in a composition will use as working directory the one defined by the respective image. If you want to change that, you need to use the
working_dir parameter at the service level.
Here is an example:
If you run this composition, you will see in the logs that the alpine image will use
/tmp as a working directory and the python one will use
my_service_1 | /tmp my_test_service_1 | /root
The networking in Codefresh compositions works just like normal Docker-compose. Each service is assigned a hostname that matches its name and is accessible by other services.
Here is an example
In this composition the MySql instance will be available at host
db:3306 accessible from the node image. When the node tests run, they will be pointed to that host and port combination to access it.
Notice also that like docker compose the order that the services are launched is not guaranteed. A quick way to solve this issue is with a sleep statement like shown above. This will make sure that the database is truly up before the tests run.
A better approach would be to use solutions such as wait-for-it which are much more robust. Here is an example:
In this composition a Java application is launched at
app:8080 and then a second image is used for integration tests that target that URL (passed as a parameter to Maven).
wait-for-it.sh script will make sure that the Java application is truly up before the tests are started. Notice that in the example above the script is included in the testing image (created by
Accessing your project folder from a composition
By default, the services of a composition run in a completely isolated manner. There are several scenarios however where you wish to access your Git files such as:
- Using test data that is available in the project folder
- Preloading a database with a data script found in Git
- Running integration tests and then using their results for reporting
Here is an example where the shared volume is mounted in a composition.
In this pipeline:
- The first freestyle step writes a simple test file in the shared volume.
- The composition starts and both services (
my_unit_tests) attach the same volume.
- The sample service reads from the shared volume (i.e. using test data that was created before).
- The sample unit test service writes to the shared volume (emulating test results).
- The last freestyle step reads the file that was written by the composition.
Therefore, in this pipeline you can see both ways of data sharing, bringing files into a composition and getting results out of it. Notice that we need to mount the shared volume only in the composition services. The freestyle steps automatically mount
/codefresh/volume on their own.
Note: In order to mount the shared volume in one of your composition services, you must mount it in the
composition_candidatealso. It is not compulsory to mount the shared volume in all services of a composition. Only those that actually use it for file transfer, should mount it.
Composition variables versus environment variables
Docker compose supports two kinds of variables in its syntax:
- There are environment variables that are used in the docker-compose file itself (
- There are environment variables that are passed in containers (
Codefresh supports both kinds, but notice that variables mentioned in the
composition_variables yaml group refer to the first kind. Any variables defined there are NOT passed automatically to containers (use the
environment yaml group for that purpose).
This can be illustrated with the following example:
If you run the compositio,n you will see that the
printenv command shows the following:
test_service_1 | FIRST_KEY=VALUE
FIRST_KEY variable which is defined explicitly in the
environment yaml part is correctly passed to the alpine container. The
ANOTHER_KEY is not visible in the container at all.
You should use the
composition_variables yaml group for variables that you wish to reuse in other parts of your composition using the
composition already contains a service with the same name as the
composition_candidate, the two service definitions are combined, with preference given to the
For example, we create a new Codefresh composition named ‘test_composition’:
Now we want to reuse this composition during our build for testing purposes.
We can add the following composition step to our
codefresh.yml file and define the composition step so that
test_service always uses the latest image that was built.
In the above example, both
composition_candidates define a service named
test_service. After merging these definitions,
test_service will maintain the
command that was defined in the original composition but will refer to the image built by the step named