Oral  Brekke

Oral Brekke

1668846300

How to Faster CI Builds with Docker Layer Caching and BuildKit

This article takes a look at how to speed up your Docker-based builds on CircleCI, GitLab CI, and GitHub Actions with Docker layer Caching and BuildKit.

Docker Layer Caching

Docker caches each layer as an image is built, and each layer will only be re-built if it or the layer above it has changed since the last build. So, you can significantly speed up builds with Docker cache. Let's take a look at a quick example.

Dockerfile:

# pull base image
FROM python:3.9.7-slim

# install netcat
RUN apt-get update && \
    apt-get -y install netcat && \
    apt-get clean

# set working directory
WORKDIR /usr/src/app

# install requirements
COPY ./requirements.txt .
RUN pip install -r requirements.txt

# add app
COPY . .

# run server
CMD gunicorn -b 0.0.0.0:5000 manage:app

You can find the full source code for this project in the docker-ci-cache repo on GitHub.

The first Docker build can take several minutes to complete, depending on your connection speed. Subsequent builds should only take a few seconds since the layers get cached after that first build:

[+] Building 0.4s (12/12) FINISHED
 => [internal] load build definition from Dockerfile                                                                     0.0s
 => => transferring dockerfile: 37B                                                                                      0.0s
 => [internal] load .dockerignore                                                                                        0.0s
 => => transferring context: 35B                                                                                         0.0s
 => [internal] load metadata for docker.io/library/python:3.9.7-slim                                                     0.3s
 => [internal] load build context                                                                                        0.0s
 => => transferring context: 555B                                                                                        0.0s
 => [1/7] FROM docker.io/library/python:3.9.7-slim@sha256:bdefda2b80c5b4d993ef83d2445d81b2b894bf627b62bd7b0f01244de2b6a  0.0s
 => CACHED [2/7] RUN apt-get update &&     apt-get -y install netcat &&     apt-get clean                                0.0s
 => CACHED [3/7] WORKDIR /usr/src/app                                                                                    0.0s
 => CACHED [4/7] COPY ./requirements.txt .                                                                               0.0s
 => CACHED [5/7] RUN pip install -r requirements.txt                                                                     0.0s
 => CACHED [6/7] COPY project .                                                                                          0.0s
 => CACHED [7/7] COPY manage.py .                                                                                        0.0s
 => exporting to image                                                                                                   0.0s
 => => exporting layers                                                                                                  0.0s
 => => writing image sha256:2b8b7c5a6d1b77d5bcd689ab265b0281ad531bd2e34729cff82285f5abdcb59f                             0.0s
 => => naming to docker.io/library/cache                                                                                 0.0s

Even if you make a change to the source code it should still only take a few seconds to build as the dependencies will not need to be downloaded. Only the last two layers have to be re-built, in other words:

 => [6/7] COPY project .
 => [7/7] COPY manage.py .

To avoid invalidating the cache:

  1. Start your Dockerfile with commands that are less likely to change
  2. Place commands that are more likely to change (like COPY . .) as late as possible
  3. Add only the necessary files (use a .dockerignore file)

For more tips and best practices, check out the Docker Best Practices for Python Developers article.

BuildKit

If you're using a Docker version >= 19.03 you can use BuildKit, a container image builder, in place of the traditional image builder back-end inside the Docker engine. Without BuildKit, if an image doesn't exist on your local image registry, you would need to pull the remote images before building in order to take advantage of Docker layer caching.

Example:

$ docker pull mjhea0/docker-ci-cache:latest

$ docker docker build --tag mjhea0/docker-ci-cache:latest .

With BuildKit, you don't need to pull the remote images before building since it caches each build layer in your image registry. Then, when you build the image, each layer is downloaded as needed during the build.

To enable BuildKit, set the DOCKER_BUILDKIT environment variable to 1. Then, to turn on the inline layer caching, use the BUILDKIT_INLINE_CACHE build argument.

Example:

export DOCKER_BUILDKIT=1

# Build and cache image
$ docker build --tag mjhea0/docker-ci-cache:latest --build-arg BUILDKIT_INLINE_CACHE=1 .

# Build image from remote cache
$ docker build --cache-from mjhea0/docker-ci-cache:latest .

CI Environments

Since CI platforms provide a fresh environment for every build, you'll need to use a remote image registry as the source of the cache for BuildKit's layer caching.

Steps:

Log in to the image registry (like Docker Hub, Elastic Container Registry (ECR), and Quay, to name a few).

It's worth noting that both GitLab and GitHub have their own registries for use within your repositories (both public and private) on their platforms -- GitLab Container Registry and GitHub Packages, respectively.

Use Docker build's --cache-from option to use the existing image as the cache source.

  1. Push the new image to the registry if the build is successful.

Let's look at how to do this on CircleCI, GitLab CI, and GitHub Actions, using both single and multi-stage Docker builds with and without Docker Compose. Each of the examples use Docker Hub as the image registry with REGISTRY_USER and REGISTRY_PASS set as variables in the CI builds in order to push to and pull from the registry.

Make sure to set REGISTRY_USER and REGISTRY_PASS as environment variables in the build environment:

  1. CircleCI
  2. GitLab CI
  3. GitHub Actions

Single-stage Builds

CircleCI:

# _config-examples/single-stage/circle.yml

version: 2.1

jobs:
  build:
    machine:
      image: ubuntu-2004:202010-01
    environment:
      CACHE_IMAGE: mjhea0/docker-ci-cache
      DOCKER_BUILDKIT: 1
    steps:
      - checkout
      - run:
          name: Log in to docker hub
          command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS
      - run:
          name: Build from dockerfile
          command: |
            docker build \
              --cache-from $CACHE_IMAGE:latest \
              --tag $CACHE_IMAGE:latest \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              "."
      - run:
          name: Push to docker hub
          command: docker push $CACHE_IMAGE:latest

GitLab CI:

# _config-examples/single-stage/.gitlab-ci.yml

image: docker:stable
services:
  - docker:dind

variables:
  DOCKER_DRIVER: overlay2
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1

stages:
  - build

docker-build:
  stage: build
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker build
        --cache-from $CACHE_IMAGE:latest
        --tag $CACHE_IMAGE:latest
        --file ./Dockerfile
        --build-arg BUILDKIT_INLINE_CACHE=1
        "."
  after_script:
    - docker push $CACHE_IMAGE:latest

GitHub Actions:

# _config-examples/single-stage/github.yml

name: Docker Build

on: [push]

env:
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1

jobs:
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout master
        uses: actions/checkout@v1
      - name: Log in to docker hub
        run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }}
      - name: Build from dockerfile
        run: |
          docker build \
            --cache-from $CACHE_IMAGE:latest \
            --tag $CACHE_IMAGE:latest \
            --build-arg BUILDKIT_INLINE_CACHE=1 \
            "."
      - name: Push to docker hub
        run: docker push $CACHE_IMAGE:latest

Compose

If you're using Docker Compose, you can add the cache_from option to the compose file, which maps back to the docker build --cache-from <image> command when you run docker-compose build.

Example:

version: '3.8'

services:

  web:
    build:
      context: .
      cache_from:
        - mjhea0/docker-ci-cache:latest
    image: mjhea0/docker-ci-cache:latest

To take advantage of BuildKit, make sure you're using a version of Docker Compose >= 1.25.0. To enable BuildKit, set the DOCKER_BUILDKIT and COMPOSE_DOCKER_CLI_BUILD environment variables to 1. Then, again, to turn on the inline layer caching, use the BUILDKIT_INLINE_CACHE build argument.

CircleCI:

# _config-examples/single-stage/compose/circle.yml

version: 2.1

jobs:
  build:
    machine:
      image: ubuntu-2004:202010-01
    environment:
      CACHE_IMAGE: mjhea0/docker-ci-cache
      DOCKER_BUILDKIT: 1
      COMPOSE_DOCKER_CLI_BUILD: 1
    steps:
      - checkout
      - run:
          name: Log in to docker hub
          command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS
      - run:
          name: Build images
          command: docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1
      - run:
          name: Push to docker hub
          command: docker push $CACHE_IMAGE:latest

GitLab CI:

# _config-examples/single-stage/compose/.gitlab-ci.yml

image: docker/compose:latest
services:
  - docker:dind

variables:
  DOCKER_DRIVER: overlay2
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1

stages:
  - build

docker-build:
  stage: build
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1
  after_script:
    - docker push $CACHE_IMAGE:latest

GitHub Actions:

# _config-examples/single-stage/compose/github.yml

name: Docker Build

on: [push]

env:
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1

jobs:
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout master
        uses: actions/checkout@v1
      - name: Log in to docker hub
        run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }}
      - name: Build Docker images
        run: docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1
      - name: Push to docker hub
        run: docker push $CACHE_IMAGE:latest

Multi-stage Builds

With the multi-stage build pattern, you'll have to apply the same workflow (build, then push) for each intermediate stage since those images are discarded before the final image is created. The --target option can be used to build each stage of the multi-stage build separately.

Dockerfile.multi:

# base
FROM python:3.9.7 as base
COPY ./requirements.txt /
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /wheels -r requirements.txt

# stage
FROM python:3.9.7-slim
RUN apt-get update && \
    apt-get -y install netcat && \
    apt-get clean
WORKDIR /usr/src/app
COPY --from=base /wheels /wheels
COPY --from=base requirements.txt .
RUN pip install --no-cache /wheels/*
COPY . /usr/src/app
CMD gunicorn -b 0.0.0.0:5000 manage:app

CircleCI:

# _config-examples/multi-stage/circle.yml

version: 2.1

jobs:
  build:
    machine:
      image: ubuntu-2004:202010-01
    environment:
      CACHE_IMAGE: mjhea0/docker-ci-cache
      DOCKER_BUILDKIT: 1
    steps:
      - checkout
      - run:
          name: Log in to docker hub
          command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS
      - run:
          name: Build base from dockerfile
          command: |
            docker build \
              --target base \
              --cache-from $CACHE_IMAGE:base \
              --tag $CACHE_IMAGE:base \
              --file ./Dockerfile.multi \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              "."
      - run:
          name: Build stage from dockerfile
          command: |
            docker build \
              --cache-from $CACHE_IMAGE:base \
              --cache-from $CACHE_IMAGE:stage \
              --tag $CACHE_IMAGE:stage \
              --file ./Dockerfile.multi \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              "."
      - run:
          name: Push base image to docker hub
          command: docker push $CACHE_IMAGE:base
      - run:
          name: Push stage image to docker hub
          command: docker push $CACHE_IMAGE:stage

GitLab CI:

# _config-examples/multi-stage/.gitlab-ci.yml

image: docker:stable
services:
  - docker:dind


variables:
  DOCKER_DRIVER: overlay2
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1

stages:
  - build

docker-build:
  stage: build
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker build
        --target base
        --cache-from $CACHE_IMAGE:base
        --tag $CACHE_IMAGE:base
        --file ./Dockerfile.multi
        --build-arg BUILDKIT_INLINE_CACHE=1
        "."
    - docker build
        --cache-from $CACHE_IMAGE:base
        --cache-from $CACHE_IMAGE:stage
        --tag $CACHE_IMAGE:stage
        --file ./Dockerfile.multi
        --build-arg BUILDKIT_INLINE_CACHE=1
        "."
  after_script:
    - docker push $CACHE_IMAGE:stage

GitHub Actions:

# _config-examples/multi-stage/github.yml

name: Docker Build

on: [push]

env:
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1

jobs:
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout master
        uses: actions/checkout@v1
      - name: Log in to docker hub
        run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }}
      - name: Build base from dockerfile
        run: |
          docker build \
            --target base \
            --cache-from $CACHE_IMAGE:base \
            --tag $CACHE_IMAGE:base \
            --file ./Dockerfile.multi \
            --build-arg BUILDKIT_INLINE_CACHE=1 \
            "."
      - name: Build stage from dockerfile
        run: |
          docker build \
            --cache-from $CACHE_IMAGE:base \
            --cache-from $CACHE_IMAGE:stage \
            --tag $CACHE_IMAGE:stage \
            --file ./Dockerfile.multi \
            --build-arg BUILDKIT_INLINE_CACHE=1 \
            "."
      - name: Push base image to docker hub
        run: docker push $CACHE_IMAGE:base
      - name: Push stage image to docker hub
        run: docker push $CACHE_IMAGE:stage

Compose

Example compose file:

version: '3.8'

services:

  web:
    build:
      context: .
      cache_from:
        - mjhea0/docker-ci-cache:stage
    image: mjhea0/docker-ci-cache:stage

CircleCI:

# _config-examples/multi-stage/compose/circle.yml

version: 2.1

jobs:
  build:
    machine:
      image: ubuntu-2004:202010-01
    environment:
      CACHE_IMAGE: mjhea0/docker-ci-cache
      DOCKER_BUILDKIT: 1
      COMPOSE_DOCKER_CLI_BUILD: 1
    steps:
      - checkout
      - run:
          name: Log in to docker hub
          command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS
      - run:
          name: Build base from dockerfile
          command: |
            docker build \
              --target base \
              --cache-from $CACHE_IMAGE:base \
              --tag $CACHE_IMAGE:base \
              --file ./Dockerfile.multi \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              "."
      - run:
          name: Build Docker images
          command: docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1
      - run:
          name: Push base image to docker hub
          command: docker push $CACHE_IMAGE:base
      - run:
          name: Push stage image to docker hub
          command: docker push $CACHE_IMAGE:stage

GitLab CI:

# _config-examples/multi-stage/compose/.gitlab-ci.yml

image: docker/compose:latest
services:
  - docker:dind

variables:
  DOCKER_DRIVER: overlay
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1

stages:
  - build

docker-build:
  stage: build
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker build
        --target base
        --cache-from $CACHE_IMAGE:base
        --tag $CACHE_IMAGE:base
        --file ./Dockerfile.multi
        --build-arg BUILDKIT_INLINE_CACHE=1
        "."
    - docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1
  after_script:
    - docker push $CACHE_IMAGE:base
    - docker push $CACHE_IMAGE:stage

GitHub Actions:

# _config-examples/multi-stage/compose/github.yml

name: Docker Build

on: [push]

env:
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1

jobs:
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout master
        uses: actions/checkout@v1
      - name: Log in to docker hub
        run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }}
      - name: Build base from dockerfile
        run: |
          docker build \
            --target base \
            --cache-from $CACHE_IMAGE:base \
            --tag $CACHE_IMAGE:base \
            --file ./Dockerfile.multi \
            --build-arg BUILDKIT_INLINE_CACHE=1 \
            "."
      - name: Build images
        run: docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1
      - name: Push base image to docker hub
        run: docker push $CACHE_IMAGE:base
      - name: Push stage image to docker hub
        run: docker push $CACHE_IMAGE:stage

Conclusion

The caching strategies outlined in this article should work well for single-stage builds and multi-stage builds with two or three stages.

Each stage added to a build step requires a new build and push along with the addition of the --cache-from options for each parent stage. Thus, each new stage will add more clutter, making the CI file increasingly more difficult to read. Fortunately, BuildKit supports multi-stage builds with Docker layer caching built using a single stage. Review the following articles for more info on such advanced BuildKit patterns:

  1. Advanced Dockerfiles: Faster Builds and Smaller Images Using BuildKit and Multistage Builds
  2. Docker build cache sharing on multi-hosts with BuildKit and buildx
  3. Speed up multi-stage Docker builds in CI/CD with Buildkit’s registry cache

Finally, it's important to note that while caching may speed up your CI builds, you should re-build your images without cache from time to time in order to download the latest OS patches and security updates. For more on this, review this thread.

--

The code can be found in the docker-ci-cache repo:

  1. Single-stage examples
  2. Multi-stage examples

Cheers!

Original article source at: https://testdriven.io/

#docker #caching 

What is GEEK

Buddha Community

How to Faster CI Builds with Docker Layer Caching and BuildKit

Faster CI Builds with Docker Layer Caching and BuildKit

Docker Layer Caching

Docker caches each layer as an image is built, and each layer will only be re-built if it or the layer above it has changed since the last build. So, you can significantly speed up builds with Docker cache. Let’s take a look at a quick example.

Dockerfile:

## pull base image
FROM python:3.8.3-slim

## install netcat
RUN apt-get update && \
    apt-get -y install netcat && \
    apt-get clean

## set working directory
WORKDIR /usr/src/app

## install requirements
COPY ./requirements.txt .
RUN pip install -r requirements.txt

## add app
COPY . .

## run server
CMD gunicorn -b 0.0.0.0:5000 manage:app

You can find the full source code for this project in the  docker-ci-cache repo on GitHub.

The first Docker build can take several minutes to complete, depending on your connection speed. Subsequent builds should only take a few seconds since the layers get cached after that first build:

Sending build context to Docker daemon  721.4kB
Step 1/7 : FROM python:3.8.3-slim
 ---> 091e4db1c781
Step 2/7 : RUN apt-get update &&     apt-get -y install netcat &&     apt-get clean
 ---> Using cache
 ---> 74f2b1dca14e
Step 3/7 : WORKDIR /usr/src/app
 ---> Using cache
 ---> 0afd94dba053
Step 4/7 : COPY ./requirements.txt .
 ---> Using cache
 ---> 7249c15cbee8
Step 5/7 : RUN pip install -r requirements.txt
 ---> Using cache
 ---> 13bd0d323961
Step 6/7 : COPY . .
 ---> Using cache
 ---> 1bc421036d94
Step 7/7 : CMD gunicorn -b 0.0.0.0:5000 manage:app
 ---> Using cache
 ---> 18df0fe6ba3f
Successfully built 18df0fe6ba3f

Even if you make a change to the source code it should still only take a few seconds to build as the dependencies will not need to be downloaded. Only the last two layers have to be re-built, in other words:

Step 6/7 : COPY . .
 ---> f99ca1babd63
Step 7/7 : CMD gunicorn -b 0.0.0.0:5000 manage:app
 ---> Running in 0054b494cdc6

To avoid invalidating the cache:

  1. Start your Dockerfile with commands that are less likely to change
  2. Place commands that are more likely to change (like COPY . .) as late as possible
  3. Add only the necessary files (use a .dockerignore file)

#docker #caching #buildkit #travis #circleci #gitlab

Oral  Brekke

Oral Brekke

1668846300

How to Faster CI Builds with Docker Layer Caching and BuildKit

This article takes a look at how to speed up your Docker-based builds on CircleCI, GitLab CI, and GitHub Actions with Docker layer Caching and BuildKit.

Docker Layer Caching

Docker caches each layer as an image is built, and each layer will only be re-built if it or the layer above it has changed since the last build. So, you can significantly speed up builds with Docker cache. Let's take a look at a quick example.

Dockerfile:

# pull base image
FROM python:3.9.7-slim

# install netcat
RUN apt-get update && \
    apt-get -y install netcat && \
    apt-get clean

# set working directory
WORKDIR /usr/src/app

# install requirements
COPY ./requirements.txt .
RUN pip install -r requirements.txt

# add app
COPY . .

# run server
CMD gunicorn -b 0.0.0.0:5000 manage:app

You can find the full source code for this project in the docker-ci-cache repo on GitHub.

The first Docker build can take several minutes to complete, depending on your connection speed. Subsequent builds should only take a few seconds since the layers get cached after that first build:

[+] Building 0.4s (12/12) FINISHED
 => [internal] load build definition from Dockerfile                                                                     0.0s
 => => transferring dockerfile: 37B                                                                                      0.0s
 => [internal] load .dockerignore                                                                                        0.0s
 => => transferring context: 35B                                                                                         0.0s
 => [internal] load metadata for docker.io/library/python:3.9.7-slim                                                     0.3s
 => [internal] load build context                                                                                        0.0s
 => => transferring context: 555B                                                                                        0.0s
 => [1/7] FROM docker.io/library/python:3.9.7-slim@sha256:bdefda2b80c5b4d993ef83d2445d81b2b894bf627b62bd7b0f01244de2b6a  0.0s
 => CACHED [2/7] RUN apt-get update &&     apt-get -y install netcat &&     apt-get clean                                0.0s
 => CACHED [3/7] WORKDIR /usr/src/app                                                                                    0.0s
 => CACHED [4/7] COPY ./requirements.txt .                                                                               0.0s
 => CACHED [5/7] RUN pip install -r requirements.txt                                                                     0.0s
 => CACHED [6/7] COPY project .                                                                                          0.0s
 => CACHED [7/7] COPY manage.py .                                                                                        0.0s
 => exporting to image                                                                                                   0.0s
 => => exporting layers                                                                                                  0.0s
 => => writing image sha256:2b8b7c5a6d1b77d5bcd689ab265b0281ad531bd2e34729cff82285f5abdcb59f                             0.0s
 => => naming to docker.io/library/cache                                                                                 0.0s

Even if you make a change to the source code it should still only take a few seconds to build as the dependencies will not need to be downloaded. Only the last two layers have to be re-built, in other words:

 => [6/7] COPY project .
 => [7/7] COPY manage.py .

To avoid invalidating the cache:

  1. Start your Dockerfile with commands that are less likely to change
  2. Place commands that are more likely to change (like COPY . .) as late as possible
  3. Add only the necessary files (use a .dockerignore file)

For more tips and best practices, check out the Docker Best Practices for Python Developers article.

BuildKit

If you're using a Docker version >= 19.03 you can use BuildKit, a container image builder, in place of the traditional image builder back-end inside the Docker engine. Without BuildKit, if an image doesn't exist on your local image registry, you would need to pull the remote images before building in order to take advantage of Docker layer caching.

Example:

$ docker pull mjhea0/docker-ci-cache:latest

$ docker docker build --tag mjhea0/docker-ci-cache:latest .

With BuildKit, you don't need to pull the remote images before building since it caches each build layer in your image registry. Then, when you build the image, each layer is downloaded as needed during the build.

To enable BuildKit, set the DOCKER_BUILDKIT environment variable to 1. Then, to turn on the inline layer caching, use the BUILDKIT_INLINE_CACHE build argument.

Example:

export DOCKER_BUILDKIT=1

# Build and cache image
$ docker build --tag mjhea0/docker-ci-cache:latest --build-arg BUILDKIT_INLINE_CACHE=1 .

# Build image from remote cache
$ docker build --cache-from mjhea0/docker-ci-cache:latest .

CI Environments

Since CI platforms provide a fresh environment for every build, you'll need to use a remote image registry as the source of the cache for BuildKit's layer caching.

Steps:

Log in to the image registry (like Docker Hub, Elastic Container Registry (ECR), and Quay, to name a few).

It's worth noting that both GitLab and GitHub have their own registries for use within your repositories (both public and private) on their platforms -- GitLab Container Registry and GitHub Packages, respectively.

Use Docker build's --cache-from option to use the existing image as the cache source.

  1. Push the new image to the registry if the build is successful.

Let's look at how to do this on CircleCI, GitLab CI, and GitHub Actions, using both single and multi-stage Docker builds with and without Docker Compose. Each of the examples use Docker Hub as the image registry with REGISTRY_USER and REGISTRY_PASS set as variables in the CI builds in order to push to and pull from the registry.

Make sure to set REGISTRY_USER and REGISTRY_PASS as environment variables in the build environment:

  1. CircleCI
  2. GitLab CI
  3. GitHub Actions

Single-stage Builds

CircleCI:

# _config-examples/single-stage/circle.yml

version: 2.1

jobs:
  build:
    machine:
      image: ubuntu-2004:202010-01
    environment:
      CACHE_IMAGE: mjhea0/docker-ci-cache
      DOCKER_BUILDKIT: 1
    steps:
      - checkout
      - run:
          name: Log in to docker hub
          command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS
      - run:
          name: Build from dockerfile
          command: |
            docker build \
              --cache-from $CACHE_IMAGE:latest \
              --tag $CACHE_IMAGE:latest \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              "."
      - run:
          name: Push to docker hub
          command: docker push $CACHE_IMAGE:latest

GitLab CI:

# _config-examples/single-stage/.gitlab-ci.yml

image: docker:stable
services:
  - docker:dind

variables:
  DOCKER_DRIVER: overlay2
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1

stages:
  - build

docker-build:
  stage: build
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker build
        --cache-from $CACHE_IMAGE:latest
        --tag $CACHE_IMAGE:latest
        --file ./Dockerfile
        --build-arg BUILDKIT_INLINE_CACHE=1
        "."
  after_script:
    - docker push $CACHE_IMAGE:latest

GitHub Actions:

# _config-examples/single-stage/github.yml

name: Docker Build

on: [push]

env:
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1

jobs:
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout master
        uses: actions/checkout@v1
      - name: Log in to docker hub
        run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }}
      - name: Build from dockerfile
        run: |
          docker build \
            --cache-from $CACHE_IMAGE:latest \
            --tag $CACHE_IMAGE:latest \
            --build-arg BUILDKIT_INLINE_CACHE=1 \
            "."
      - name: Push to docker hub
        run: docker push $CACHE_IMAGE:latest

Compose

If you're using Docker Compose, you can add the cache_from option to the compose file, which maps back to the docker build --cache-from <image> command when you run docker-compose build.

Example:

version: '3.8'

services:

  web:
    build:
      context: .
      cache_from:
        - mjhea0/docker-ci-cache:latest
    image: mjhea0/docker-ci-cache:latest

To take advantage of BuildKit, make sure you're using a version of Docker Compose >= 1.25.0. To enable BuildKit, set the DOCKER_BUILDKIT and COMPOSE_DOCKER_CLI_BUILD environment variables to 1. Then, again, to turn on the inline layer caching, use the BUILDKIT_INLINE_CACHE build argument.

CircleCI:

# _config-examples/single-stage/compose/circle.yml

version: 2.1

jobs:
  build:
    machine:
      image: ubuntu-2004:202010-01
    environment:
      CACHE_IMAGE: mjhea0/docker-ci-cache
      DOCKER_BUILDKIT: 1
      COMPOSE_DOCKER_CLI_BUILD: 1
    steps:
      - checkout
      - run:
          name: Log in to docker hub
          command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS
      - run:
          name: Build images
          command: docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1
      - run:
          name: Push to docker hub
          command: docker push $CACHE_IMAGE:latest

GitLab CI:

# _config-examples/single-stage/compose/.gitlab-ci.yml

image: docker/compose:latest
services:
  - docker:dind

variables:
  DOCKER_DRIVER: overlay2
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1

stages:
  - build

docker-build:
  stage: build
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1
  after_script:
    - docker push $CACHE_IMAGE:latest

GitHub Actions:

# _config-examples/single-stage/compose/github.yml

name: Docker Build

on: [push]

env:
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1

jobs:
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout master
        uses: actions/checkout@v1
      - name: Log in to docker hub
        run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }}
      - name: Build Docker images
        run: docker-compose build --build-arg BUILDKIT_INLINE_CACHE=1
      - name: Push to docker hub
        run: docker push $CACHE_IMAGE:latest

Multi-stage Builds

With the multi-stage build pattern, you'll have to apply the same workflow (build, then push) for each intermediate stage since those images are discarded before the final image is created. The --target option can be used to build each stage of the multi-stage build separately.

Dockerfile.multi:

# base
FROM python:3.9.7 as base
COPY ./requirements.txt /
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /wheels -r requirements.txt

# stage
FROM python:3.9.7-slim
RUN apt-get update && \
    apt-get -y install netcat && \
    apt-get clean
WORKDIR /usr/src/app
COPY --from=base /wheels /wheels
COPY --from=base requirements.txt .
RUN pip install --no-cache /wheels/*
COPY . /usr/src/app
CMD gunicorn -b 0.0.0.0:5000 manage:app

CircleCI:

# _config-examples/multi-stage/circle.yml

version: 2.1

jobs:
  build:
    machine:
      image: ubuntu-2004:202010-01
    environment:
      CACHE_IMAGE: mjhea0/docker-ci-cache
      DOCKER_BUILDKIT: 1
    steps:
      - checkout
      - run:
          name: Log in to docker hub
          command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS
      - run:
          name: Build base from dockerfile
          command: |
            docker build \
              --target base \
              --cache-from $CACHE_IMAGE:base \
              --tag $CACHE_IMAGE:base \
              --file ./Dockerfile.multi \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              "."
      - run:
          name: Build stage from dockerfile
          command: |
            docker build \
              --cache-from $CACHE_IMAGE:base \
              --cache-from $CACHE_IMAGE:stage \
              --tag $CACHE_IMAGE:stage \
              --file ./Dockerfile.multi \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              "."
      - run:
          name: Push base image to docker hub
          command: docker push $CACHE_IMAGE:base
      - run:
          name: Push stage image to docker hub
          command: docker push $CACHE_IMAGE:stage

GitLab CI:

# _config-examples/multi-stage/.gitlab-ci.yml

image: docker:stable
services:
  - docker:dind


variables:
  DOCKER_DRIVER: overlay2
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1

stages:
  - build

docker-build:
  stage: build
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker build
        --target base
        --cache-from $CACHE_IMAGE:base
        --tag $CACHE_IMAGE:base
        --file ./Dockerfile.multi
        --build-arg BUILDKIT_INLINE_CACHE=1
        "."
    - docker build
        --cache-from $CACHE_IMAGE:base
        --cache-from $CACHE_IMAGE:stage
        --tag $CACHE_IMAGE:stage
        --file ./Dockerfile.multi
        --build-arg BUILDKIT_INLINE_CACHE=1
        "."
  after_script:
    - docker push $CACHE_IMAGE:stage

GitHub Actions:

# _config-examples/multi-stage/github.yml

name: Docker Build

on: [push]

env:
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1

jobs:
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout master
        uses: actions/checkout@v1
      - name: Log in to docker hub
        run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }}
      - name: Build base from dockerfile
        run: |
          docker build \
            --target base \
            --cache-from $CACHE_IMAGE:base \
            --tag $CACHE_IMAGE:base \
            --file ./Dockerfile.multi \
            --build-arg BUILDKIT_INLINE_CACHE=1 \
            "."
      - name: Build stage from dockerfile
        run: |
          docker build \
            --cache-from $CACHE_IMAGE:base \
            --cache-from $CACHE_IMAGE:stage \
            --tag $CACHE_IMAGE:stage \
            --file ./Dockerfile.multi \
            --build-arg BUILDKIT_INLINE_CACHE=1 \
            "."
      - name: Push base image to docker hub
        run: docker push $CACHE_IMAGE:base
      - name: Push stage image to docker hub
        run: docker push $CACHE_IMAGE:stage

Compose

Example compose file:

version: '3.8'

services:

  web:
    build:
      context: .
      cache_from:
        - mjhea0/docker-ci-cache:stage
    image: mjhea0/docker-ci-cache:stage

CircleCI:

# _config-examples/multi-stage/compose/circle.yml

version: 2.1

jobs:
  build:
    machine:
      image: ubuntu-2004:202010-01
    environment:
      CACHE_IMAGE: mjhea0/docker-ci-cache
      DOCKER_BUILDKIT: 1
      COMPOSE_DOCKER_CLI_BUILD: 1
    steps:
      - checkout
      - run:
          name: Log in to docker hub
          command: docker login -u $REGISTRY_USER -p $REGISTRY_PASS
      - run:
          name: Build base from dockerfile
          command: |
            docker build \
              --target base \
              --cache-from $CACHE_IMAGE:base \
              --tag $CACHE_IMAGE:base \
              --file ./Dockerfile.multi \
              --build-arg BUILDKIT_INLINE_CACHE=1 \
              "."
      - run:
          name: Build Docker images
          command: docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1
      - run:
          name: Push base image to docker hub
          command: docker push $CACHE_IMAGE:base
      - run:
          name: Push stage image to docker hub
          command: docker push $CACHE_IMAGE:stage

GitLab CI:

# _config-examples/multi-stage/compose/.gitlab-ci.yml

image: docker/compose:latest
services:
  - docker:dind

variables:
  DOCKER_DRIVER: overlay
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1

stages:
  - build

docker-build:
  stage: build
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker build
        --target base
        --cache-from $CACHE_IMAGE:base
        --tag $CACHE_IMAGE:base
        --file ./Dockerfile.multi
        --build-arg BUILDKIT_INLINE_CACHE=1
        "."
    - docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1
  after_script:
    - docker push $CACHE_IMAGE:base
    - docker push $CACHE_IMAGE:stage

GitHub Actions:

# _config-examples/multi-stage/compose/github.yml

name: Docker Build

on: [push]

env:
  CACHE_IMAGE: mjhea0/docker-ci-cache
  DOCKER_BUILDKIT: 1
  COMPOSE_DOCKER_CLI_BUILD: 1

jobs:
  build:
    name: Build Docker Image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout master
        uses: actions/checkout@v1
      - name: Log in to docker hub
        run: docker login -u ${{ secrets.REGISTRY_USER }} -p ${{ secrets.REGISTRY_PASS }}
      - name: Build base from dockerfile
        run: |
          docker build \
            --target base \
            --cache-from $CACHE_IMAGE:base \
            --tag $CACHE_IMAGE:base \
            --file ./Dockerfile.multi \
            --build-arg BUILDKIT_INLINE_CACHE=1 \
            "."
      - name: Build images
        run: docker-compose -f docker-compose.multi.yml build --build-arg BUILDKIT_INLINE_CACHE=1
      - name: Push base image to docker hub
        run: docker push $CACHE_IMAGE:base
      - name: Push stage image to docker hub
        run: docker push $CACHE_IMAGE:stage

Conclusion

The caching strategies outlined in this article should work well for single-stage builds and multi-stage builds with two or three stages.

Each stage added to a build step requires a new build and push along with the addition of the --cache-from options for each parent stage. Thus, each new stage will add more clutter, making the CI file increasingly more difficult to read. Fortunately, BuildKit supports multi-stage builds with Docker layer caching built using a single stage. Review the following articles for more info on such advanced BuildKit patterns:

  1. Advanced Dockerfiles: Faster Builds and Smaller Images Using BuildKit and Multistage Builds
  2. Docker build cache sharing on multi-hosts with BuildKit and buildx
  3. Speed up multi-stage Docker builds in CI/CD with Buildkit’s registry cache

Finally, it's important to note that while caching may speed up your CI builds, you should re-build your images without cache from time to time in order to download the latest OS patches and security updates. For more on this, review this thread.

--

The code can be found in the docker-ci-cache repo:

  1. Single-stage examples
  2. Multi-stage examples

Cheers!

Original article source at: https://testdriven.io/

#docker #caching 

Iliana  Welch

Iliana Welch

1595249460

Docker Explained: Docker Architecture | Docker Registries

Following the second video about Docker basics, in this video, I explain Docker architecture and explain the different building blocks of the docker engine; docker client, API, Docker Daemon. I also explain what a docker registry is and I finish the video with a demo explaining and illustrating how to use Docker hub

In this video lesson you will learn:

  • What is Docker Host
  • What is Docker Engine
  • Learn about Docker Architecture
  • Learn about Docker client and Docker Daemon
  • Docker Hub and Registries
  • Simple demo to understand using images from registries

#docker #docker hub #docker host #docker engine #docker architecture #api

Beata Zubek

1572946450

WordPress in Docker. Part 1: Dockerization

This entry-level guide will tell you why and how to Dockerize your WordPress projects.

#wordpress #docker #dockerization #dockerize #ci/cd

Iliana  Welch

Iliana Welch

1597368540

Docker Tutorial for Beginners 8 - Build and Run C++ Applications in a Docker Container

Docker is an open platform that allows use package, develop, run, and ship software applications in different environments using containers.
In this course We will learn How to Write Dockerfiles, Working with the Docker Toolbox, How to Work with the Docker Machine, How to Use Docker Compose to fire up multiple containers, How to Work with Docker Kinematic, Push images to Docker Hub, Pull images from a Docker Registery, Push stacks of servers to Docker Hub.
How to install Docker on Mac.

#docker tutorial #c++ #docker container #docker #docker hub #devopstools