Protecting your keys in public Docker containers

Access to the mirrors and caches in StableBuild are limited through API keys which are listed directly in your Dockerfile. For example, here your API key is both used in FROM and set as an ARG which is used by sb-apt.sh to authenticate with the package registry.

FROM stablebuild-my-secret-api-key.dockermirror.stablebuild.com/ubuntu:focal-20231003

ARG SB_API_KEY=stablebuild-my-secret-api-key
ARG APT_PIN_DATE=2023-11-10T10:40:01Z

COPY ./sb-apt.sh /opt/sb-apt.sh
RUN bash /opt/sb-apt.sh load-apt-sources ubuntu

RUN apt update && apt install -y curl

It's important to keep these keys private, as these keys are used to bill your account for traffic. If you see weird traffic spikes, check the raw usage logs to see if a key might be leaked. You can rotate keys via Dashboard > Keys.

Finding these keys in your Docker image

The approach to list keys in your Dockerfile is fine if you build Docker containers for internal use, but problematic if you want to publish your Docker containers publicly (in Docker Hub, or another public registry). Anyone pulling the container can see the layers that make up the container, and your API key will be listed in there. In addition, the build can write files to your container containing your API key. For instance: in the example above the key can be seen both when you inspect the layers, and its persisted on file system.

To view this, first build the container above via:

docker build -t unsecure-demo .

And then inspect the layers via dive to see that the key is present:

dive unsecure-demo

You can also find the key on the file system. Run the container with an interactive shell via:

docker run --rm -it unsecure-demo bash

And you can find your key persisted on disk via:

grep -rio --exclude-dir={ece,pytorch,sys,proc} 'stablebuild-my-secret-api-key' /
find / -name "*stablebuild-my-secret-api-key*"

Protecting your keys

Let's protect these keys. First, our base image. The FROM line:

FROM stablebuild-my-secret-api-key.dockermirror.stablebuild.com/ubuntu:focal-20231003

Is not leaked to any layers (visible via dive) or on the file system; but is in the image metadata. To fix this you can create a multi-stage Docker image instead:

FROM stablebuild-my-secret-api-key.dockermirror.stablebuild.com/ubuntu:focal-20231003 AS base
FROM scratch
COPY --from=base / /

Prevent key leakage through layers

Next, let's tackle the key leaking through the layers:

ARG SB_API_KEY=stablebuild-my-secret-api-key

Instead of hard-coding the key you can use Docker build secrets. Secrets are mounted in at build time, and are not included in the final image. Here you can use:

FROM stablebuild-my-secret-api-key.dockermirror.stablebuild.com/ubuntu:focal-20231003 AS base
FROM scratch
COPY --from=base / /

ARG APT_PIN_DATE=2023-11-10T10:40:01Z

COPY ./sb-apt.sh /opt/sb-apt.sh

# We mount secret "sb-api-key" to file "/kaniko/sb-api-key.txt"
# And then we load the content of the file into the env variable SB_API_KEY
RUN --mount=type=secret,id=sb-api-key,target=/kaniko/sb-api-key.txt \
    SB_API_KEY=$(cat /kaniko/sb-api-key.txt) bash /opt/sb-apt.sh load-apt-sources ubuntu

RUN apt update && apt install -y curl

If you build this container via:

export SB_API_KEY=stablebuild-my-secret-api-key
docker build -t demo-build-secrets --secret id=sb-api-key,env=SB_API_KEY .

And then inspect the image again via dive:

dive demo-build-secrets

The hard-coded key is gone:

Prevent key leakage on file system

To avoid leaking the keys through the file system, you'll need to clean up any file where the key is persisted in the same step as where you use the key - otherwise someone can inspect the file system at a specific layer and recover your key that way.

From inspecting the file system earlier we found that the key is listed in /etc/apt/sources.list (written by sb-apt.sh) and that the key is present in the file names of files at /var/lib/apt/lists/ (apt cache files). We can rewrite our Dockerfile to:

FROM stablebuild-my-secret-api-key.dockermirror.stablebuild.com/ubuntu:focal-20231003 AS base
FROM scratch
COPY --from=base / /

ARG APT_PIN_DATE=2023-11-10T10:40:01Z

COPY ./sb-apt.sh /opt/sb-apt.sh

# Backup the original /etc/apt/sources.list (will restore later)
RUN cp /etc/apt/sources.list /etc/apt/sources.list.bak

# 1. We mount secret "sb-api-key" to file "/kaniko/sb-api-key.txt"
# 2. Load the content of the file into the env variable SB_API_KEY, and run `sb-apt.sh` to load the sources -
#    this creates '/etc/apt/sources.list' with the key on FS
# 3. Do the apt update && apt install in one run command (so /etc/apt/sources.list is not persisted in a layer)
# 4. Copy the original sources.list back
# 5. Remove cached files from apt which have our key in /var/lib/apt/lists/
RUN --mount=type=secret,id=sb-api-key,target=/kaniko/sb-api-key.txt \
    SB_API_KEY=$(cat /kaniko/sb-api-key.txt) bash /opt/sb-apt.sh load-apt-sources ubuntu && \
    apt update && \
    apt install -y curl && \
    cp /etc/apt/sources.list.bak /etc/apt/sources.list && \
    rm -r /var/lib/apt/lists/

When you build this container via:

export SB_API_KEY=stablebuild-my-secret-api-key
docker build -t demo-clean-fs --secret id=sb-api-key,env=SB_API_KEY .

And then run the container and search the file system, there are no results anymore:

$ docker run --rm -it demo-clean-fs bash
root@012e063d3d45:/# grep -rio --exclude-dir={ece,pytorch,sys,proc} 'stablebuild-my-secret-api-key' /
# no results
root@012e063d3d45:/# find / -name "*stablebuild-my-secret-api-key*"
# no results

Final check, inspecting the exported image

As a final check we can export the full image (including all layers and metadata) to disk, and do a final scan for our API key:

$ mkdir -p out && \
    docker save -o out.tar demo-clean-fs && \
    mkdir -p out && \
    tar -xvf out.tar -C out/
    
$ cd out
$ grep stablebuild-my-secret-api-key -R .
# no results!

Great 🎉! Your container now no longer contains your key, and you can safely push this container to Docker Hub or another public registry.

The above works for the Ubuntu package registry, but if you use other mirrors or caches this will require some manual work in inspecting the layers and file system to ensure no keys are leaked. Both dive and just opening an interactive shell into the container should make this relatively painless though.

Using build secrets with Kaniko

If you use Kaniko to build your containers then build secrets are not available. However, you can use the fact that the /kaniko folder on your file system is shared between the Kaniko container and the build process. So you can use this in your Dockerfile:

RUN --mount=type=secret,id=sb-api-key,target=/kaniko/sb-api-key.txt \
    SB_API_KEY=$(cat /kaniko/sb-api-key.txt) bash /opt/sb-apt.sh load-apt-sources ubuntu

Here, the target /kaniko/sb-api-key.txt is on a shared file system, so you can just write the key to this location inside the Kaniko container before calling /kaniko/executor:

echo -n "stablebuild-my-secret-api-key" > /kaniko/sb-api-key.txt && \
    /kaniko/executor --dockerfile=./Dockerfile ...

You can do this f.e. by setting the entrypoint of the gcr.io/kaniko-project/executor container to sh, and the arguments to [ "-c", "echo -n 'stablebuild-my-secret-api-key' > /kaniko/sb-api-key.txt && /kaniko/executor --dockerfile=./Dockerfile" ].

Last updated