Lab: Distribution and Trust
Time: Approximately 30 minutes
This lab focuses on understanding and securing image distribution. We’ll
start with a simple
docker pull and build up to using Docker Content Trust (DCT).
You will complete the following steps as part of this lab.
- Step 1 - Pulling images by tag
- Step 2 - Pulling images by digest
- Step 3 - Docker Content Trust
- Step 4 - Official Images
- Step 5 - Extra for experts
You will need all of the following to complete this lab:
- A Docker Host running Docker 1.10 or higher (preferably Docker 1.12 or higher).
- This lab uses environment variables which can be clunky when working with
sudo. Therefore none of the Docker commands in this particular lab are preceded with
Step 1: Pulling images by tag
The most common and basic way to pull Docker images is by
tag. The is where you specify an image name followed by an alphanumeric tag. The image name and tag are separated by a colon
:. For example:
$ docker pull alpine:edge
This command will pull the Alpine image tagged as
edge. The corresponding image can be found here on Docker Store.
If no tag is specified, Docker will pull the image with the
Pull the Alpine image with the
$ docker pull alpine:edge edge: Pulling from library/alpine e587fa4f6e1f: Pull complete Digest: sha256:e5ab6f0941eb01c41595d35856f16215021a941e9893501d632ed4c0ee4e53a6 Status: Downloaded newer image for alpine:edge
Confirm that the pull was successful.
$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE alpine edge 3fc33d6d5e74 4 weeks ago 4.799 MB
Run a new container from the image and ping www.docker.com.
$ docker run --rm -it alpine:edge sh / # ping www.docker.com PING www.docker.com (126.96.36.199): 56 data bytes 64 bytes from 188.8.131.52: seq=0 ttl=40 time=94.424 ms 64 bytes from 184.108.40.206: seq=1 ttl=40 time=94.549 ms 64 bytes from 220.127.116.11: seq=2 ttl=40 time=94.455 ms ^C
Exit the container.
In this step you pulled an image by tag and verified the pull by running a container from it.
Step 2: Pulling images by digest
Pulling by tag is easy and convenient. However, tags are mutable, and the same tag can refer to different images over time. For example, you can add updates to an image and push the updated image using the same tag as a previous version of the image. This scenario where a single tag points to multiple versions of an image can lead to bugs and vulnerabilities in your production environments.
This is why pulling by digest is such a powerful operation. Thanks to the content-addressable storage model used by Docker images, we can target pulls to specific image contents by pulling by digest. In this step you’ll see how to pull by digest.
- Pull the Alpine image with the
$ docker pull alpine@sha256:b7233dafbed64e3738630b69382a8b231726aa1014ccaabc1947c5308a8910a7 sha256:b7233dafbed64e3738630b69382a8b231726aa1014ccaabc1947c5308a8910a7: Pulling from library/alpine 6589332ef57c: Pull complete Digest: sha256:b7233dafbed64e3738630b69382a8b231726aa1014ccaabc1947c5308a8910a7 Status: Downloaded newer image for alpine@sha256:b7233dafbed64e3738630b69382a8b231726aa1014ccaabc1947c5308a8910a7
Check that the pull succeeded.
$ docker images --digests alpine REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE alpine edge <none> 3fc33d6d5e74 4 weeks ago 4.799 MB alpine <none> sha256:b7233dafbed64e3738630b69382a8b231726aa1014ccaabc1947c5308a8910a7 79c898d40088 7 weeks ago 4.799 MB
Notice that there are now two Alpine images in the Docker Hosts local repository. One lists the
edgetag. The other lists
<none>as the tag along with the
The content addressable storage model used by Docker images means that any changes made to an image will result in a new digest for the updated image. This means it is not possible for two images with different contents to have the same digest.
In this step you learned how to pull images by digest and list image digests using the
docker images command. For more information about the
docker pull command, see the documentation.
Step 3: Docker Content Trust
It’s not easy to find the digest of a particular image tag. This is because it is computed from the hash of the image contents and stored in the image manifest. The image manifest is then stored in the Registry. This is why we needed a
docker pull by tag to find digests previously. It would also be desirable to have additional security guarantees such as image freshness.
Enter Docker Content Trust: a system currently in the Docker Engine that verifies the publisher of images without sacrificing usability. Docker Content Trust implements The Update Framework (TUF), an NSF-funded research project succeeding Thandy of the Tor project. TUF uses a key hierarchy to ensure recoverable key compromise and robust freshness guarantees.
Under the hood, Docker Content Trust handles name resolution from IMAGE tags to IMAGE digests by signing its own metadata – when Content Trust is enabled, docker will verify the signatures and expiration dates in the metadata before rewriting a pull by tag command to a pull by digest.
In this step you will enable Docker Content Trust, sign images as you push them, and pull signed and unsigned images.
- Enable Docker Content Trust by setting the DOCKER_CONTENT_TRUST environment variable.
$ export DOCKER_CONTENT_TRUST=1
Note: If you are using
sudowith your Docker commands, you will need to preceded the above command so that it looks like this
sudo export DOCKER_CONTENT_TRUST=1
It is worth nothing that although Docker Content Trust is now enabled, all Docker commands remain the same. Docker Content Trust will work silently in the background.
$ docker pull riyaz/dockercon:trust Pull (1 of 1): riyaz/dockercon:trust@sha256:88a7163227a54bf0343aae9e7a4404fdcdcfef8cc777daf9686714f4376ede46 sha256:88a7163227a54bf0343aae9e7a4404fdcdcfef8cc777daf9686714f4376ede46: Pulling from riyaz/dockercon fae91920dcd4: Pull complete Digest: sha256:88a7163227a54bf0343aae9e7a4404fdcdcfef8cc777daf9686714f4376ede46 Status: Downloaded newer image for riyaz/dockercon@sha256:88a7163227a54bf0343aae9e7a4404fdcdcfef8cc777daf9686714f4376ede46 Tagging riyaz/dockercon@sha256:88a7163227a54bf0343aae9e7a4404fdcdcfef8cc777daf9686714f4376ede46 as riyaz/dockercon:trust
Look closely at the output of the
docker pullcommand and take particular notice of the name translation - how the command is translated to the digest as shown below:
Pull (1 of 1): riyaz/dockercon:trust@sha256:88a7163227a54bf0343aae9e7a4404fdcdcfef8cc777daf9686714f4376ede46
Pull and unsigned image.
$ docker pull riyaz/dockercon:untrusted No trust data for untrusted
You cannot pull unsigned images with Docker Content Trust enabled. Once Docker Content Trust is enabled you can only pull, run, or build with trusted images.
- Tag and push your own signed image with Docker Content Trust.
$ docker tag alpine:edge <your-docker-id>/alpine:trusted $ docker push <your-docker-id>/alpine:trusted The push refers to a repository [docker.io/nigelpoulton/alpine] 4fe15f8d0ae6: Pushed trusted: digest: sha256:dc89ce8401da81f24f7ba3f0ab2914ed9013608bdba0b7e7e5d964817067dc06 size: 528 Signing and pushing trust metadata You are about to create a new root signing key passphrase. This passphrase will be used to protect the most sensitive key in your signing system. Please choose a long, complex passphrase and be careful to keep the password and the key file itself secure and backed up. It is highly recommended that you use a password manager to generate the passphrase and keep it safe. There will be no way to recover this key. You can find the key in your config directory. Enter passphrase for new root key with ID fef644e: Repeat passphrase for new root key with ID fef644e: Enter passphrase for new repository key with ID b4fd76d (docker.io/nigelpoulton/alpine): Repeat passphrase for new repository key with ID b4fd76d (docker.io/nigelpoulton/alpine): Finished initializing "docker.io/nigelpoulton/alpine" Successfully signed "docker.io/nigelpoulton/alpine":trusted
This command will prompt you for passphrases. This is because Docker Content Trust is generating a hierarchy of keys with different signing roles. Each key is encrypted with a passphrase, and it is best practice is to provide different passphrases for each key.
The root key is the most important key in TUF as it can rotate any other key in the system. The root key should be kept offline, ideally in hardware crypto device. It is stored in
~/.docker/trust/private/root_keys by default.
The tagging key is the only local key required to push new tags to an existing repo, and is stored in
~/.docker/trust/private/tuf_keys by default.
Feel free to explore the
~/.docker/trust directory to view the internal metadata and key information that Docker Content Trust generates.
Disable Docker Content Trust.
$ export DOCKER_CONTENT_TRUST=0
Pull the image you just pushed in the previous step.
$ docker pull <your-docker-id>/alpine:trusted trusted: Pulling from nigelpoulton/alpine Digest: sha256:dc89ce8401da81f24f7ba3f0ab2914ed9013608bdba0b7e7e5d964817067dc06 Status: Image is up to date for nigelpoulton/alpine:trusted
Enable Docker Content Trust.
$ export DOCKER_CONTENT_TRUST=1
Pull the same image again.
$ docker pull <your-docker-id>/alpine:trusted Pull (1 of 1): nigelpoulton/alpine:trusted@sha256:dc89ce8401da81f24f7ba3f0ab2914ed9013608bdba0b7e7e5d964817067dc06 sha256:dc89ce8401da81f24f7ba3f0ab2914ed9013608bdba0b7e7e5d964817067dc06: Pulling from nigelpoulton/alpine Digest: sha256:dc89ce8401da81f24f7ba3f0ab2914ed9013608bdba0b7e7e5d964817067dc06 Status: Downloaded newer image for nigelpoulton/alpine@sha256:dc89ce8401da81f24f7ba3f0ab2914ed9013608bdba0b7e7e5d964817067dc06 Tagging nigelpoulton/alpine@sha256:dc89ce8401da81f24f7ba3f0ab2914ed9013608bdba0b7e7e5d964817067dc06 as nigelpoulton/alpine:trusted
Take note of the difference between the pull of a signed image with Docker Content Trust enabled and disabled. With Docker Content Trust enabled the image pull is converted from a tagged image pull to a digest image pull.
In this step you have seen how to enable and disable Docker Content Trust. You have also seen how to sign images that you push. For more information about Docker Content Trust, see the documentation.
Step 4: Official images
All images in Docker Hub under the
library organization (currently viewable at: https://hub.docker.com/explore/)
are deemed “Official Images.” These images undergo a rigorous, open-source
review process to ensure they follow best practices. These best practices include signing, being lean, and having clearly written Dockerfiles. For these reasons, it is strongly recommended that you use official images whenever possible.
Official images can be pulled with just their name and tag. You do not have to precede the image name with
library/ or any other repository name.
Step 5: Extra for Experts
Docker Content Trust is powered by Notary, an open-source TUF-client and server that can operate over arbitrary trusted collections of data. Notary has its own CLI with robust features such as the ability to rotate keys and remove trust data. In this step you will play with the Notary CLI and a local instance of the Notary server instead of the one deployed alongside Docker Hub.
- Get a notary client.
This can be done by downloading a binary directly from the releases page, or by cloning the notary repository into a valid Go repository structure (instructions at the end of the README) and building a client by running
make binaries. If you build the notary binary yourself, it will be placed in the
bin subdirectory within the notary git repo directory.
- Use the notary client to inspect the existing Alpine repository on Docker Hub.
$ notary -s https://notary.docker.io -d ~/.docker/trust list docker.io/library/alpine NAME DIGEST SIZE (BYTES) ROLE ------------------------------------------------------------------------------------------------------ 2.6 e9cec9aec697d8b9d450edd32860ecd363f2f3174c8338beb5f809422d182c63 1374 targets 2.7 9f08005dff552038f0ad2f46b8e65ff3d25641747d3912e3ea8da6785046561a 1374 targets 3.1 0796cca706c64170c29cfefbdd67f32e25dab2247fc31956c86773dae825800f 506 targets 3.2 9c6c40abb6a9180603068a413deca450ef13c381974b392a25af948ca87c3c14 506 targets 3.3 4fa633f4feff6a8f02acfc7424efd5cb3e76686ed3218abf4ca0fa4a2a358423 506 targets 3.4 3dcdb92d7432d56604d4545cbd324b14e647b313626d99b889d0626de158f73a 506 targets edge e5ab6f0941eb01c41595d35856f16215021a941e9893501d632ed4c0ee4e53a6 506 targets latest 3dcdb92d7432d56604d4545cbd324b14e647b313626d99b889d0626de158f73a 506 targets
docker.io/ must be prepended to the image name for hub images. You should also try your own image you pushed to hub earlier!
- If you haven’t already, clone the Notary repository.
$ git clone https://github.com/docker/notary.git Cloning into 'notary'... remote: Counting objects: 17156, done. remote: Compressing objects: 100% (26/26), done. remote: Total 17156 (delta 5), reused 0 (delta 0), pack-reused 17129 Receiving objects: 100% (17156/17156), 27.07 MiB | 11.20 MiB/s, done. Resolving deltas: 100% (9734/9734), done. Checking connectivity... done.
Bring up a local notary server and signer.
You will need to run this command from inside the root folder of the notary repository. The operation may take a minute or two to complete.
$ docker-compose up -d Pulling mysql (mariadb:10.1.10)... 10.1.10: Pulling from library/mariadb 03e1855d4f31: Pull complete a3ed95caeb02: Pull complete ea9cb3d7d346: Pull complete e47839e262bb: Pull complete f568a56c1fd0: Pull complete cc98c1dfbf81: Pull complete 98a99d2efdc4: Pull complete 0b304232c8e6: Pull complete d65a44f4573e: Pull complete Digest: sha256:10d0179f08a4fb0c785142ca73367921f46a93c2ee7c84831ae3543522156a6c Status: Downloaded newer image for mariadb:10.1.10 <SNIP> Creating notary_mysql_1 Creating notary_signer_1 Creating notary_server_1
/etc/hosts(or if using
$(docker-machine ip) notary-server).
Copy the config and certificate required to talk to your local Notary server.
$ mkdir -p ~/.notary && cp cmd/notary/config.json cmd/notary/root-ca.crt ~/.notary
Initialize a new trusted collection on your local server
$ notary init example.com/scripts No root keys found. Generating a new root key... You are about to create a new root signing key passphrase. This passphrase will be used to protect the most sensitive key in your signing system. Please choose a long, complex passphrase and be careful to keep the password and the key file itself secure and backed up. It is highly recommended that you use a password manager to generate the passphrase and keep it safe. There will be no way to recover this key. You can find the key in your config directory. Enter passphrase for new root key with ID 6f3b2d0: Repeat passphrase for new root key with ID 6f3b2d0: Enter passphrase for new targets key with ID 4bc290d (example.com/scripts): Repeat passphrase for new targets key with ID 4bc290d (example.com/scripts):
You will be prompted for passphrases for the root and repository keys. This is for the same reasons as it was when you first pushed a repo with Docker Content Trust enabled.
- Add content to your trusted collection by running a sequence of
notary add example.com/scripts <NAME> <FILE>,
notary publish example.com/scripts, and
notary list example.com/scripts. This is a the sequence, in order:
- Stages adding a target with
- Attempts to publish this target to the server with
- Fetches and displays all trust data for example.com (verifying output as it is downloaded)
- Stages adding a target with
To remove targets, you can make use of the
notary remove example.com/scripts <NAME> command, followed by a
notary publish example.com/scripts.
Please note that this
docker-compose setup will persist notary’s database data in a volume; if you’d like to wipe out all state when running this lab running multiple times, you will have to remove this volume in addition to restarting the notary server, signer, and database containers.
Key rotation with the notary client
Let’s look at key rotation with Notary.
In the event of key-compromise, it’s a simple procedure to rotate keys with notary. Simply determine which key you wish to rotate, and run
notary key rotate.
List all of the keys in your notary repository
$ notary key list ROLE GUN KEY ID LOCATION ------------------------------------------------------------------------------------------------------------------------------------ root 6f3b2d0269343cafafca59f05d738389a27c9408cbe1345188325913620a6b80 file (/root/.notary/private) snapshot example.com/scripts 7c9487f2d2a42d1f91020285483d15b5d85d8e22799c64141c5cafc87997795c file (/root/.notary/private) targets example.com/scripts 4bc290d883fa33020a528abbce61accf78745f4f28be243f3cc9f44f11297258 file (/root/.notary/private)
Rotate a key.
In this example we’ll rotate the
$ notary key rotate example.com/scripts targets Enter passphrase for new targets key with ID a4413c0 (example.com/scripts): Repeat passphrase for new targets key with ID a4413c0 (example.com/scripts): Enter passphrase for root key with ID 6f3b2d0: Enter passphrase for snapshot key with ID 7c9487f (example.com/scripts):
Confirm that the targets key ID has changed.
$ notary key list ROLE GUN KEY ID LOCATION ------------------------------------------------------------------------------------------------------------------------------------ root 6f3b2d0269343cafafca59f05d738389a27c9408cbe1345188325913620a6b80 file (/root/.notary/private) snapshot example.com/scripts 7c9487f2d2a42d1f91020285483d15b5d85d8e22799c64141c5cafc87997795c file (/root/.notary/private) targets example.com/scripts a4413c080788746ac34db2077a0bcc41be84dfce34d8a7d2d55942d125e4c69f file (/root/.notary/private)
For more information about keys, please see the this section of the Notary Service Architecture documentation
Role delegation with notary
Now we’ll explore delegation roles in notary. Delegation roles are a subset of the targets role, and are ideal for assigning signing privileges to collaborators and CI systems because no private key sharing is required. Here’s a demo of setting up delegation roles to illustrate the steps below:
Rotate the snapshot key to the server.
This is done by default when creating new Content Trust repositories in Docker 1.10+.
$ notary key rotate example.com/scripts snapshot -r Enter passphrase for root key with ID 6f3b2d0:
This is so that delegation roles will only require their own delegation’s private key to publish to trusted collections.
Have your delegate generate an x509 certificate + private key pair with openssl - instructions here.
Retrieve their certificate,
Add their delegation role.
$ notary delegation add example.com/scripts targets/releases delegation.crt --all-paths
This command will allow the collaborator to push any target (from
--all-paths) to the
targets/releasesrole if they can sign with their private key
delegation.keyin order to produce a valid signature that can be verified by
delegation.crt’s public key material.
Be aware that this commmand only stages the delegation role addition.
Publish the addition of the delegation role
$ notary publish example.com/scripts
Check that the delegation role was added.
$ notary delegation list example.com/scripts
Your collaborator should now be able to publish content (
docker push) with Docker Content Trust enabled, or with a
notary add example.com/scripts <NAME> <FILE> -r targets/releases. You can verify their pushes by running a
notary list example.com/scripts -r targets/releases
You can add additional keys to the same role with additional
delegation add commands, like so:
notary delegation add example.com/scripts targets/releases delegation2.crt delegation3.crt, followed by a publish
For more commands over delegation roles, please consult the notary advanced usage documentation.
Congratulations. You now know the advantages of image digests over image tags. You have also seen how to enable Docker Content Trust and push signed images. Finally you have seen how to perform some advanced tasks using the notary server and client.