Friday, May 13, 2016

Ubuntu 16.04 and Docker 1.11 - Accessing secured private registry

After trying to access a private registry that is secured via ssl, and adding the certificate authority (ca.pem) under

  /etc/docker/certs.d/<IP of registry>

Where my IP is 10.68.230.7, the pull/push requests still failed.

$ docker pull 10.68.230.7/alpine
Using default tag: latest
Error response from daemon: Get https://10.68.230.7/v1/_ping: x509: certificate signed by unknown authority

It seems docker does not like certificates with the ca.pem file name.  To fix this rename the ca.pem file to ca.crt.

$ cp /etc/docker/certs.d/10.68.230.7/ca.pem /etc/docker/certs.d/10.68.230.7/ca.crt

Then it works.

$ docker pull 10.68.230.7/alpine
Using default tag: latest
latest: Pulling from alpine
d0ca440e8637: Already exists 
Digest: sha256:5c826f3f0f5c34aca4df43360ec0faef6326b18bd311309cc8ae3a83f799d1eb
Status: Downloaded newer image for 10.68.230.7/alpine:latest

Ubuntu 16.04, systemd and Docker

Ubuntu 16.04 LTS is now available.  After having made the switch from 14.04 without really looking at the changes except for the kernel number (4.x), I was pleasantly surprised by the fact that it now uses systemd.

Trying to setup docker to pull/push from a private registry using security, I first attempted to change the logging level to debug by adding -D in /etc/default/docker and after restarting docker noticed that no "debug" logs were shown.

This took me a while to find out but /etc/default/docker is not used anymore.

This fact is in the /etc/default/docker file but easy to miss:

# Docker Upstart and SysVinit configuration file
#
# THIS FILE DOES NOT APPLY TO SYSTEMD
#
#   Please see the documentation for "systemd drop-ins":
#   https://docs.docker.com/engine/articles/systemd/
#
# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"
# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"
# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"
# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"

Also, one can find out where a service file and configuration is located:

$ systemctl show --property=FragmentPath docker
FragmentPath=/usr/lib/systemd/system/docker.service
$ grep EnvironmentFile /usr/lib/systemd/system/docker.service
grep: /usr/lib/systemd/system/docker.service: No such file or directory

What the above tells us is that the docker service has no configuration file at the moment.

There are different ways to configure services in systemd.  The option described below is OK but deviates from a standard systemd setup in the config location as, since this is Ubuntu,  /etc/default path is used instead of /etc/sysconfig.

The first step to use a config file is to add the required informarion  to the /lib/systemd/system/docker.service file by adding an EnvironmentFile attribute in the [Service] section:

$ vi /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket
[Service]
Type=notify
# see https://docs.docker.com/engine/admin/systemd
EnvironmentFile=-/etc/default/docker
ExecStart=/usr/bin/docker daemon -H fd:// $DOCKER_OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $BLOCK_REGISTRY \
          $INSECURE_REGISTRY
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
[Install]
WantedBy=multi-user.target


Now when the docker service is restarted it will use /etc/default/docker as its environment file, and pull the environment variables from it.

As such, the final step is to add the matching options in /etc/default/docker:
# Docker Upstart and SysVinit configuration file

# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"

# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"

# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"

# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"
INSECURE_REGISTRY=""
DOCKER_STORAGE_OPTIONS=""
DOCKER_NETWORK_OPTIONS=""
BLOCK_REGISTRY=""

It should work.