Saturday, September 17, 2016

Managing Docker containers using Terraform on Windows 10

It started with evaluating Apcera, a container management platform, at work.  While Apcera "can" run on BareOS(ie one needs to install Ubuntu 14.04, and run some scripts), it is primarily made to run on top of an IaaS provider like OpenStack or AWS. After successfully installing Apcera on BareOS (see https://github.com/baboune/UnofficialApceraDoc/), my next goal was to set it up on OpenStack to troubleshoot some cloud-init issue that a colleague was experiencing.

When installing towards an IaaS provider, Apcera relies on Terraform. But it only supports Terraform 6.16, and not the latest version 0.7.3  and 0.7 is a major revision change with breaking compatibility changes.  Needless to say that this was not in the Apcera documentation and that a certain time was lost figuring out why the provided terraform files were failing (parsing errors, and Integer values are not supported (issue 6254)).

So, I decided to take a look at Terraform over the week end, and learn about it a little.

Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. It can manage existing and popular service providers as well as custom in-house solutions. It is a infrastructure as code, execution plan, resource graph, and change automation tool.  It stops where Configuration Management tools like Puppet, Ansible and Chef starts.

One of the supported providers by Terraform is Docker. And, a priori, it looked simple enough to try and make a few terraform learnings.  It is quick and local to my Windows 10 setup. So, pressing Windows key -> Docker Quickstart Terminal launches the Docker default VM (VirtualBox provider) via docker-machine.

The next step is to create a project, and to attempt to launch a simple Ubuntu container as per the tutorial (see https://www.terraform.io/docs/providers/docker/index.html).

In this example tutorial, the terraform source looks like this:

# Configure the Docker provider
provider "docker" {
    host = "tcp://127.0.0.1:1234/"
}

# Create a container
resource "docker_container" "foo" {
    image = "${docker_image.ubuntu.latest}"
    name = "foo"
}

resource "docker_image" "ubuntu" {
    name = "ubuntu:latest"
}

the above needs to be saved in a "*.tf" file, e.g. example.tf in a local directory. Note that ".tf" is the terraform file extension and all files found within a directory with this extension are automatically included when running a terraform command.

There I faced a first problem, which IP and protocol to use in the provider section?  I knew that the IP is the Docker VirtualBox VM (192.168.99.100) but not the port, and the protocol should be http as the Remote Docker API is REST based.

Finding this information is easy. In the Quick Start console, simply type docker-machine config.

baboune MINGW64 ~
$ docker-machine config
--tlsverify
--tlscacert="C:\\Users\\baboune\\.docker\\machine\\certs\\ca.pem"
--tlscert="C:\\Users\\baboune\\.docker\\machine\\certs\\cert.pem"
--tlskey="C:\\Users\\baboune\\.docker\\machine\\certs\\key.pem"
--H=tcp://192.168.99.100:2376

Thus the host IP/protocol is tcp://192.168.99.100:2376.

But after updating the example.tf file with that information, and trying a terraform plan command, I still got a malformed HTTP response error (second problem).

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.

Error refreshing state: 1 error(s) occurred:
* Error pinging Docker server: Get http://192.168.99.100:2376/_ping: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

A quick Google search indicates no matching results for my Docker setup (Docker Toolbox, recent docker version).  There are a few similar issues (here) but those are using boot2docker which has been deprecated in favor of docker-machine many releases ago. Also, a quick docker info will show that I am running release 1.12.1.

A netstat on the VM shows that port 2376 is open and associated with the docker server:

$ nestat -anp | grep 2376
tcp           0       0         :::2376        :::*        LISTEN        2662/dockerd

And a curl to the remote Docker API returns no visible results:

$ curl -XGET http://192.168.99.100:2376/_ping

$

But works.

It is at this point that I make the connection with the certificates directory listed by the docker-machine config command, and the Docker Remote API documentation.  Looking back to the terraform Docker documentation, it says:

The following arguments are supported:
  • host - (Required) This is the address to the Docker host. If this is blank, the DOCKER_HOST environment variable will also be read.
  • cert_path - (Optional) Path to a directory with certificate information for connecting to the Docker host via TLS. If this is blank, the DOCKER_CERT_PATH will also be checked.
While cert_path is indicated as optional, it seems that terraform might require this attribute to be set for on most deployments since docker-machine, as per Docker Remote API documentation, instantiates a Docker daemon that uses an encrypted TCP socket using TLS.

The working example.tf then becomes:

# Configure the Docker provider
provider "docker" {
    host = "tcp://127.0.0.1:1234/"
    cert_path = "c:\\Users\\Nicolas\\.docker\\machine\\certs"
}

# Create a container
resource "docker_container" "foo" {
    image = "${docker_image.ubuntu.latest}"
    name = "foo"
}

resource "docker_image" "ubuntu" {
    name = "ubuntu:latest"
}
HashiCorp allows users to update the documentation via GitHub, which leads me to create a pull request https://github.com/hashicorp/terraform/pull/8895 to update it with at least the default dockerd port information.

And terraform plan now works:


$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

+ docker_container.foo
    bridge:           "<computed>"
    gateway:          "<computed>"
    image:            "${docker_image.ubuntu.latest}"
    ip_address:       "<computed>"
    ip_prefix_length: "<computed>"
    log_driver:       "json-file"
    must_run:         "true"
    name:             "foo"
    restart:          "no"

+ docker_image.ubuntu
    latest: "<computed>"
    name:   "ubuntu:latest"

Plan: 2 to add, 0 to change, 0 to destroy.

Friday, May 13, 2016

Ubuntu 16.04 and Docker 1.11 - Accessing secured private registry

After trying to access a private registry that is secured via ssl, and adding the certificate authority (ca.pem) under

  /etc/docker/certs.d/<IP of registry>

Where my IP is 10.68.230.7, the pull/push requests still failed.

$ docker pull 10.68.230.7/alpine
Using default tag: latest
Error response from daemon: Get https://10.68.230.7/v1/_ping: x509: certificate signed by unknown authority

It seems docker does not like certificates with the ca.pem file name.  To fix this rename the ca.pem file to ca.crt.

$ cp /etc/docker/certs.d/10.68.230.7/ca.pem /etc/docker/certs.d/10.68.230.7/ca.crt

Then it works.

$ docker pull 10.68.230.7/alpine
Using default tag: latest
latest: Pulling from alpine
d0ca440e8637: Already exists 
Digest: sha256:5c826f3f0f5c34aca4df43360ec0faef6326b18bd311309cc8ae3a83f799d1eb
Status: Downloaded newer image for 10.68.230.7/alpine:latest

Ubuntu 16.04, systemd and Docker

Ubuntu 16.04 LTS is now available.  After having made the switch from 14.04 without really looking at the changes except for the kernel number (4.x), I was pleasantly surprised by the fact that it now uses systemd.

Trying to setup docker to pull/push from a private registry using security, I first attempted to change the logging level to debug by adding -D in /etc/default/docker and after restarting docker noticed that no "debug" logs were shown.

This took me a while to find out but /etc/default/docker is not used anymore.

This fact is in the /etc/default/docker file but easy to miss:

# Docker Upstart and SysVinit configuration file
#
# THIS FILE DOES NOT APPLY TO SYSTEMD
#
#   Please see the documentation for "systemd drop-ins":
#   https://docs.docker.com/engine/articles/systemd/
#
# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"
# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"
# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"
# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"

Also, one can find out where a service file and configuration is located:

$ systemctl show --property=FragmentPath docker
FragmentPath=/usr/lib/systemd/system/docker.service
$ grep EnvironmentFile /usr/lib/systemd/system/docker.service
grep: /usr/lib/systemd/system/docker.service: No such file or directory

What the above tells us is that the docker service has no configuration file at the moment.

There are different ways to configure services in systemd.  The option described below is OK but deviates from a standard systemd setup in the config location as, since this is Ubuntu,  /etc/default path is used instead of /etc/sysconfig.

The first step to use a config file is to add the required informarion  to the /lib/systemd/system/docker.service file by adding an EnvironmentFile attribute in the [Service] section:

$ vi /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket
[Service]
Type=notify
# see https://docs.docker.com/engine/admin/systemd
EnvironmentFile=-/etc/default/docker
ExecStart=/usr/bin/docker daemon -H fd:// $DOCKER_OPTIONS \
          $DOCKER_STORAGE_OPTIONS \
          $DOCKER_NETWORK_OPTIONS \
          $BLOCK_REGISTRY \
          $INSECURE_REGISTRY
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
[Install]
WantedBy=multi-user.target


Now when the docker service is restarted it will use /etc/default/docker as its environment file, and pull the environment variables from it.

As such, the final step is to add the matching options in /etc/default/docker:
# Docker Upstart and SysVinit configuration file

# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"

# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"

# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"

# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"
INSECURE_REGISTRY=""
DOCKER_STORAGE_OPTIONS=""
DOCKER_NETWORK_OPTIONS=""
BLOCK_REGISTRY=""

It should work.