One thing that has been coming up more and more in my experiments has been the
need to create custom container images for various tasks. I also would like to
set up a system that doesn’t require any containerization specific tools on my
development workstation like Docker or Podman. This means
I need to figure out a way to build containers from within a Kubernetes
cluster. Most of my time is spent on OpenShift but I also have a
Rancher cluster at home that I would like to use the same process
on. This unfortunately rules out the
object that OpenShift provides since it won’t be portable between the two.
What this leaves us with is a K8s
Job controller running an
instance of the Buildah container image. We’re going
with Buildah over something like Docker-in-Docker since it doesn’t
require me to expose any sockets to the container making things at least a
little more secure. Additionally, since
buildah can take input from a file
on the local filesystem, a HTTP/HTTPS endpoint, or a Git repository I am going
to write a Helm chart that will provide all the configuration
flexibility that I want.
A lot of the configuration in the chart was adapted from this blog post along with modifications to allow the user to pass in configuration through the different methods. I also added templates for both push and pull secrets to authenticate with Docker repositories. With both Quay and DockerHub requiring authentication for a lot of resources it made sense to build that functionality in. The final chart is in this GitHub repository and I plan on adapting and expanding it as time goes on.
To keep the container as isolated as possible we’re going to use the
--storage-driver vfs option because we don’t have to mount the
device with the container. We are also going to mount the container cache
directory and the build directory as
emptyDir volumes instead of attaching
PersistentVolumes. It might be possible to update the chart to allow
ReadWriteMany image cache volume but setting up a caching proxy
registry is probably less fuss and less dependent on your
underlying storage technology.
I did learn, as detailed here, that it is super
important to provide the
--storage-driver vfs option to all of the
calls so that it can find the image it just created. If you don’t it will look
like your image has disappeared when it hasn’t and the built image won’t push
to the registry.
The simplest use case for this chart is the
ConfigMap based configuration
where the contents of the
Dockerfile are stored in a
ConfigMap and loaded
into the container as a volume mount. The example below shows how you could
use this method to build an Ubuntu container with Ansible installed which you
then push back to DockerHub.
# build-from-configmap-values.yml --- build: configMap: enabled: true dockerfileData: | FROM docker.io/library/ubuntu:20.10 RUN apt-get update && \ apt-get install -y software-properties-common && \ apt-add-repository ppa:ansible/ansible && \ apt-get install -y ansible && \ apt-get clean source: authSecret: enabled: true registry: docker.io username: <REDACTED> password: <REDACTED> destination: registry: docker.io repository: example/ubuntu-ansible tag: latest authSecret: enabled: true username: <REDACTED> password: <REDACTED>
I’m still testing the HTTPS and Git functionality but I think the ability to build images entirely within Kubernetes is going to be really helpful in my continued experimentation.