ole的知识小站

Jenkins Docker in Docker Agent

—

由

wangjm
于

Jenkins Docker in Docker Agent

转载来源 ： https://rokpoto.com/jenkins-docker-in-docker-agent/

So you installed Jenkins helm chart on Kubernetes cluster. Now you want to build Docker images. How would you do that? There are several ways. Today, we’ll focus on creating and using Jenkins Docker in Docker agent for that purpose. 因此，您在 Kubernetes 集群上安装了 Jenkins helm Chart。现在您想要构建 Docker 图像。你会怎么做？有几种方法。今天，我们将重点关注为此目的创建和使用 Jenkins Docker in Docker 代理。

If you later find this article useful take a look at the disclaimer for information on how to thank me. 如果您以后发现这篇文章有用，请查看免责声明，了解如何感谢我的信息。

Table Of Contents 目录

  1. Introduction 介绍
  2. Jenkins Docker in Docker Agent demo Docker Agent 演示中的 Jenkins Docker
  3. Summary 概括

Introduction 介绍

If all you need is Docker in Docker Jenkins agent images, jump straight to the demo. Otherwise, if you want to understand the context better, keep reading. 如果您需要的只是 Docker in Docker Jenkins 代理图像，请直接跳至演示。否则，如果您想更好地理解上下文，请继续阅读。

First, let’s clarify what is Docker in Docker agent in the context of Jenkins builds. In short, CI/CD systems like Jenkins build docker images, push them to some registry and probably run them for testing purposes. 首先，让我们澄清一下 Jenkins 构建上下文中的 Docker in Docker 代理是什么。简而言之，像 Jenkins 这样的 CI/CD 系统会构建 docker 映像，将它们推送到某个注册表，并可能出于测试目的运行它们。

Jenkins living and building Docker images in middle ages Jenkins 在中世纪生活并构建 Docker 镜像

You’ve probably seen Jenkins doing that for you. I’ve seen it too and it was bad. Why? Because Jenkins and its agents were configured and run manually on VMs as Java processes. It has become increasingly frustrating to maintain such a system because installations of Jenkins plugins, tools on agents like docker and maven were manual. That’s why, eventually no one really maintained this system, the plugins not updated and no improvements introduced over time. If such Jenkins serves critical business purposes, then such a state becomes a permanent one. 您可能已经看到 Jenkins 为您做这件事。我也看过，而且很糟糕。为什么？因为 Jenkins 及其代理是作为 Java 进程在虚拟机上手动配置和运行的。维护这样一个系统变得越来越令人沮丧，因为 Jenkins 插件、 docker 和 maven 等代理上的工具的安装都是手动的。这就是为什么，最终没有人真正维护这个系统，插件没有更新，并且随着时间的推移也没有引入任何改进。如果这样的 Jenkins 服务于关键的业务目的，那么这种状态就会成为永久性的状态。 Everybody fears to do anything, so everybody does nothing. And no one really knows when this Jenkins will fall into the abyss. 每个人都害怕做任何事，所以每个人都什么都不做。而没有人真正知道这个 Jenkins 什么时候会掉入深渊。

Jenkins building Docker images in a modern world Jenkins 在现代世界中构建 Docker 镜像

You may ask how to run Jenkins and its agents in the modern world? Right, leverage what Kubernetes and Docker may offer. Thanks to these tools, Jenkins and its agents run as Kubernetes pods. Moreover, agents are dynamically provisioned for each build. So, at any given moment, Jenkins controller is the only long running web application. Hence, this way saves a lot of resources and doesn’t require constantly running VMs and/or physical machines for long running agents. It may sound too good. 你可能会问在现代世界如何运行 Jenkins 及其代理？是的，利用 Kubernetes 和 Docker 可能提供的功能。借助这些工具， Jenkins 及其代理作为 Kubernetes Pod 运行。此外，代理是为每个构建动态配置的。因此，在任何给定时刻， Jenkins 控制器都是唯一长时间运行的 Web 应用程序。因此，这种方式节省了大量资源，并且不需要为长时间运行的代理持续运行虚拟机和/或物理机。这听起来可能太好了。 Indeed, to enjoy these benefits, one will have to build images of the controller and the agents. 事实上，为了享受这些好处，我们必须构建控制器和代理的映像。

Docker in Docker Jenkins agent Docker Jenkins 代理中的 Docker

Today, we focus on the agent which will allow building Docker images. Wait, will we need an image which has Docker daemon installed inside? That may sound challenging, because Docker daemon will run in another (Docker) container. That’s why the name Docker in Docker. 今天，我们重点关注允许构建 Docker 镜像的代理。等等，我们需要一个内部安装了 Docker 守护进程的镜像吗？这听起来可能具有挑战性，因为 Docker 守护进程将在另一个（Docker）容器中运行。这就是 Docker 中 Docker 这个名字的由来。 Though, because Docker is basically a client-server application, this term may mean 2 different things. 不过，由于 Docker 基本上是一个客户端-服务器应用程序，因此这个术语可能有两种不同的含义。

Docker Client or Docker Daemon in Docker? Docker 中的 Docker 客户端还是 Docker 守护进程？

Firstly, docker client may run inside Docker container. It needs Docker server to interact with. Firstly, it may be Docker daemon which runs the container itself. 首先，docker客户端可以在Docker容器内运行。它需要与 Docker 服务器进行交互。首先，它可能是运行容器本身的 Docker 守护进程。 And the container with Docker client will interact with the daemon via docker socket which is mapped to some path inside the container via volumes. 带有 Docker 客户端的容器将通过 docker 套接字与守护进程交互，该套接字通过卷映射到容器内的某个路径。 Another option is Docker daemon running inside Docker container and the client is running either in the same container or different one. 另一种选择是 Docker 守护进程在 Docker 容器内运行，而客户端则在同一容器或不同容器中运行。

Why to use Docker Daemon in Docker? 为什么要在Docker中使用Docker Daemon？

Firstly, because we may not have control over container runtime in Kubernetes cluster. It could be Docker, but it could be containerd as well. Next, giving access to the host’s Docker daemon via socket to individual builder containers may have security implications. In addition, this way containers will share resources with all the deployed workloads to Kubernetes cluster, something undesirable. 首先，因为我们可能无法控制 Kubernetes 集群中的容器运行时。它可能是 Docker，但也可能是 containerd 。接下来，通过套接字向各个构建器容器提供对主机 Docker 守护进程的访问可能会产生安全隐患。此外，这种方式容器将与 Kubernetes 集群中部署的所有工作负载共享资源，这是不可取的。 Mostly, because single builds could jeopardize critical services like Jenkins itself. You may say, wait, is running docker daemon for each build better in terms of resources consumption? Yes, that may sound heavy. However, these daemons run in isolation from the host’s Docker daemon only for the duration of the builds. 主要是因为单个构建可能会危及 Jenkins 本身等关键服务。您可能会说，等等，就资源消耗而言，为每个构建运行 docker 守护进程是否更好？是的，这听起来可能很沉重。但是，这些守护进程仅在构建期间与主机的 Docker 守护进程隔离运行。

Enough talking. Let’s show how to build an image for Docker in Docker agent and show sample Jenkins builds using it. 说够了。让我们展示如何在 Docker 代理中构建 Docker 镜像，并展示使用它的 Jenkins 构建示例。

Jenkins Docker in Docker Agent demo Docker Agent 演示中的 Jenkins Docker

Demo Prerequisites 演示先决条件

First, you’ll need Docker for building and running images. 首先，您需要 Docker 来构建和运行图像。

Next, you’ll need Kubernetes cluster. If you don’t have one, install on your machine minikube and kubectl. 接下来，您需要 Kubernetes 集群。如果您没有，请在您的计算机上安装 minikube 和 kubectl 。

Then, start Kubernetes cluster using minikube start --profile custom. 然后，使用 minikube start --profile custom 启动 Kubernetes 集群。

If you prefer, you can also repeat this demo on a managed Kubernetes cluster. Check out how easy it is to create Kubernetes Cluster on Linode. Get 100$ credit on Linode using this link. Linode is a cloud service provider recently purchased by Akamai. With this purchase, Akamai became a competitor in the cloud providers market. 如果您愿意，还可以在托管 Kubernetes 集群上重复此演示。了解在 Linode 上创建 Kubernetes 集群是多么容易。使用此链接在 Linode 上获得 100 美元的积分。 Linode是最近被Akamai收购的一家云服务提供商。通过此次收购，Akamai 成为云提供商市场的竞争对手。

Install helm as well. We’ll use it for installing Jenkins in the cluster. 同时安装 helm 。我们将使用它在集群中安装 Jenkins 。

Install Jenkins helm chart 安装 Jenkins helm 图表

There are several ways to install Jenkins according to an official Jenkins documentation. We’ll install Jenkins using helm. I picked this way because it allows to configure Jenkins easily using values.yaml. More specifically, we’ll configure Jenkins to use Docker in Docker agent image. 根据 Jenkins 官方文档，有多种安装 Jenkins 的方法。我们将使用 helm 安装 Jenkins。我选择这种方式是因为它允许使用 values.yaml 轻松配置 Jenkins。更具体地说，我们将配置 Jenkins 使用 Docker in Docker 代理映像。

Before we install Jenkins, let’s make some preparation steps: 在安装 Jenkins 之前，我们先做一些准备步骤：

# create dedicated namespace where Jenkins helm chart will be installed
kubectl create namespace jenkins

Let’s configure Jenkins by putting relevant values to values.yaml 让我们通过将相关值放入 values.yaml 来配置 Jenkins

  • download official helm chart’s values.yaml 下载官方 Helm Chart 的 values.yaml
wget -4 https://raw.githubusercontent.com/jenkinsci/helm-charts/main/charts/jenkins/values.yaml

  • many Jenkins aspects can be configured usingvalues.yaml, let’s put the minimum necessary for installation of Jenkins on Minikube:

    可以使用 values.yaml 配置 Jenkins 的许多方面，让我们将安装 Jenkins 所需的最低限度放在 Minikube 上：

    • set serviceType to NodePort 将 serviceType 设置为 NodePort

  • install Jenkins release using below commands: 使用以下命令安装 Jenkins 版本：

helm repo add jenkinsci https://charts.jenkins.io
helm repo update
helm install jenkins -n jenkins -f values.yaml jenkinsci/jenkins
  • after Jenkins images are downloaded and deployed you will see below status: 下载并部署 Jenkins 镜像后，您将看到以下状态：
kubectl get all -n jenkins
pod/jenkins-0   2/2     Running   0          2m57s

NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S
)          AGE
service/jenkins         NodePort    10.100.21.112   <none>        8080:3
0653/TCP   2m57s
service/jenkins-agent   ClusterIP   10.96.252.104   <none>        50000/
TCP        2m57s

NAME                       READY   AGE
statefulset.apps/jenkins   1/1     2m57s

Now, we’ll follow the documentation in full: 现在，我们将完整地遵循文档：

  • To get Jenkins URL we’ll run below commands: 要获取 Jenkins URL，我们将运行以下命令：
jsonpath="{.spec.ports[0].nodePort}"
NODE_PORT=$(kubectl get -n jenkins -o jsonpath=$jsonpath services jenkins)
jsonpath="{.items[0].status.addresses[0].address}"
NODE_IP=$(kubectl get nodes -n jenkins -o jsonpath=$jsonpath)
echo http://$NODE_IP:$NODE_PORT/login
  • Next, navigate to Jenkins URL and login using admin user and password that could be retrieved using below commands: 接下来，导航到 Jenkins URL 并使用 admin 用户和密码登录，可以使用以下命令检索该用户和密码：
jsonpath="{.data.jenkins-admin-password}"
secret=$(kubectl get secret -n jenkins jenkins -o jsonpath=$jsonpath)
echo $(echo $secret | base64 --decode)

That’s how easy it is to raise Jenkins using helm. Chart’s values.yaml provides a nice way to configure almost all aspects of Jenkins in config as code way. Therefore, we’ll put a reference to Docker in Docker agent there as well. Yet, first let’s create the images for this agent. 这就是使用 helm 提升 Jenkins 的简单性。 Chart 的 values.yaml 提供了一种以 config as code 方式配置 Jenkins 几乎所有方面的好方法。因此，我们也会在那里放置对 Docker in Docker 代理的引用。然而，首先让我们为这个代理创建图像。

Create Docker in Docker Agent image Create Docker in Docker Agent 图像

To create Docker in Docker Jenkins agent, we’ll use below 2 images: 要创建 Docker in Docker Jenkins 代理，我们将使用以下 2 个图像：

  • We’ll use the official Docker dind image for Docker daemon in Docker part. 我们将使用官方 Docker dind 映像作为 Docker daemon in Docker 部分。
  • For Docker client in Docker part, we’ll use image built from below Dockerfile: 对于 Docker client in Docker 部分，我们将使用从 Dockerfile 下面构建的图像：
FROM jenkins/jnlp-agent-docker
USER root

COPY entrypoint.sh /entrypoint.sh
RUN chown jenkins:jenkins /entrypoint.sh
RUN chmod +x /entrypoint.sh

USER jenkins
ENTRYPOINT "/entrypoint.sh"
  • entrypoint.sh busy waits for docker-daemon to become available and starts Jenkins agent which will connect to Jenkins controller using Jenkins remoting. entrypoint.sh 忙等待 docker-daemon 变得可用并启动 Jenkins 代理，该代理将使用 Jenkins remoting 连接到 Jenkins 控制器。
#!/usr/bin/env bash

RETRIES=6

sleep_exp_backoff=1

for((i=0;i<RETRIES;i++)); do
    docker version
    dockerd_available=$?
    if [ $dockerd_available == 0 ]; then
        break 
    fi
    sleep ${sleep_exp_backoff}
    sleep_exp_backoff="$((sleep_exp_backoff * 2))"
done

exec /usr/local/bin/jenkins-agent "$@"

You can find above files in dind-client-jenkins-agent repository. 您可以在 dind-client-jenkins-agent 存储库中找到上述文件。

I guess I owe you a bit of explanation here, because we said we’ll use Docker daemon in Docker Jenkins agent while additionally mentioning some Docker client in Docker part. 我想我在这里需要向您解释一下，因为我们说过我们将使用 Docker daemon in Docker Jenkins 代理，同时另外提到一些 Docker client in Docker 部分。

As we said, Jenkins builds will run in dynamically provisioned Kubernetes pods. And as we already know that Kubernetes pods may run more than one container. Hence in our setup, each pod for Docker in Docker agent will have 2 containers. The first one is the container with Docker client. And it’s basically Jenkins agent, because it extends jenkins/jnlp-agent-docker base image. The second one is the container with Docker daemon inside. It will run as a side car container in the same pod. 正如我们所说，Jenkins 构建将在动态配置的 Kubernetes pod 中运行。我们已经知道 Kubernetes Pod 可能运行多个容器。因此，在我们的设置中， Docker in Docker 代理的每个 pod 将有 2 个容器。第一个是带有 Docker 客户端的容器。它基本上是 Jenkins 代理，因为它扩展了 jenkins/jnlp-agent-docker 基础镜像。第二个是里面有 Docker 守护进程的容器。它将作为同一个吊舱中的边车容器运行。

Finally, let’s build dind-client-jenkins-agent agent image and push it to DockerHub 最后，我们构建 dind-client-jenkins-agent 代理镜像并将其推送到 DockerHub

# to point your shell to minikube's docker-daemon
$ eval $(minikube  docker-env)
# build the image
$ docker build -t dind-client-jenkins-agent .
# tag the image
$ docker tag dind-client-jenkins-agent [docker_hub_id]/dind-client-jenkins-agent
# push the image
$ docker push [docker_hub_id]/dind-client-jenkins-agent

You probably wonder how all of this works. It’s time to see this agent in action. 您可能想知道这一切是如何运作的。是时候看看这个代理的实际操作了。

Use Docker in Docker agent 使用 Docker in Docker 代理

Upgrade Jenkins helm release 升级 Jenkins helm 版本

Firstly, let’s put references to the agent to Jenkins Helm chart’s values.yaml. 首先，我们将对代理的引用放入 Jenkins Helm 图表的 values.yaml 中。

To see default settings for all agents, have a look at the key agent 要查看所有代理的默认设置，请查看键 agent

We’ll put reference and configuration of Jenkins Docker in Docker agent under key additionalAgents: 我们将 Jenkins Docker in Docker 代理的引用和配置放在 additionalAgents 密钥下：

additionalAgents: 
  dind:
    podName: dind-agent
    customJenkinsLabels: dind-agent
    image: dind-client-jenkins-agent
    tag: latest
    envVars:
     - name: DOCKER_HOST
       value: "tcp://localhost:2375"
    alwaysPullImage: true
    yamlTemplate:  |-  
     spec: 
         containers:
           - name: dind-daemon 
             image: docker:20.10-dind
             securityContext: 
               privileged: true
             env: 
               - name: DOCKER_TLS_VERIFY
                 value: ""

Note that because Docker client and Docker daemon run in containers under the same pod, they share the same network namespace and the client can connect to the daemon using localhost:2375. To see why we used privileged and DOCKER_TLS_VERIFY refer to dind image description. You can see full values.yaml in jenkins-dind-demo repository. 请注意，由于 Docker 客户端和 Docker 守护进程在同一 pod 下的容器中运行，因此它们共享相同的网络命名空间，并且客户端可以使用 localhost:2375 连接到守护进程。要了解为什么我们使用 privileged 和 DOCKER_TLS_VERIFY ，请参阅 dind 图像描述。您可以在 jenkins-dind-demo 存储库中看到完整的 values.yaml 。

Now, let’s upgrade our Jenkins release using new values.yaml: 现在，让我们使用新的 values.yaml: 升级我们的 Jenkins 版本

helm upgrade jenkins jenkinsci/jenkins -n jenkins -f values.yaml

Run sample Jenkins job using Docker in Docker agent 在 Docker 代理中使用 Docker 运行示例 Jenkins 作业

Now, let’s create a sample Jenkins pipeline which will use our Docker in Docker agent. We’ll use it to build Docker image from this repository we used while exploring CI/CD using Jenkins and Docker. 现在，让我们创建一个示例 Jenkins 管道，它将使用我们的 Docker in Docker 代理。我们将使用它从我们在使用 Jenkins 和 Docker 探索 CI/CD 时使用的存储库构建 Docker 映像。

img

Jenkinsfile is straightforward: Jenkinsfile 很简单：

pipeline {
    agent  {
        label 'dind-agent'
    }

    stages {
        stage("build image") {
            steps {
                script {
                    sh "docker build jenkins -t docker-in-jenkins"
                }
            }
        }
    }
}

Pay attention to the agent’s label dind-agent. It’s the same label we have assigned to dind agent in customJenkinsLabels field. That’s why, builds for this pipeline will use our Docker in Docker images while provisioning Kubernetes pods for the builds. 注意代理的标签 dind-agent 。它与我们在 customJenkinsLabels 字段中分配给 dind 代理的标签相同。这就是为什么，此管道的构建将使用我们的 Docker in Docker 映像，同时为构建配置 Kubernetes pod。

Let’s run the pipeline in Jenkins UI and inspect pods using kubectl while it’s running. Wow, we see that dind pod contains 2 containers (one with docker client and another one with docker daemon) and it was dynamically provisioned just for this build. 让我们在 Jenkins UI 中运行管道，并在运行时使用 kubectl 检查 pod。哇，我们看到 dind pod 包含 2 个容器（一个带有 docker 客户端，另一个带有 docker 守护进程），并且它是专门为此构建动态配置的。

$ kubectl get pod -n jenkins
pod/dind-agent-w622h   2/2     Running   0          35s
pod/jenkins-0          2/2     Running   2          23h

Inspect the job’s Console output to see the actual Kubernetes deployment manifests used to provision the agent. After a while we see that the image was successfully built and the dind pod terminated. Of course, one can use docker push, docker run or any other docker CLI command in the pipeline. That’s cool, isn’t it? 检查作业的 Console output 以查看用于配置代理的实际 Kubernetes 部署清单。一段时间后，我们看到镜像已成功构建并且 dind pod 终止。当然，可以在管道中使用 docker push 、 docker run 或任何其他 docker CLI 命令。这很酷，不是吗？

Final words on performance and alternatives 关于性能和替代方案的最后一句话

docker cache 码头缓存

Of course, to fully leverage docker in docker agent, one would need to somehow cache docker build results produced on the agent. It would mimic the docker build default behavior. 当然，为了充分利用 docker in docker 代理，需要以某种方式缓存代理上生成的 docker 构建结果。它将模仿 docker 构建默认行为。

Podman, Kaniko – daemonless and rootless alternatives Podman、Kaniko – 无守护和无根替代品

Wait, do we really need Docker daemon agent while there’s Podman and Kaniko? True, these tools don’t require a daemon. So our alternative agent could use just one container (with podman or kaniko inside). Yet, Kaniko is only capable of building and pushing the images, while sometimes it’s useful to run them, e.g. as part of tests. I 等等，当有 Podman 和 Kaniko 时，我们真的需要 Docker 守护进程代理吗？确实，这些工具不需要守护进程。因此，我们的替代代理可以仅使用一个容器（内部有 podman 或 kaniko ）。然而， Kaniko 只能构建和推送图像，而有时运行它们很有用，例如作为测试的一部分。我

Then, there’s Podman which uses the same CLI as Docker. Podman Jenkins Agent I described as well uses it. For example, you can use podman build instead of docker build). It’s indeed lightweight and doesn’t require a daemon Docker needs. Yet, from my understanding it’s still not widely adopted. Have a look, for instance, at the number of questions at Stack Overflow tagged with podman and docker tags and make conclusions yourself. 然后， Podman 使用与 Docker 相同的 CLI。我描述的 Podman Jenkins Agent 也使用它。例如，您可以使用 podman build 而不是 docker build ）。它确实是轻量级的，不需要守护进程 Docker 需要。然而，据我了解，它仍然没有被广泛采用。例如，查看 Stack Overflow 上标有 podman 和 docker 标签的问题数量，然后自己得出结论。

Moreover, I personally encountered random bugs while building images using podman and there was even a use cases when I was able to use only docker agent. For example, running Minikube inside the agent. So it’s minikube running as a container inside docker daemon container running as part of a pod inside Kubernetes cluster which itself may run in minikube container (as in our demo) which is managed by some host’s docker daemon. That may sound crazy, but there are use cases when we may need it, for example for deploying and testing a helm chart as part of Jenkins pipeline. I covered this use case in Running minikube in Docker container post. 此外，我个人在使用 podman 构建图像时遇到了随机错误，甚至在我只能使用 docker agent 的用例中。例如，在代理内运行 Minikube 。因此，它作为 docker daemon 容器内的容器运行，作为 Kubernetes 集群内 pod 的一部分运行，而集群本身可以在 minikube 容器中运行（如我们的演示中所示）由某些主机的 docker daemon 管理。这听起来可能很疯狂，但在某些情况下我们可能需要它，例如作为 Jenkins 管道的一部分部署和测试 helm 图表。我在《在 Docker 容器中运行 minikube》一文中介绍了这个用例。

Summary 概括

That’s it about Jenkins Docker in Docker agent. You may find useful my other Kubernetes and Jenkins articles. As always, feel free to share. If you found this article useful, take a look at the disclaimer for information on how to thank me. 这就是 Docker 代理中的 Jenkins Docker。您可能会发现我的其他 Kubernetes 和 Jenkins 文章很有用。一如既往，请随意分享。如果您觉得这篇文章有用，请查看免责声明，了解如何感谢我的信息。

Recommended Kubernetes courses on Pluralsight: Pluralsight 上推荐的 Kubernetes 课程：

Sign up using this link to get exclusive discounts like 50% off your first month or 15% off an annual subscription) 使用此链接注册即可获得独家折扣，例如首月 50% 折扣或年度订阅 15% 折扣）

Recommended Kubernetes books on Amazon: 亚马逊上推荐的 Kubernetes 书籍：

评论

发表回复

您的电子邮箱地址不会被公开。 必填项已用 * 标注