ole的知识小站

00 k8s安装_deprecated

—

由

wangjm
于

家庭网络k8s部署测试

参考： 官网容器运行时的文档：https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/

参考： 官网kubeadm创建集群的文档： https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

参考： 网友创建集群的模板： https://blog.csdn.net/qq_44956318/article/details/121335756

参考： 网友创建集群的模板： https://www.cnblogs.com/RainingNight/p/using-kubeadm-to-create-a-cluster-1-12.html

参考：阿里云ack集群网络参考： https://help.aliyun.com/zh/ack/ack-managed-and-ack-dedicated/user-guide/plan-cidr-blocks-for-an-ack-cluster-1?spm=a2c4g.11186623.0.0.43b77678ng1wZI

前置说明

场景

场景： k8s云服务商性价比不足。（阿里云ack标准版开三个节点无负载，一天要百来块RMB。 实际上云厂商有提供边缘云服务，暂没有深入去了解）

场景： 自有IDC或企业有公网ip池，想配k8s。

场景：想亲自搭建一个k8s集群，对k8s整体有个理解。

prerequisite

安装和简单使用过kubectl和minikube （https://kubernetes.io/docs/tasks/tools/）

有大体上看过官网concepts和tutorials文档部分（https://kubernetes.io/docs/concepts/， https://kubernetes.io/docs/tutorials/）

使用阿里云ack或其他云服务厂商提供的k8s服务，创建或实践过简单的storageclass, configmap, secret, pod, deployment, statefulset, service, ingress

关于本文的说明

这里主要使用kubeadm初始化k8s，但是有一些系统级的配置、容器运行时、网络相关的内容，可能需要手动配置。

注意，下面的实践，主要基于systemd作为1号进程的linux发行版（sysV不一定适合）。安装的k8s是目前最新的1.27.4

实践操作的两台主机：

# 1号主机
cat /etc/redhat-release 
CentOS Stream release 8
uname -a
Linux jingmin-kube-master1 4.18.0-448.el8.x86_64 #1 SMP Wed Jan 18 15:02:46 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

# 2号主机，是台archlinux
uname -a
Linux jingmin-kube-archlinux 6.1.44-1-lts #1 SMP PREEMPT_DYNAMIC Tue, 08 Aug 2023 19:07:19 +0000 x86_64 GNU/Linux

两台主机网络,通过交换机互联

# 1号主机enp3s0接口，在192.168.1.0/24网段，ip 192.168.1.1
[root@jingmin-kube-master1 containerd]# ip addr
1: lo: ...
2: enp2s0: ...
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 60:be:b4:07:0a:e7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global noprefixroute enp3s0
       valid_lft forever preferred_lft forever
    inet6 ...
4: enp4s0: ...
5: enp5s0: ...
6: ppp0: ...


# 2号主机enp6s0接口，在192.168.1.0/24网段，ip为192.168.1.7
[root@jingmin-kube-archlinux ~]# ip addr
1: lo: ...
2: enp5s0: ...
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 2a:06:13:3d:06:ef brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.7/24 brd 192.168.1.255 scope global noprefixroute enp6s0
       valid_lft forever preferred_lft forever
    inet6 ...

配置容器运行时CRI

容器运行时介绍

容器化发展大概是这样的： 裸机部署应用 -》容器化单机部署（docker run 单镜像)-》容器化单机部署（docker compose 一组应用）-》多节点容器化部署k8s （允许跨节点，伸缩。。）

容器化最初是docker独一家，开源，但仅支持单机使用。redhat, google等公司及开发者的加入，容器编排逐渐成熟，目前看起来最受市场认可的是google的k8s标准。然后有社区维护的k8s和各云厂商提供的k8s服务。

docker和k8s在oci层面达成了一致。（OCI – Open Container Initiative. It standardizes container images and runtimes.）。也就是说它们使用一样的镜像，底层是一样的运行时。

基于oci标准，docker和k8s有各自的运行时实现。对于已经安装了docker的主机，k8s支持直接使用docker的运行时containerd。对于没有安装docker的主机，可以直接安装cri-o等运行时。k8s在containerd/cri-o之上抽象出一个统一的接口层cri， 供k8s上层使用。

An infographic illustrating the container ecosystem.

参考了这里的说明：https://phoenixnap.com/kb/docker-vs-containerd-vs-cri-o

配置容器运行时前置条件

主要是一些操作系统级别的设置

设置主机名

#主机1
hostnamectl set-hostname jingmin-kube-master1

#主机2
hostnamectl set-hostname jingmin-kube-archlinux

加载内核模块

linux开启内核转发，加载转发和容器需要的模块

参考：https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/#install-and-configure-prerequisites

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# 设置所需的 sysctl 参数，参数在重新启动后保持不变
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# 应用 sysctl 参数而不重新启动
sudo sysctl --system

通过运行以下指令确认 br_netfilter 和 overlay 模块被加载：

lsmod | grep br_netfilter
lsmod | grep overlay

通过运行以下指令确认 net.bridge.bridge-nf-call-iptables、net.bridge.bridge-nf-call-ip6tables 和 net.ipv4.ip_forward 系统变量在你的 sysctl 配置中被设置为 1：

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

关swap

查看目前是否有swap

free -m

修改系统加载时不使用swap

cat "vm.swappiness = 0" >> /etc/sysctl.d/k8s.conf
sysctl --system

关闭所有的swap (本次启动期间有效)

swapoff -a

关闭所有的swap (下次启动有效)

vim /etc/fstab

很多人说这里直接注释掉swap分区那一行就可以了。

但是有的系统不适用。systemd会自动扫描gpt分区，加载其中的swap分区。要么fdisk工具删分区，要么这里/etc/fstab中swap分区的options那一列由defaults改为noauto，即不自动挂载。

安装k8s工具

主要是安装 kubeadm、kubelet 和 kubectl

参考官方文档： https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

参考： https://discuss.kubernetes.io/t/facing-challanges-for-installation-of-kubernest-cluster-setup-through-kubeadm-on-rhel8-9-vm-404-for-https-packages-cloud-google-com-yum-repos-kubernetes-el7-x86-64-repodata-repomd-xml-ip-142-250-115-139/27345/2

centos/fedora/redhat是这样：

# 谷歌源已失效。
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

# 使用这个源
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.27/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.27/rpm/repodata/repomd.xml.key
EOF

# 将 SELinux 设置为 permissive 模式（相当于将其禁用）
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet

安装配置containerd运行时

安装containerd

安装docker-ce ，也就是社区版docker（安装docker时， centos可能默认替换为开源的podman)

yum install docker-ce

从前面的docker/k8s分层图，可以知道，安装docker时，已经安装了containerd。

配置containerd

参考： https://kubernetes.io/zh-cn/docs/setup/production-environment/container-runtimes/#containerd

参考： https://kubernetes.io/docs/setup/production-environment/container-runtimes/#override-pause-image-containerd

可以使用默认的模板:

containerd config default > /etc/containerd/config.toml

也可以直接创建/etc/containerd/config.toml。

然后调整或加入以下配置：

vim /etc/containerd/config.toml

#disabled_plugins = ["cri"]
enabled_plugins = ["cri"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.9"

安装docker时，默认没有启用k8s需要的cri插件，这里启用，并设置为使用systemd管理的cgroup。

我主机上自带的pause版本较老，后面kubeadm初始化k8s的时候有警告。改用新一点的版本3.9。

配置containerd代理（可选）

后面用kubeadm初始化k8s的时候，发现有镜像拉不下来，可以选择这里加下代理配置。

vim /lib/systemd/system/containerd.service

临时配置下containerd.service的代理(加了Evironment两行)，用完可以注释掉

# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
Environment="HTTP_PROXY=http://192.168.1.7:8889"
Environment="HTTPS_PROXY=http://192.168.1.7:8889"
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

重载服务，重启服务

systemctl daemon-reload
systemctl restart containerd

配置开机启动相关服务

systemctl enable containerd
systemctl enable kubelet
#systemctl enable docker

配置docker代理（可选）

如果手动docker pull镜像， 可以设置下docker代理（kubeadm调用的是containerd，不是这里的代理）：

https://docs.docker.com/config/daemon/systemd/#httphttps-proxy

https://docs.docker.com/network/proxy/

初始化master节点

现在开始安装k8s，这里使用kubadm安装

官网kubeadm创建集群的文档： https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

官网kubeadm创建集群的中文文档： https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

方法一： 可以参考这位网友的方法，直接调整kubeadm的配置文件

参考： https://www.cnblogs.com/RainingNight/p/using-kubeadm-to-create-a-cluster-1-12.html

方法二： 也可以在安装时指定选项（这种方式使用默认的谷歌镜像源gcr.io，需要按前面的操作，配置代理）

参考： https://blog.csdn.net/qq_44956318/article/details/121335756

kubeadm init \
--apiserver-advertise-address=192.168.1.1 \
--control-plane-endpoint=192.168.1.1 \
--kubernetes-version v1.27.4 \
--service-cidr=172.31.0.0/20 \
--pod-network-cidr=172.30.0.0/16

记录下生成的内容，后面要用

[root@jingmin-kube-master1 yum.repos.d]# kubeadm init \
> --apiserver-advertise-address=192.168.1.1 \
> --control-plane-endpoint=192.168.1.1 \
> --kubernetes-version v1.27.4 \
> --service-cidr=172.31.0.0/20 \
> --pod-network-cidr=172.30.0.0/16
[init] Using Kubernetes version: v1.27.4
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W0821 14:13:53.077831    4835 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [jingmin-kube-master1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [172.31.0.1 192.168.1.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [jingmin-kube-master1 localhost] and IPs [192.168.1.1 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [jingmin-kube-master1 localhost] and IPs [192.168.1.1 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 5.502714 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node jingmin-kube-master1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node jingmin-kube-master1 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: piro3i.5temz83jyjau5r88
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join 192.168.1.1:6443 --token piro3i.5temz83jyjau5r88 \
    --discovery-token-ca-cert-hash sha256:bdc79fe69b3bb037f29068ce7a7f39a88e3c8d166317131121da42b4f2f6bf1b \
    --control-plane 

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.1.1:6443 --token piro3i.5temz83jyjau5r88 \
    --discovery-token-ca-cert-hash sha256:bdc79fe69b3bb037f29068ce7a7f39a88e3c8d166317131121da42b4f2f6bf1b

按照上面的提示，master节点上执行

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

然后kubectl命令就可以使用了，可以查看和操作k8s中的各种资源了。

网络上的版本：

默认情况下主节点是不会调度运行pod的，如果想要master节点也可以运行pod，执行如下指令

kubectl taint nodes --all node-role.kubernetes.io/master-

末尾的减号标识去除相应的taint

官网参考： https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#control-plane-node-isolation

如果你希望能够在控制平面节点上调度 Pod，例如单机 Kubernetes 集群，请运行:

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

安装网络组件

安装网络组件flannel

参考：https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network

参考：https://kubernetes.io/zh-cn/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-networking-model

参考： https://kubernetes.io/zh-cn/docs/concepts/cluster-administration/addons/#networking-and-network-policy

参考：https://github.com/flannel-io/flannel#deploying-flannel-manually

参考： https://github.com/flannel-io/flannel/blob/master/Documentation/kubernetes.md

wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

vim kube-flannel.yml

修改其中关于pod-cidr相关的设置，网段改为与上面pod-network-cidr一致(net-conf.json相关部分)：

  net-conf.json: |
    {
      "Network": "172.30.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }

参考： https://github.com/flannel-io/flannel/blob/master/Documentation/configuration.md#key-command-line-options

参考： https://github.com/containernetworking/cni/issues/486

flannel默认会使用默认网关所在的接口。

由于我在网关路由器上也安装了k8s，而它的默认网关是PPPoE拨号连接建立的ppp0接口地址。而与k8s通信的接口是另一块网卡接口。拨号连接断开重连，会导致k8s网络异常。

（我家庭网络网关路由上也装了k8s，拨号上网会建立起一个ppp0接口，默认安装flannel可能用的这个接口。ppp0重连，公网ip变动后，k8s服务异常。）

其中容器的参数追加两行iface-can-reach的配置，如下：

      containers:
      - args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface-can-reach
        - 192.168.1.1

kubectl logs看了下flannel日志，网关路由器上的flannel这样好像选了lo网卡。算了，也能用了。实在不行可以试试--iface-regex参数，正则指定下网段（暂未尝试）。

最终，flannel完整的yaml配置文件如下：

apiVersion: v1
kind: Namespace
metadata:
  labels:
    k8s-app: flannel
    pod-security.kubernetes.io/enforce: privileged
  name: kube-flannel
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: flannel
  name: flannel
  namespace: kube-flannel
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: flannel
  name: flannel
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
- apiGroups:
  - networking.k8s.io
  resources:
  - clustercidrs
  verbs:
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: flannel
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-flannel
---
apiVersion: v1
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "172.30.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
kind: ConfigMap
metadata:
  labels:
    app: flannel
    k8s-app: flannel
    tier: node
  name: kube-flannel-cfg
  namespace: kube-flannel
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: flannel
    k8s-app: flannel
    tier: node
  name: kube-flannel-ds
  namespace: kube-flannel
spec:
  selector:
    matchLabels:
      app: flannel
      k8s-app: flannel
  template:
    metadata:
      labels:
        app: flannel
        k8s-app: flannel
        tier: node
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      containers:
      - args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface-can-reach
        - 192.168.1.1
        command:
        - /opt/bin/flanneld
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: EVENT_QUEUE_DEPTH
          value: "5000"
        image: docker.io/flannel/flannel:v0.22.1
        name: kube-flannel
        resources:
          requests:
            cpu: 100m
            memory: 50Mi
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            - NET_RAW
          privileged: false
        volumeMounts:
        - mountPath: /run/flannel
          name: run
        - mountPath: /etc/kube-flannel/
          name: flannel-cfg
        - mountPath: /run/xtables.lock
          name: xtables-lock
      hostNetwork: true
      initContainers:
      - args:
        - -f
        - /flannel
        - /opt/cni/bin/flannel
        command:
        - cp
        image: docker.io/flannel/flannel-cni-plugin:v1.2.0
        name: install-cni-plugin
        volumeMounts:
        - mountPath: /opt/cni/bin
          name: cni-plugin
      - args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        command:
        - cp
        image: docker.io/flannel/flannel:v0.22.1
        name: install-cni
        volumeMounts:
        - mountPath: /etc/cni/net.d
          name: cni
        - mountPath: /etc/kube-flannel/
          name: flannel-cfg
      priorityClassName: system-node-critical
      serviceAccountName: flannel
      tolerations:
      - effect: NoSchedule
        operator: Exists
      volumes:
      - hostPath:
          path: /run/flannel
        name: run
      - hostPath:
          path: /opt/cni/bin
        name: cni-plugin
      - hostPath:
          path: /etc/cni/net.d
        name: cni
      - configMap:
          name: kube-flannel-cfg
        name: flannel-cfg
      - hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
        name: xtables-lock

应用此插件(网络插件)

kubectl apply -f ./kube-flannel.yml

安装失败时的重置

如果安装过程中出现问题，尽量查下日志， 以及google下。

journalctl -xeu containerd

实在不行，删除重来

参考：https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-reset/

参考： https://stackoverflow.com/questions/57648829/how-to-fix-timeout-at-waiting-for-the-kubelet-to-boot-up-the-control-plane-as-st

kubeadm reset

也可用于卸载集群

参考： https://www.cnblogs.com/RainingNight/p/using-kubeadm-to-create-a-cluster-1-12.html

参考： https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#remove-the-node

想要撤销kubeadm执行的操作，首先要排除节点，并确保该节点为空, 然后再将其关闭。

在Master节点上运行：

kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node name>

然后在需要移除的节点上，重置kubeadm的安装状态：

sudo kubeadm reset

如果你想重新配置集群，使用新的参数重新运行kubeadm init或者kubeadm join即可。

If your cluster was setup to utilize IPVS, run ipvsadm –clear (or similar) to reset your system’s IPVS tables.

 ipvsadm --clear

在 Kubernetes 中，使用 iptables 来实现网络隔离和转发。如果您需要重置 iptables 防火墙规则，您可以执行以下命令：

iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

这个命令将清除所有的 iptables 防火墙规则，并将计数器重置为零。这将影响所有正在运行的容器，因此请谨慎使用。在执行此命令之前，请确保您已经备份了 iptables 配置文件以及任何其他重要的配置文件，以防止数据丢失或损坏。

需要注意的是，在 Kubernetes 中，Iptables 是由 kube-proxy 进程自动管理的，如果您重置 iptables 规则，kube-proxy 将会自动重建 iptables 规则。因此，重置 iptables 规则只是一个临时的解决方案，如果您需要永久更改 iptables 规则，应该修改 kube-proxy 的配置文件并重新启动 kube-proxy 进程。

节点重新加入集群

参考： https://aiops.com/news/post/13773.html

kubernetes集群删除节点

以下操作都是在主节点下操作

一、先将节点设置为维护模式

我们可以通过输入以下命令来查看节点名称

kubectl get nodes

我们要删除的节点名称为k8s-node2，

我们通过以下命令将k8s-node2节点设置为维护模式

kubectl drain k8s-node2 –delete-local-data –force –ignore-daemonsets node/k8s-node2

二、删除节点

kubectl delete node k8s-node2

三、确认是否已经删除

kubectl get nodes

这样就完成了集群节点删除 。

如果节点想要重新加入集群，可以通过以下操作来重新加入。

一、生成token

我们在主节点输入以下命令来重新生成token

kubeadm token create –print-join-command

二、node节点重新加入集群

以下操作在node节点操作

停掉kubelet

systemctl stop kubelet

删除之前的相关文件

rm -rf /etc/kubernetes/*

重新加入集群，这里复制上一步生成的token，重新加入节点即可。

安装node节点

node节点配置运行时

同样需要确认和安装容器运行时。在node节点上来一遍 配置容器运行时CRI

安装过docker，看起来已经包含了containerd

ls /run/containerd/containerd.sock

有结果，确认是containerd (这里有说明： https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-runtime)

mkdir -p /etc/containerd
vim /etc/containerd/config.toml

调整配置，内容略（前面章节有）。

直接从master那边复制过来也行：

scp 192.168.1.1:/etc/containerd/config.toml /etc/containerd/

后面在工作节点上 kubeadm join操作的时候，这里有点问题。gcr.io不通，导致创建 kube-proxy和flannel的相关pod一直失败。

可以配置代理，kubectl delete pod 删一下旧的pod,会自动创建新的。

也可以参考这位网友的方法，直接调整kubeadm的配置文件。参考： https://www.cnblogs.com/RainingNight/p/using-kubeadm-to-create-a-cluster-1-12.html

systemctl enable kubelet
systemctl enable containerd
#systemctl enable docker

node节点加入到k8s集群

使用前面master节点 kubeadm init成功之后的输出（添加节点的输出）。

kubeadm join 192.168.1.1:6443 --token x5tj4c.68ytog0bnjryxavj \
    --discovery-token-ca-cert-hash sha256:f13c1a70262df9fff72f29e85d28eec385229368eb8f3fad1085516d44cfbc1a

docker添加代理(也可以直接从master节点复制过来scp 192.168.1.1:/etc/docker/daemon.json ./)：

vim /etc/docker/daemon.json
{
  "proxies": {
    "http-proxy": "http://192.168.1.7:8889",
    "https-proxy": "http://192.168.1.7:8889",
    "no-proxy": "*.test.example.com,.example.org,*.ole12138.top,.ole12138.top,127.0.0.0/8,10.0.0.0,172.16.0.0/12,192.168.0.0/16"
  }
}

重启服务

systemctl daemon-reload 
systemctl restart docker

复制主节点kubectl配置

scp 192.168.1.1:/root/.kube/config /root/.kube/

kubeadm 生成的token过期后，集群增加节点

参考： https://www.cnblogs.com/hongdada/p/9854696.html

通过kubeadm初始化后，都会提供node加入的token:

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 18.16.202.35:6443 --token zr8n5j.yfkanjio0lfsupc0 --discovery-token-ca-cert-hash sha256:380b775b7f9ea362d45e4400be92adc4f71d86793ba6aae091ddb53c489d218c

默认token的有效期为24小时，当过期之后，该token就不可用了。

解决方法如下：

  1. 重新生成新的token

    [root@node1 flannel]# kubeadm  token create
kiyfhw.xiacqbch8o8fa8qj
[root@node1 flannel]# kubeadm  token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION   EXTRA GROUPS
gvvqwk.hn56nlsgsv11mik6   <invalid>   2018-10-25T14:16:06+08:00   authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token
kiyfhw.xiacqbch8o8fa8qj   23h         2018-10-27T06:39:24+08:00   authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token

  2. 获取ca证书sha256编码hash值

    [root@node1 flannel]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
5417eb1b68bd4e7a4c82aded83abc55ec91bd601e45734d6aba85de8b1ebb057

  3. 节点加入集群

      kubeadm join 18.16.202.35:6443 --token kiyfhw.xiacqbch8o8fa8qj --discovery-token-ca-cert-hash sha256:5417eb1b68bd4e7a4c82aded83abc55ec91bd601e45734d6aba85de8b1ebb057

几秒钟后，您应该注意到kubectl get nodes在主服务器上运行时输出中的此节点。

上面太繁琐，一步到位：

kubeadm token create --print-join-command

第二种方法：

# 第二种方法
token=$(kubeadm token generate)
kubeadm token create $token --print-join-command --ttl=0
参考：

https://blog.csdn.net/mailjoin/article/details/79686934

网络存储配置

k8s将各节点的cpu，内存都统一管理起来。但是节点的外存却不在调度与管理范围之内。

前面安装好k8s的master和node节点之后，允许各节点使用各自节点上的硬盘（hostPath类型的存储）。这种存储使用是有限制的，当pod发生调度到其他节点时，在之前节点hostPath存储的数据并不会同步到新节点上。

hostPath只适合存储临时数据。

对于需要持久化的数据，需要使用nfs网络存储，或ceph分布式存储，或其他云厂商提供的云存储服务等。

配置nfs网络存储

nfs服务器

内网找一台主机，装nfs服务，提供nfs存储服务，方便k8s使用。

我这台电脑是archlinux，安装命令如下

pacman -Syy nfs-utils

nfs主要分两个版本 v3或v4。 注意：v4并不一定更好，协议和机制有点不同，所有现在是市面上两个版本的nfs都有使用的。

nfs服务器端全局配置在/etc/nfs.conf， 将文件系统导出为nfs目录的配置在/etc/exports中配置，然后执行exportfs -arv使配置生效。

# 启动服务
systemctl enable nfsv4-server
systemctl start nfsv4-server

# 编辑配置，将/data/ole/nfs下的几个目录导出为nfs文件夹，供内网挂载使用
vim /etc/exports
cat /etc/exports

# /etc/exports - exports(5) - directories exported to NFS clients
#
# Example for NFSv3:
#  /srv/home        hostname1(rw,sync) hostname2(ro,sync)
# Example for NFSv4:
#  /srv/nfs4        hostname1(rw,sync,fsid=0)
#  /srv/nfs4/home   hostname1(rw,sync,nohide)
# Using Kerberos and integrity checking:
#  /srv/nfs4        *(rw,sync,sec=krb5i,fsid=0)
#  /srv/nfs4/home   *(rw,sync,sec=krb5i,nohide)
#
# Use `exportfs -arv` to reload.

/ole/data/nfs/public 192.168.0.0/16(rw) 172.16.0.0/12(rw) 10.0.0.0/8(rw)
/ole/data/nfs/sync   192.168.0.0/16(rw,sync) 172.16.0.0/12(rw,sync) 10.0.0.0/8(rw,sync)
/ole/data/nfs/async  192.168.0.0/16(rw,async) 172.16.0.0/12(rw,async) 10.0.0.0/8(rw,async)
/ole/data/nfs/no_root_squash   192.168.0.0/16(rw,sync,no_root_squash) 172.16.0.0/12(rw,sync,no_root_squash) 10.0.0.0/8(rw,sync,no_root_squash)

后面k8s里使用的是这里/ole/data/nfs/no_root_squash的配置，注意这个配置及其不安全。暂时就我一个人用，就给它这个权限了。。。

What is root_squash? 什么是root_squash？

root_squash will allow the root user on the client to both access and create files on the NFS server as root. root_squash 将允许客户端上的 root 用户以 root 身份在 NFS 服务器上访问和创建文件。 Technically speaking, this option will force NFS to change the client’s root to an anonymous ID and, in effect, this will increase security by preventing ownership of the root account on one system migrating to the other system. 从技术上讲，此选项将强制 NFS 将客户端的 root 更改为匿名 ID，实际上，这将通过防止一个系统上的 root 帐户的所有权迁移到另一系统来提高安全性。 This is needed if you are hosting root filesystems on the NFS server (especially for diskless clients); with this in mind, it can be used (sparingly) for selected hosts, but you should not use no_root_squash unless you are aware of the consequences. 如果您在 NFS 服务器上托管根文件系统（特别是无盘客户端），则需要这样做；考虑到这一点，它可以（谨慎地）用于选定的主机，但您不应该使用 no_root_squash ，除非您知道后果。

参考： https://www.thegeekdiary.com/basic-nfs-security-nfs-no_root_squash-and-suid/

no_root_squash：登入 NFS 主机使用分享目录的使用者，如果是 root 的话，那么对于这个分享的目录来说，他就具有 root 的权限！这个项目『极不安全』，不建议使用！ root_squash：在登入 NFS 主机使用分享之目录的使用者如果是 root 时，那么这个使用者的权限将被压缩成为匿名使用者，通常他的 UID 与 GID 都会变成 nobody 那个系统账号的身份；

修改了下文件夹权限（好像是因为权限不足）

chmod 777 /ole/data/nfs/no_root_squash

注意以上操作是极其不负责任和不安全的。。。以后再说

然后使之生效

exportfs -arv

查看网络上的nfs共享文件夹

showmount -e

可以看到已经允许挂载

[root@jingmin-kube-archlinux k8s]# showmount -e
Export list for jingmin-kube-archlinux:
/ole/data/nfs/no_root_squash 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
/ole/data/nfs/async          10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
/ole/data/nfs/sync           10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
/ole/data/nfs/public         10.0.0.0/8,172.16.0.0/12,192.168.0.0/16

在k8s中配置nfs相关的storageclass和provisioner

创建nfs provisioner和storageclass，以及rbac相关的权限对象。

参考:https://www.cnblogs.com/devopsyyds/p/16246116.html

参考：https://jimmysong.io/kubernetes-handbook/practice/using-nfs-for-persistent-storage.html

参考：https://medium.com/@myte/kubernetes-nfs-and-dynamic-nfs-provisioning-97e2afb8b4a9

注意这里，设为了默认存储类storageclass.kubernetes.io/is-default-class: "true"

[root@jingmin-kube-archlinux k8s]# cat ./nfs-storageclass.yaml 
## 创建了一个存储类
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-storage
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: fuseim.pri/ifs  
parameters:
  archiveOnDelete: "true"  ## 删除pv的时候，pv的内容是否要备份

---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    # replace with namespace where provisioner is deployed
    kubernetes.io/metadata.name: nfs
  # replace with namespace where provisioner is deployed
  name: nfs
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/nfs-subdir-external-provisioner:v4.0.2
          # resources:
          #    limits:
          #      cpu: 10m
          #    requests:
          #      cpu: 10m
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs  
            - name: NFS_SERVER
              value: 192.168.1.7 ## 指定自己nfs服务器地址
            - name: NFS_PATH  
              value: /ole/data/nfs/no_root_squash  ## nfs服务器共享的目录
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.1.7
            path: /ole/data/nfs/no_root_squash
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfs
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfs
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

确认已经创建了storageclass

[root@jingmin-kube-archlinux k8s]# kubectl get sc
NAME                    PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-storage (default)   fuseim.pri/ifs   Delete          Immediate           false                  15m

确认对应命名空间下已经创建了相应的资源

[root@jingmin-kube-archlinux k8s]# kubectl get all -n nfs
NAME                                          READY   STATUS    RESTARTS   AGE
pod/nfs-client-provisioner-5c7b77d69d-26498   1/1     Running   0          15m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nfs-client-provisioner   1/1     1            1           15m

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/nfs-client-provisioner-5c7b77d69d   1         1         1       15m

测试nfs存储类可用

创建一个测试pvc （persistent volume claim，永久卷存储请求。也就是申请一块永久存储）

[root@jingmin-kube-archlinux k8s]# cat ./test-pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
  namespace: default
spec:
  #storageClassName: nfs-storage
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 200Mi

执行,并确认申请到了存储

[root@jingmin-kube-archlinux k8s]# kubectl apply -f ./test-pvc.yaml 
persistentvolumeclaim/test-pvc created
[root@jingmin-kube-archlinux k8s]# kubectl get pvc
NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-pvc   Bound    pvc-c559dceb-0965-4fbc-ba2d-2d11b76e0751   200Mi      RWX            nfs-storage    6s
[root@jingmin-kube-archlinux k8s]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM              STORAGECLASS   REASON   AGE
pvc-c559dceb-0965-4fbc-ba2d-2d11b76e0751   200Mi      RWX            Delete           Bound    default/test-pvc   nfs-storage             9s

删除测试用的pvc

[root@jingmin-kube-archlinux k8s]# kubectl delete pvc test-pvc

节点压力驱逐

参考： https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/kubelet-integration/

参考： https://kubernetes.io/zh-cn/docs/concepts/scheduling-eviction/node-pressure-eviction/

参考： https://cloud.tencent.com/developer/article/1664414

参考： https://www.cnblogs.com/lianngkyle/p/16582132.html

kubeadm默认配置的kubelet可以没有加参数，这里设置一下节点压力驱逐的参数。防止内存、磁盘不足后系统hang住。

以下是错误做法（注意到有同目录下drop-file）

[root@jingmin-kube-master1 system]# cd /usr/lib/systemd/system
[root@jingmin-kube-master1 system]# ls
...
kubelet.service
kubelet.service.d
...

有个drop-file,八成是里面会覆盖掉默认的配置

[root@jingmin-kube-master1 system]# cd kubelet.service.d/
[root@jingmin-kube-master1 kubelet.service.d]# ls
10-kubeadm.conf
[root@jingmin-kube-master1 kubelet.service.d]# cat 10-kubeadm.conf 
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

果然，最后两行，覆盖了原来的命令。

同时还注意到，用户自定义的配置，应该放到/etc/sysconfig/kubelet里。

OK，恢复下原来的/usr/lib/systemd/system/kubelet.service文件。(仅适用于radhat系linux发行版，其他发行版也是类似的改环境变量的值)

修改下/etc/sysconfig/kubelet

[root@jingmin-kube-master1 kubelet.service.d]# vim /etc/sysconfig/kubelet 
[root@jingmin-kube-master1 kubelet.service.d]# cat /etc/sysconfig/kubelet 
KUBELET_EXTRA_ARGS="--node-ip=192.168.1.1 \
--kube-reserved=cpu=200m,memory=250Mi \
--eviction-hard=memory.available<5%,nodefs.available<10%,imagefs.available<10% \
--eviction-soft=memory.available<10%,nodefs.available<15%,imagefs.available<15% \
--eviction-soft-grace-period=memory.available=2m,nodefs.available=2m,imagefs.available=2m \
--eviction-max-pod-grace-period=30 \
--eviction-minimum-reclaim=memory.available=0Mi,nodefs.available=500Mi,imagefs.available=500Mi"

重载服务配置，重启服务

[root@jingmin-kube-master1 kubelet.service.d]# systemctl daemon-reload 
[root@jingmin-kube-master1 kubelet.service.d]# systemctl restart kubelet.service 
[root@jingmin-kube-master1 kubelet.service.d]# systemctl status kubelet.service

如果服务状态异常，可以看下日志

[root@jingmin-kube-master1 kubelet.service.d]# journalctl -exu kubelet

评论

发表回复

您的电子邮箱地址不会被公开。 必填项已用 * 标注