第一步安装版本：
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
[root@kubesphere ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
[root@master1 ~]# yum install -y kubeadm-1.23.9
Repository extras is listed more than once in the configuration
Last metadata expiration check: 0:00:27 ago on Tue 20 Sep 2022 02:15:52 PM CST.
Dependencies resolved.
==================================================================================================================
Package Architecture Version Repository Size
==================================================================================================================
Upgrading:
kubeadm x86_64 1.23.9-0 kubernetes 9.0 M
Transaction Summary
==================================================================================================================
Upgrade 1 Package
Total download size: 9.0 M
Downloading Packages:
4f2cd27ecd6913e34408df70f465a104feb1f 45% [================- ] 320
|
验证版本：
1
2
3
|
kubeadm version
[root@master1 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.9", GitCommit:"c1de2d70269039fe55efb98e737d9a29f9155246", GitTreeState:"clean", BuildDate:"2022-07-13T14:25:37Z", GoVersion:"go1.17.11", Compiler:"gc", Platform:"linux/amd64"}
|
排空节点
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
kubectl drain master1 --ignore-daemonsets
node/master1 cordoned
DEPRECATED WARNING: Aborting the drain command in a list of nodes will be deprecated in v1.23.
The new behavior will make the drain command go through all nodes even if one or more nodes failed during the drain.
For now, users can try such experience via: --ignore-errors
error: unable to drain node "master1", aborting command...
There are pending nodes to be drained:
master1
error: cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-6bf679fb9b-s8rqf, kube-system/traefik-58896d6b47-z49vm
[root@master1 ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.22.3
[upgrade/versions] kubeadm version: v1.23.9
W0920 14:20:38.050246 143343 version.go:103] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get "https://storage.googleapis.com/kubernetes-release/release/stable.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0920 14:20:38.050327 143343 version.go:104] falling back to the local client version: v1.23.9
[upgrade/versions] Target version: v1.23.9
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
|
#升级kubernetes
[root@master1 ~]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.22.3
[upgrade/versions] kubeadm version: v1.23.9
W0920 14:20:38.050246 143343 version.go:103] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get "https://storage.googleapis.com/kubernetes-release/release/stable.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0920 14:20:38.050327 143343 version.go:104] falling back to the local client version: v1.23.9
[upgrade/versions] Target version: v1.23.9
W0920 14:20:48.060253 143343 version.go:103] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.22.txt": Get "https://storage.googleapis.com/kubernetes-release/release/stable-1.22.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0920 14:20:48.060282 143343 version.go:104] falling back to the local client version: v1.23.9
[upgrade/versions] Latest version in the v1.22 series: v1.23.9
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 5 x v1.22.3 v1.23.9
Upgrade to the latest version in the v1.22 series:
COMPONENT CURRENT TARGET
kube-apiserver v1.22.3 v1.23.9
kube-controller-manager v1.22.3 v1.23.9
kube-scheduler v1.22.3 v1.23.9
kube-proxy v1.22.3 v1.23.9
CoreDNS 1.8.0 v1.8.6
etcd 3.5.0-0 3.5.1-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.23.9
_____________________________________________________________________
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________
|
升级到目标 1.23.9
1
2
3
4
5
6
|
[root@master1 ~]# kubeadm upgrade apply v1.23.9
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
cat <<EOF > ./kubeadm-config.yaml
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.23.9
imageRepository: registry.aliyuncs.com/k8sxio
controlPlaneEndpoint: "${APISERVER_NAME}:6443"
networking:
serviceSubnet: "10.96.0.0/16"
podSubnet: "${POD_SUBNET}"
dnsDomain: "cluster.local"
dns:
type: CoreDNS
imageRepository: swr.cn-east-2.myhuaweicloud.com${2}
imageTag: 1.8.0
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
EOF
|
1
|
etcdctl --endpoints 10.7.20.26:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
|
重新生成 etcd 证书
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
|
[root@master1 ~]# cd /etc/kubernetes/pki/etcd
[root@master1 etcd]#
[root@master1 etcd]# ls
ca.crt ca.key ca.srl healthcheck-client.crt healthcheck-client.key peer.crt peer.key server.crt
[root@master1 etcd]# kubeadm init phase certs etcd-server
W0920 15:34:38.027280 196570 version.go:103] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://storage.googleapis.com/kubernetes-release/release/stable-1.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0920 15:34:38.027362 196570 version.go:104] falling back to the local client version: v1.23.9
[certs] Using existing etcd/server certificate and key on disk
[root@master1 etcd]# kubeadm init phase certs etcd-ca
W0920 15:37:00.933472 198298 version.go:103] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://storage.googleapis.com/kubernetes-release/release/stable-1.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0920 15:37:00.933538 198298 version.go:104] falling back to the local client version: v1.23.9
[certs] Using existing etcd/ca certificate authority
[root@master1 etcd]#
[root@master1 etcd]# kubeadm init phase certs etcd-server
W0920 15:37:10.988708 198404 version.go:103] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0920 15:37:10.988787 198404 version.go:104] falling back to the local client version: v1.23.9
[certs] Using existing etcd/server certificate and key on disk
[root@master1 etcd]# kubeadm init phase certs etcd-healthcheck-client
W0920 15:37:21.026383 198456 version.go:103] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://storage.googleapis.com/kubernetes-release/release/stable-1.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0920 15:37:21.026465 198456 version.go:104] falling back to the local client version: v1.23.9
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[root@master1 etcd]# kubeadm init phase certs etcd-peer
W0920 15:37:31.068216 198549 version.go:103] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0920 15:37:31.068296 198549 version.go:104] falling back to the local client version: v1.23.9
[certs] Using existing etcd/peer certificate and key on disk
[root@master1 etcd]# kubeadm init phase certs apiserver-etcd-client
W0920 15:37:41.100238 198668 version.go:103] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://storage.googleapis.com/kubernetes-release/release/stable-1.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0920 15:37:41.100339 198668 version.go:104] falling back to the local client version: v1.23.9
[certs] Using existing apiserver-etcd-client certificate and key on disk
[root@master1 etcd]# ll
total 32
-rw-r--r-- 1 root root 1086 Dec 17 2021 ca.crt
-rw------- 1 root root 1675 Dec 17 2021 ca.key
-rw-r--r-- 1 root root 41 Feb 15 2022 ca.srl
-rw-r--r-- 1 root root 1111 Feb 15 2022 healthcheck-client.crt
-rw------- 1 root root 1675 Dec 17 2021 healthcheck-client.key
-rw-r--r-- 1 root root 1147 Feb 15 2022 peer.crt
-rw------- 1 root root 1679 Dec 17 2021 peer.key
-rw-r--r-- 1 root root 1151 Feb 15 2022 server.crt
[root@master1 etcd]# etcdctl --endpoints 10.7.20.26:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
Error: open /etc/kubernetes/pki/etcd/server.key: no such file or directory
[root@master1 etcd]# kubeadm init phase certs etcd-server
W0920 15:39:31.079342 200213 version.go:103] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://storage.googleapis.com/kubernetes-release/release/stable-1.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
W0920 15:39:31.079407 200213 version.go:104] falling back to the local client version: v1.23.9
[certs] Using existing etcd/server certificate and key on disk
[root@master1 etcd]# ll
total 32
-rw-r--r-- 1 root root 1086 Dec 17 2021 ca.crt
-rw------- 1 root root 1675 Dec 17 2021 ca.key
-rw-r--r-- 1 root root 41 Feb 15 2022 ca.srl
-rw-r--r-- 1 root root 1111 Feb 15 2022 healthcheck-client.crt
-rw------- 1 root root 1675 Dec 17 2021 healthcheck-client.key
-rw-r--r-- 1 root root 1147 Feb 15 2022 peer.crt
-rw------- 1 root root 1679 Dec 17 2021 peer.key
-rw-r--r-- 1 root root 1151 Feb 15 2022 server.crt
[root@master1 etcd]# etcdctl --endpoints 10.7.20.26:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key member list
{"level":"warn","ts":"2022-09-20T15:42:00.161+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0004328c0/10.7.20.26:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: remote error: tls: internal error\""}
Error: context deadline exceeded
[root@master1 etcd]# netstats -nlp|grep 2379
-bash: netstats: command not found
[root@master1 etcd]# netstat -nlp|grep 2379
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 2065/etcd
tcp 0 0 10.7.20.26:2379 0.0.0.0:* LISTEN 2065/etcd
[root@master1 etcd]# etcdctl --endpoints 10.7.20.26:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key member list
^C
[root@master1 etcd]# etcdctl --endpoints 10.7.20.26:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
{"level":"warn","ts":"2022-09-20T15:46:14.337+0800","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000376540/10.7.20.26:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: authentication handshake failed: remote error: tls: bad certificate\""}
Error: context deadline exceeded
[root@master1 etcd]# kill -9 2065
[root@master1 etcd]# etcdctl --endpoints 10.7.20.26:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
685818b53ba9af21, started, master1, https://192.168.20.26:2380, https://10.7.20.26:2379, false
|
执行更新命令，报错
1
2
3
4
5
6
7
8
|
[failed to renew certificates for component "kube-apiserver": failed to renew certificate apiserver: must specify at least one ExtKeyUsage, rename /etc/kubernetes/tmp/kubeadm-backup-manifests-2022-09-20-16-08-53/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml: no such file or directory]
couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced
k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade.rollbackOldManifests
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade/staticpods.go:524
k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade.upgradeComponent
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade/staticpods.go:230
k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade.StaticPodControlPlane
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/upgrade/staticpods.go:481
|
1
|
kubeadm upgrade apply -y v1.23.9 --ignore-preflight-errors=all --allow-experimental-upgrades --allow-release-candidate-upgrades --etcd-upgrade=false -v 6
|
重新生成 etcd 的证书
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
cat<<EOF>openssl-server.conf
[ req ]
default_bits = 2048
default_md = sha256
distinguished_name = req_distinguished_name
attributes = req_attributes
req_extensions = v3_req
[ req_distinguished_name ]
[ req_attributes ]
[ v3_req ]
basicConstraints = critical,CA:true
keyUsage = critical,Digital Signature, Key Encipherment, Certificate Sign
extendedKeyUsage = TLS Web Server Authentication, TLS Web Client Authentication
subjectAltName = @alt_names
[alt_names]
DNS.1 = 10.7.20.26
DNS.2 = localhost
IP.1 = 192.168.145.11
IP.2 = 10.7.20.26
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
cat<<EOF>openssl-ca.conf
[ req ]
default_bits = 2048
default_md = sha256
distinguished_name = req_distinguished_name
attributes = req_attributes
req_extensions = v3_req
[ req_distinguished_name ]
[ req_attributes ]
[ v3_req ]
basicConstraints = critical,CA:true
keyUsage = critical,Digital Signature, Key Encipherment, Certificate Sign
EOF
#用私钥和配置问阿金生成签署请求，CN是签署机构
openssl req -new -key ca.key -out ca.csr -subj "/CN=etcd-ca" -config openssl-ca.conf -extensions v3_req
#验证csr扩展属性是否生效
openssl req -noout -text -in ca.csr
|
取消对控制面节点的保护
1
2
|
[root@k8s-master ~]# kubectl uncordon k8s-master
node/k8s-master uncordoned
|
升级工作节点（node 节点）
4.1、升级 kubeadm
[root@k8s-node01 ~]# yum install -y kubeadm-1.23.0 –disableexcludes=kubernetes
4.2、保护节点
[root@k8s-master ~]# kubectl drain k8s-node01 –ignore-daemonsets
4.3、升级 kubelet 配置
[root@k8s-node01 ~]# kubeadm upgrade node
4.4、升级 kubelet 与 kubectl
[root@k8s-node01 ~]# yum install -y kubelet-1.23.0 kubectl-1.23.0 –disableexcludes=kubernetes
重启 kubelet
[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl restart kubelet
4.5、取消对节点的保护
[root@k8s-master ~]# kubectl uncordon k8s-node01
安装情况后，显示 node 一直都是 not ready，这个时候发现错误如下：
1
2
3
|
failed to list *v1.Service: Get "https://localhost:6443/api/v1/services?limit=500&resource
ListAndWatch" name:vendor/k8s.io/client-go/informers/factory.go:134 (23-Sep-2022 15:00:50.786) (total time: 19014m
pi/v1/services?limit=500&resourceVersion=0": read tcp 127.0.0.1:42432->127.0.0.1:6443: read: connection reset by p
|
这个时候很可能错误就是防火墙问题，关闭防火墙即可。