本地能够解析 k8s 内的服务域名
k8s 内运行的应用都有对应的服务，所以通过以下命令可以找到所有的服务（域名）及其 IP 地址和端口（有省略）：
1
2
3
4
5
6
7
8
9
|
[root@xfhuang-pc ~ (⎈ |kubernetes-admin@cluster.local:default)]$ kubectl get svc -n default
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
es-web NodePort 10.233.11.130 <none> 20501:58769/TCP,5005:36483/TCP 64d
flink-operator-webhook-service ClusterIP 10.233.27.164 <none> 443/TCP 157d
kubernetes ClusterIP 10.233.0.1 <none> 443/TCP 208d
minio NodePort 10.233.33.72 <none> 9000:30012/TCP,9090:30013/TCP 154d
mysql NodePort 10.233.22.116 <none> 3306:30006/TCP 154d
nacos-operator ClusterIP 10.233.6.58 <none> 8080/TCP 208d
sshd NodePort 10.233.15.132 <none> 22:32222/TCP 21h
|
NAME 就是 k8s 内的域名，CLUSTER-IP 就是其 IP。
可以简单地将他们提取出来保存到/etc/hosts 文件即可使用。
1
2
3
4
5
6
7
8
9
|
$ cat /etc/hosts | grep nginx
10.109.151.73 nginx
$ ping nginx
PING nginx (10.109.151.73): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
^C
--- nginx ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
|
域名已能手动方式解析正确。
也有其它方案可直接利用 K8s 的相关 API 来获取全量/监控变更，比如：sercand/kuberesolver。 这样更简洁，维护性高。
本地属于 K8s 内的流量转发到 K8s 内网
这一步其实是最难的。我一开始想着手动改路由表，后来我搜索到了一款非常好用的工具：
1
2
3
|
sshuttle: where transparent proxy meets VPN meets ssh
Transparent proxy server that works as a poor man's VPN. Forwards over ssh. Doesn't require admin. Works with Linux and MacOS. Supports DNS tunneling.
|
这是一款非常容易使用的通过 SSH 建立 VPN 的工具。
VPN 的原理大家知道：把本机网络的流量通过 VPN 网络发送出去。 简单说就是：你不在办公室，却能连上办公室的网络并使用。完全就像处于办公室内一样。 所以我赶紧在我的 K8s 内部署了一个 sshd 服务，尝试把它搞起来。
在 k8s 创建 sshd 服务
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
|
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations: {}
labels: {}
name: sshd
namespace: default
resourceVersion: '132405525'
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: sshd
app.kubernetes.io/name: sshd
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/instance: sshd
app.kubernetes.io/name: sshd
spec:
containers:
- args:
- >
set -euo pipefail
/entry.sh
echo 'ecdsa-sha2-nistp256
AAAAE3VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOsJRdlR0g2AatKk93rjkzbSnDTKXteJMQWtZLibJU7d/fUnsbgA71a8YltSz5qaBrQ4va5ShpQOVlaJi3YgSrs='
> /etc/ssh/keys/ssh_host_ecdsa_key.pub
echo '-----BEGIN OPENSSH PRIVATE KEY-----
b4BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAaAAAABNlY2RzYS
1zaGEyLW5pc3RwMjU2AAAACG5pc3RwMjU2AAAAQQTrCUXZUdINgGrSpPd645M20pw0yl7X
iTEFrWS4myVO3f31J7G4AO9WvGJbUs+amga0OL2uUoaUDlZWiYt2IEq7AAAAsH2N+3d9jf
t3AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOsJRdlR0g2AatKk
93rjkzbSnDTKXteJMQWtZLibJU7d/fUnsbgA71a8YltSz5qaBrQ4va5ShpQOVlaJi3YgSr
sAAAAhANr5qpUS1qt0Thpli78qrLD61kUki9V2+ss3KlTPmsW/AAAAEXJvb3RAZTY5NzVl
MjgyODE0AQIDBAUG
-----END OPENSSH PRIVATE KEY-----
' > /etc/ssh/keys/ssh_host_ecdsa_key
echo 'ssh-rsa
AAAAB3NzaC1yc2EAAAADAQABAAABgQDPcnOu56yPFPZY3fUDXdaLfaDBAxf9BjpodekCwVVEUDpVx/JajRWOF0on8jB/J7tA4pBLPnsyOK29vFPG1y6wXX12jff4QamECEaPxE9L/CGs9CMyO3a4lP+9fQH5sv6IKUh/UF/F0fEEy/b6edyvCDEv+qj5tSKry4qJ1pvg0r9S+A6y8rdTwuEdznPgceGVPv2V3MUThxLDQpuvTj7BAlMe2Kbjr0IpEqMDU5wDs5fQzZ9SgG2bLAhtciOlWoJ8GjH8+wbQPWU7squEojzDRXTcnje9H4nAoJIiParfeCPOGMp9WkUa7y/owsdsHZ0A/4p57zxQAp3mDmVOR1DpTJF0QLouVfrCBBBRB/AlLj4Drv9q+vwJs8b0y0vZHx6WZBtedktAO92VCKONvSMeNLz5LlstaP9KeEYJAHnZy5MlSOrO+mkbErUmYwriZsrOreSguIXG2E6pSbuxc2nMxojPLq6A0eiBW1dUuvzvoEzPX8iooeclZuYwwT3CQJk=
xfhuang@coderhuang-wsl' >> /root/.ssh/authorized_keys
sed -i 's/GatewayPorts no/GatewayPorts yes/' /etc/ssh/sshd_config
sed -i 's/AllowTcpForwarding no/AllowTcpForwarding yes/'
/etc/ssh/sshd_config
apk add python2
/usr/sbin/sshd -D -e -f /etc/ssh/sshd_config
command:
- /bin/sh
- '-c'
env:
- name: SSH_ENABLE_ROOT
value: 'true'
image: panubo/sshd
imagePullPolicy: Always
name: sshd
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
hostname: pipeline
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:
- lastTransitionTime: '2023-05-10T08:27:40Z'
lastUpdateTime: '2023-05-10T08:27:40Z'
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: 'True'
type: Available
- lastTransitionTime: '2023-05-10T08:27:29Z'
lastUpdateTime: '2023-05-10T09:28:50Z'
message: ReplicaSet "sshd-8648b9bf78" has successfully progressed.
reason: NewReplicaSetAvailable
status: 'True'
type: Progressing
observedGeneration: 5
readyReplicas: 1
replicas: 1
updatedReplicas: 1
---
apiVersion: v1
kind: Service
metadata:
annotations: {}
name: sshd
namespace: default
resourceVersion: '132379096'
spec:
clusterIP: 10.233.15.132
clusterIPs:
- 10.233.15.132
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: sshd
nodePort: 32222
port: 22
protocol: TCP
targetPort: 22
selector:
app.kubernetes.io/instance: sshd
app.kubernetes.io/name: sshd
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
|
上面出现的服务器的公钥、私钥和我自己的公钥都是临时通过 ssh-keygen 生成的。
然后，部署它：
由于是 NodePort 服务，所以我可以在集群外部也能访问它。
然后我在本机 .ssh/config 里面增加了一条 ssh 快捷登录记录：
1
2
3
4
5
6
7
8
|
Host sshd
# 这是 K8s 所在机器某一节点
HostName 10.7.20.25
# sshd 服务的 NodePort
Port 32222
User root
# root 的私钥
IdentityFile ~/.ssh/id_rsa
|
然后通过以下方式 ssh 到 K8s 内部的 sshd 服务：
1
2
3
4
5
6
7
8
9
10
11
12
13
|
✗ ssh sshd
Welcome to Alpine!
The Alpine Wiki contains a large amount of how-to guides and general
information about administrating Alpine systems.
See <http://wiki.alpinelinux.org/>.
You can setup the system with the command: setup-alpine
You may change this message by editing /etc/motd.
pipeline:~# exit
Connection to 10.7.20.25 closed.
|
用 sshuttle 正向打通网络
查看 sshuttle 的帮助文档并执行以下命令即可：
1
2
3
4
|
sshuttle -r sshd 10.233.0.0/18
c : Connected to server.
Failed to flush caches: Unit dbus-org.freedesktop.resolve1.service not found.
fw: Received non-zero return code 1 when flushing DNS resolver cache.
|
从本机直接访问 K8s 的内部网络就算打通了，接下来。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
✗ curl -L http://10.233.25.23
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width" />
<title></title>
<link href="/layout/vendor~253ae210-4535426d3071d75ddc27.css" rel="stylesheet"><link href="/layout/vendor~1f20a385-67fd93d750da972eaf8a.css" rel="stylesheet"><link href="/layout/layout-03c1847a79f4ee5c3ef3.css" rel="stylesheet">
<link href='/favicon.ico' rel='shortcut icon'>
<script src='/static/js/system.min.js'></script>
<script src='/static/js/named-exports.min.js'></script>
<script src='/static/js/use-default.min.js'></script>
<script src='/static/js/amd.js'></script>
</head>
<body>
<script type="systemjs-importmap">
{"imports":{"react":"/static/js/react.production.min.js","react-dom":"/static/js/react-dom.production.min.js","single-spa":"/static/js/single-spa.min.js","single-spa-react":"/static/js/single-spa-react.js","moment":"/static/js/moment.min.js","react-router":"/static/js/react-router.min.js","react-router-dom":"/static/js/react-router-dom.min.js","lodash":"/static/js/lodash.min.js","echarts":"/static/js/echarts.min.js","history":"/static/js/history.production.min.js"}}
</script>
<script async src="/layout/vendor~b9cf3951-6464d67ac8cb8b976f92.js"></script><script async src="/layout/vendor~253ae210-4535426d3071d75ddc27.js"></script><script async src="/layout/vendor~1f20a385-67fd93d750da972eaf8a.js"></script>
<script type="systemjs-importmap">
{
"imports": {"@portal/layout":"/layout/layout-03c1847a79f4ee5c3ef3.js"}
}
</script>
<script>
System.import('@portal/layout');
</script>
<div id="layout"></div>
</body>
</html>
%
|
在 K8s 内部访问本机服务
前面已经创建了 sshd 服务，借助 ssh 的远程转发，要实现远程主机访问本机的服务简直不要太容易！
前面的 sshd 部署文件中有以下几行：
1
2
|
sed -i 's/GatewayPorts no/GatewayPorts yes/' /etc/ssh/sshd_config
sed -i 's/AllowTcpForwarding no/AllowTcpForwarding yes/' /etc/ssh/sshd_config
|
这两行开启了 sshd 的端口转发服务。于是用以下一行代码即可实现在 K8s 内部访问本机服务：
1
|
$ ssh -R '*:5601:localhost:5601' sshd
|
另外 alibab 实现了这个工具
https://github.com/alibaba/kt-connect/blob/master/docs/zh-cn/guide/quickstart.md
操作步骤，需要把对应的 k8s 的 config 文件下载本地即可连接。
参考文档：
https://blog.twofei.com/848/
https://github.com/sshuttle/sshuttle