helm部署gitlab17.0.2

#依赖条件安装了 metallb 和 cert-manager 安装了默认存储类部署 gitlab

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


helm upgrade --install gitlab  \

  --namespace=gitlab \

  --create-namespace \

  --timeout 600s \

  --set global.edition=ce \

  --set gitlab-runner.install=false \

  --set global.hosts.domain=example.com \

  --set certmanager-issuer.email=me@example.com gitlab/




#shuchu

Error: Unable to continue with install: CustomResourceDefinition "certificaterequests.cert-manager.io" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "gitlab"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "gitlab"



# 安装

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.8.0/cert-manager.crds.yaml



kubectl edit crd certificaterequests.cert-manager.io


#安装gitlab-runner
helm upgrade --install gitlab \

  --namespace=gitlab \

  --set gitlab-runner.install=true \

  --set gitlab-runner.certsSecretName=gitlab-runner-certs \

  --reuse-values gitlab/

注意这里遇到问题，需要给 gitlab-runner 配置 hosts 解析

重启 coredns 的 pod

1

kubectl -n kube-system rollout restart deployment.apps/coredns

修改 localdns 的配置

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60


kubectl edit cm -n kube-system node-local-dns

apiVersion: v1
data:
  Corefile: |
    cluster.local:53 {
        errors
        cache {
                success 9984 30
                denial 9984 5
        }
        reload
        loop
        bind 169.254.20.10 __PILLAR__DNS__SERVER__
        forward . 10.96.0.10 {
                force_tcp
        }
        prometheus :9253
        health 169.254.20.10:8080
        }
    in-addr.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.20.10 __PILLAR__DNS__SERVER__
        forward . 10.96.0.10 {
                force_tcp
        }
        prometheus :9253
        }
    ip6.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.20.10 __PILLAR__DNS__SERVER__
        forward . 10.96.0.10 {
                force_tcp
        }
        prometheus :9253
        }
    .:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.20.10 __PILLAR__DNS__SERVER__
        #forward . __PILLAR__UPSTREAM__SERVERS__ # 线上dns解析默认是不通过coredns
        forward . 10.96.0.10 { # 修改为转发coredns解析
                force_tcp
        }
        prometheus :9253
        }
kind: ConfigMap
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
  name: node-local-dns
  namespace: kube-system

重启 localcoredns 测试一下是否可以正常访问到 gitlab.example.com

这里涉及到的原理如下

为什么需要 NodeLocalDNS

处于 ClusterFirst 的 DNS 模式下的 Pod 可以连接到 kube-dns 的 serviceIP 进行 DNS 查询，通过 kube-proxy 组件添加的 iptables 规则将其转换为 CoreDNS 端点，最终请求到 CoreDNS Pod。

通过在每个集群节点上运行 DNS 缓存，NodeLocal DNSCache 可以缩短 DNS 查找的延迟时间、使 DNS 查找时间更加一致，以及减少发送到 kube-dns 的 DNS 查询次数。

在集群中运行 NodeLocal DNSCache 有如下几个好处：

如果本地没有 CoreDNS 实例，则具有最高 DNS QPS 的 Pod 可能必须到另一个节点进行解析，使用 NodeLocal DNSCache 后，拥有本地缓存将有助于改善延迟
跳过 iptables DNAT 和连接跟踪将有助于减少 conntrack 竞争并避免 UDP DNS 条目填满 conntrack 表（上面提到的 5s 超时问题就是这个原因造成的）
从本地缓存代理到 kube-dns 服务的连接可以升级到 TCP，TCP conntrack 条目将在连接关闭时被删除，而 UDP 条目必须超时(默认 nfconntrackudp_timeout 是 30 秒)
将 DNS 查询从 UDP 升级到 TCP 将减少归因于丢弃的 UDP 数据包和 DNS 超时的尾部等待时间，通常长达 30 秒（3 次重试+ 10 秒超时）
具体内容查看这个：https://www.lixueduan.com/posts/kubernetes/23-node-local-dns/

导出 gitlab 自签名证书

1
2
3
4
5
6
7


kubectl -n gitlab get secret gitlab-gitlab-tls --template='{{ index .data "tls.crt" }}' | base64 -d > gitlab.crt
kubectl -n gitlab create secret generic gitlab-runner-certs \
  --from-file=gitlab.example.com.crt=gitlab.crt \
  --from-file=registry.example.com.crt=gitlab.crt \
  --from-file=minio.example.com.crt=gitlab.crt

kubectl -n gitlab get secret example-com-tls --template='{{ index .data "tls.crt" }}' | base64 -d > gitlab.crt

#接下来安装 gitlab-runner

1
2
3
4
5


helm upgrade --install gitlab \
  --namespace=gitlab \
  --set gitlab-runner.install=true \
  --set gitlab-runner.certsSecretName=gitlab-runner-certs \
  --reuse-values gitlab/

接下来查看成功展示吧可以部署一个 job 测试一下新建要给项目 ci_test，添加以下这个文件 .gitlab-ci.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


# 设置执行镜像
image: busybox:latest

# 整个pipeline有两个stage
stages:
  - build
  - test

# 定义全局缓存，缓存的key来自分支信息，缓存位置是vendor文件夹
cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - vendor/

before_script:
  - export
  - echo "Before script section"

after_script:
  - echo "After script section"

build1:
  stage: build
  tags:
    - k8s
  script:
    - echo "将内容写入缓存"
    - mkdir -p vendor
    - echo "build" > vendor/hello.txt

test1:
  stage: test
  tags:
    - k8s
  script:
    - echo "从缓存读取内容"
    - cat vendor/hello.txt

注意 runner 需要有 k8s 的 tag 才可以运行。

这里使用 istio-gw 来代理服务

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126





openssl req -out example_certs1/gitlab.example.com.csr -newkey rsa:2048 -nodes -keyout example_certs1/gitlab.example.com.key -subj "/CN=gitlab.example.com/O=gitlab organization"

openssl x509 -req -sha256 -days 365 -CA example_certs1/example.com.crt -CAkey example_certs1/example.com.key ```bash
 -config san.cnf
``` -set_serial 1 -in example_certs1/gitlab.example.com.csr -out example_certs1/gitlab.example.com.crt



kubectl create -n istio-system secret tls gitlab-credential \

  --key=example_certs1/gitlab.example.com.key \

  --cert=example_certs1/gitlab.example.com.crt






cat <<EOF | kubectl apply -f -

apiVersion: gateway.networking.k8s.io/v1

kind: Gateway

metadata:

  name: mygateway

  namespace: istio-system

spec:

  gatewayClassName: istio

  listeners:

  - name: https-gitlab

    hostname: "gitlab.example.com"

    port: 443

    protocol: HTTPS

    tls:

      mode: Terminate

      certificateRefs:

      - name: gitlab-credential

    allowedRoutes:

      namespaces:

        from: Selector

        selector:

          matchLabels:

            kubernetes.io/metadata.name: gitlab

EOF




cat <<EOF | kubectl apply -f -

apiVersion: gateway.networking.k8s.io/v1

kind: HTTPRoute

metadata:

  name: helloworld

  namespace: gitlab

spec:

  parentRefs:

  - name: mygateway

    namespace: istio-system

  hostnames: ["gitlab.example.com"]

  rules:

  - matches:

    - path:

        type: PathPrefix

        value: /

    backendRefs:

    - name: gitlab-webservice-default

      port: 8181

EOF




curl -v -HHost:gitlab.example.com --resolve "gitlab.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \

  --cacert example_certs1/example.com.crt "https://gitlab.example.com:$SECURE_INGRESS_PORT/"




kubectl -n gitlab get secret | grep initial-root-password

  kubectl -n gitlab get secret gitlab-gitlab-initial-root-password -o jsonpath='{.data.password}' | base64 -d

创建 gitlab 对应的 gateway

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27



cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: mygateway
  namespace: istio-system
spec:
  gatewayClassName: istio
  listeners:
  - name: https
    hostname: "gitlab.example.com"
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - name: gitlab-credential
      options:
        gateway.istio.io/tls-terminate-mode: MUTUAL
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            kubernetes.io/metadata.name: apps-gitlab
EOF

部署 gitlab-shell

创建 gateway

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: mygateway
  namespace: istio-system
spec:
  gatewayClassName: istio
  listeners:
  - name: https
    hostname: "gitlab.example.com"
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - name: gitlab-credential
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            kubernetes.io/metadata.name: apps-gitlab
  - name: ssh
    port: 22
    protocol: TCP
    allowedRoutes:
      kinds:
        - kind: TCPRoute
EOF

在界面上可以看到暴露的 ssh 服务了

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19



# 注意这个tcp是为了给ssh服务暴露使用的
#接下里是绑定ssh端口和服务
cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
  name: tcp-app-1
  namespace: apps-gitlab
spec:
  parentRefs:
  - name: my-gateway
    sectionName: ssh
  rules:
  - backendRefs:
    - name: gitlab-gitlab-shell
      port: 22
EOF
tcproute.gateway.networking.k8s.io/tcp-app-1 created

这里需要注意一下，crd 的安装需要先做，不然会失败

1

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

注意因为使用了 tcproute 等实验特性需要在安装一下实验 crd

1

 kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/experimental-install.yaml

安装好后，实验一下 git 的 ssh 看看

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


  git push --set-upstream origin master
The authenticity of host 'gitlab.example.com (10.7.10.159)' can't be established.
ED25519 key fingerprint is SHA256:hFzSmmh6EAmyYL2oMEa6I7QWZK9F/HKerkFp0MihLTE.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'gitlab.example.com' (ED25519) to the list of known hosts.
Enumerating objects: 448, done.
Counting objects: 100% (448/448), done.
Delta compression using up to 6 threads
Compressing objects: 100% (244/244), done.
Writing objects: 100% (448/448), 6.28 MiB | 16.33 MiB/s, done.
Total 448 (delta 114), reused 448 (delta 114), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (114/114), done.
To gitlab.example.com:root/servicemesh_arch_istio.git
 * [new branch]      master -> master
branch 'master' set up to track 'origin/master'.

可以看到代码传到了 gitlab 上，同时也触发了 pipeline 点开构建的助手可以看到报错了这个问题的解决是 dns 解析失败，无法正常下载 docker。

查看 coredns 是否正常解析

用 busybox 的 nslookup 来测试 dns 是否正常

1

kubectl run -it --image 10.7.20.12:5000/busybox:1.28.4 test --restart=Never --rm /bin/sh

containerd 配置使用自定义的证书来安装 gitlab-runner 创建证书

1
2
3
4
5


openssl genrsa -out ca.key 2048
openssl req -new -x509 -days 365 -key ca.key -subj "/C=CN/ST=GD/L=SZ/O=Acme, Inc./CN=Acme Root CA" -out ca.crt

openssl req -newkey rsa:2048 -nodes -keyout server.key -subj "/C=CN/ST=GD/L=SZ/O=Acme, Inc./CN=*.example.com" -out server.csr
openssl x509 -req -extfile <(printf "subjectAltName=DNS:example.com,DNS:gitlab.example.com") -days 365 -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt

注意就是 y9ong 这个证书到 gw 上，以及所有的服务上接下来把之前的证书替换掉

1
2
3
4


kubectl delete -n istio-system secret gitlab-credential
     kubectl create -n istio-system secret tls gitlab-credential \
  --key=server.key \
  --cert=server.crt

然后访问 web 界面下载下来证书，然后把这个证书再安装到 secret 中提供给 gitlab-runner 使用，另外注意 gitlab-runner 的配置文件如下：

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269


registry:

  enabled: false

certmanager-issuer:

  #   # The email address to register certificates requested from Let's Encrypt.

  #   # Required if using Let's Encrypt.

  email: email@example.com

postgresql:

  install: false

certmanager:

  install: false

prometheus:

  install: false

nginx-ingress:

  enabled: false

global:

  image:

    registry: 10.7.20.12:5000

  edition: ce

  time_zone: Asia/Shanghai

  kas:

    enabled: false

  minio:

    enabled: false

  hosts:

    https: true

    domain: example.com

    gitlab:

      name: gitlab.example.com

    ingress:

      enabled: false

  psql:

    host: postgres-postgresql.dev

    database: gitlab_test

    username: gitlab_test

    password:

      secret: gitlab-postgresql-secret

      key: postgresql-password

  appConfig:

    lfs:

      bucket: gitlab-gitlfs

      connection:

        secret: gitlab-minio-secret

        key: connection

    artifacts:

      bucket: gitlab-artifacts

      connection:

        secret: gitlab-minio-secret

        key: connection

    uploads:

      bucket: gitlab-uploads

      connection:

        secret: gitlab-minio-secret

        key: connection

    packages:

      bucket: gitlab-packages

      connection:

        secret: gitlab-minio-secret

        key: connection

    dependencyProxy:

      enabled: true

      bucket: gitlab-dependency-proxy

      connection:

        secret: gitlab-minio-secret

        key: connection

    backups:

      bucket: gitlab-backups

      tmpBucket: gitlab-backups-tmp

    defaultProjectsFeatures:

      issues: true

      mergeRequests: true

      wiki: true

      snippets: true

      builds: true

      containerRegistry: false

gitlab:

  webservice:

    registry:

      enabled: false

    resources:

      requests:

        cpu: 150m

  sidekiq:

    registry:

      enabled: false

    resources:

      requests:

        cpu: 200m

  toolbox:

    backups:

      objectStorage:

        config:

          secret: gitlab-minio-secret

          key: config

  gitaly:

    persistence:

      size: 20Gi

gitlab-runner:

  runners:

    privileged: true

    config: |

      [[runners]]

        pre_get_sources_script = "echo '10.7.10.159 gitlab.example.com' >> /etc/hosts \n  git config --global http.sslVerify false\n"

        [runners.kubernetes]

          image = "debian:bullseye"

          privileged = true

          image_pull_secrets = []

        [runners.cache]

          Type = "s3"

          Path = "runners"

          Shared = true

          [runners.cache.s3]

            ServerAddress = "http://10.7.20.12:9000"

            AccessKey = "admin"

            SecretKey = "123456"

            BucketName = "gitlab-caches"

            BucketLocation = "cn-north-1"

            Insecure = true

  gitlabUrl: https://gitlab.example.com/

  runnerToken: ''

  runnerRegistrationToken: '61tguLPY3FYuYxNKjLqTHAiVO5pl7Wvy4kgOrUHPNC8jUuKZRx47oQ37Cr1cG4kR'

  certsSecretName: gitlab-runner-certs

  rbac:

    create: true

    rules:

      - apiGroups: [''] #"" indicates the core API group

        resources: ['*']

        verbs: ['*']

      - apiGroups: ['networking.k8s.io']

        resources: ['ingresses']

        verbs: ['*']

      - apiGroups: ['apps']

        resources: ['deployments']

        verbs: ['*']

    clusterWideAccess: true

    serviceAccountName: gitlab-runner

注意检验 CA 证书

1
2
3
4
5


curl -v -HHost:gitlab.example.com --resolve "gitlab.example.com:443:10.7.10.159" \
--cacert gitlab.example.com.crt "https://gitlab.example.com"

curl -v -HHost:gitlab.example.com --resolve "gitlab.example.com:443:10.7.10.159" \
--cacert gitlab.crt "https://gitlab.example.com"

查看是否把汗 SAN 的证书命令如下：

1
2
3
4


openssl x509 -in server.crt -text -noout | grep -A1 "Subject Alternative Name"

            X509v3 Subject Alternative Name:
                DNS:example.com, DNS:www.example.com

当看到上面的信息即可，不然会报错：

1
2
3
4



WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see https://gitlab.com/gitlab-org/gitlab/-/issues/380872
ERROR: Registering runner... failed                 runner=******** status=couldn't execute POST against https://gitlab/api/v4/runners: Post "https://gitlab/api/v4/runners": x509: certificate relies on legacy Common Name field, use SANs instead
PANIC: Failed to register the runner.

参考文档： https://stackoverflow.com/questions/75492168/gitlab-certificate-is-valid-for-domain-com-not-xxx-domain-com https://forum.gitlab.com/t/registering-specific-runner-causes-403-on-api-v4-runners/77890 https://techgrunt.io/gitlab-runner-on-lan-with-self-signed-certificates https://forum.gitlab.com/t/x509-certificate-relies-on-legacy-common-name-field-use-sans-instead/75478

gitlab runner 信任证书命令下载证书

1
2


# 下载证书
openssl s_client -showcerts -connect gitlab.test.helm.xuxiaowei.cn:443 -servername gitlab.test.helm.xuxiaowei.cn < /dev/null 2>/dev/null | openssl x509 -outform PEM > gitlab.test.helm.xuxiaowei.cn.crt

参考文档： https://gitlab-k8s.xuxiaowei.com.cn/gitlab-k8s/docs/helm/gitlab/gitlab-runner-trust-ssl.html

查看 helm 的镜像进行私有化

1

helm template gitlab-operator --set metrics.enabled=true | grep 'image: ' | sed 's/^.*image: //g' | sed "s/^\([\"']\)\(.*\)\1\$$/\2/g" | uniq >images.txt

gitlab 安装完后，使用了 bookstore1 项目来进行测试：使用 kankio 来构建镜像，这样就不需要 docker 特权了。在 gitlab-ci 中这样配置对应的镜像即可。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


build_payment:
  stage: image
  image:
    name: 10.7.20.12:5000/kaniko-project/executor:debug
    entrypoint: ['']
  script:
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/bookstore-servicemesh-domain-payment/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}/bookstore-servicemesh-domain-payment:${CI_COMMIT_TAG}"