部署完的效果：

部署的环境版本，我们在 k9s 上看一下

组件版本：

k8s: v1.28.4

gitlab: v16.7

gitlab helm chart : 7.7.0 v16.7.0 GitLab is the most comprehensive AI-powered Dev…

使用的命令

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


helm search repo gitlab
WARNING: Repo "flink-operator-repo" is corrupt or missing. Try 'helm repo update'.
WARNING: open /home/xfhuang/.cache/helm/repository/flink-operator-repo-index.yaml: no such file or directory
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
gitlab/gitlab                   7.7.0           v16.7.0         GitLab is the most comprehensive AI-powered Dev...
gitlab/gitlab-agent             1.22.0          v16.7.0         GitLab Agent for Kubernetes is a way to integra...
gitlab/gitlab-omnibus           0.1.37                          GitLab Omnibus all-in-one bundle
gitlab/gitlab-runner            0.60.0          16.7.0          GitLab Runner
gitlab/gitlab-zoekt             1.0.0           0.2.0           A Helm chart for deploying Zoekt as a search en...
gitlab/kubernetes-gitlab-demo   0.1.29                          GitLab running on Kubernetes suitable for demos
gitlab/apparmor                 0.2.0           0.1.0           AppArmor profile loader for Kubernetes
gitlab/auto-deploy-app          0.8.1                           GitLab's Auto-deploy Helm Chart
gitlab/elastic-stack            3.0.0           7.6.2           A Helm chart for Elastic Stack
gitlab/fluentd-elasticsearch    6.2.8           2.8.0           A Fluentd Helm chart for Kubernetes with Elasti...
gitlab/knative                  0.10.0          0.9.0           A Helm chart for Knative
gitlab/plantuml                 0.1.17          1.0             PlantUML server

配置过程

准备工作

添加 Helm 仓库

添加用于安装GitLab的 Helm 仓库。

1

helm repo add gitlab https://charts.gitlab.io/ 2helm repo update

创建命名空间

本次演练中将 GitLab 安装至apps-gitlab命名空间，可根据需要替换。
1

1kubectl create namespace apps-gitlab

创建 TLS 证书Secret

从已准备好的证书key文件和crt文件创建Secret。

1
2
3
4


mkdir example_certs2
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=choral.io' -keyout example_certs2/choral.io.key -out example_certs2/choral.io.crt
openssl req -out example_certs2/local.choral.io.csr -newkey rsa:2048 -nodes -keyout example_certs2/local.choral.io.key -subj "/CN=local.choral.io/O=httpbin organization"
openssl x509 -req -sha256 -days 365 -CA example_certs2/choral.io.crt -CAkey example_certs2/choral.io.key -set_serial 0 -in example_certs2/local.choral.io.csr -out example_certs2/local.choral.io.crt

1
2


kubectl create secret tls local-choral-io-tls -n apps-gitlab \
  --key=example_certs2/local.choral.io.key --cert=example_certs2/local.choral.io.crt

创建Traefik EntryPoint

创建一个新的Traefik EntryPoint，用于提供对 GitLab Shell SSH 协议的访问。

首先，导出当前Traefik部署的配置文件。

1
2

helm get values --output yaml --namespace kube-system traefik > helm-traefik.yaml

添加新的参数，更新 Traefik 部署。

1
2
3
4
5
6
7
8
9


# ports.git-ssh.expose=false 禁用公开访问 稍后会手动创建用于访问该端口的负载均衡器
# ports.git-ssh.port=8022 指定绑定端口
# ports.git-ssh.protocol=TCP 指定绑定协议
helm upgrade --install --namespace kube-system \
--values helm-traefik.yaml \
--set ports.git-ssh.expose=false \
--set ports.git-ssh.port=8022 \
--set ports.git-ssh.protocol=TCP \
traefik traefik/traefik

创建负载均衡器

创建一个新的LoadBalancer类型的Service，用于提供对22、80和443端口的访问。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


cat <<EOF | kubectl apply -f - > /dev/null
apiVersion: v1
kind: Service
metadata:
  name: traefik-git
  namespace: kube-system
  labels:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/name: traefik
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/name: traefik
  ports:
  - name: ssh
    port: 22
    protocol: TCP
    targetPort: 8022
  - name: web
    port: 80
    protocol: TCP
    targetPort: web
  - name: websecure
    port: 443
    protocol: TCP
    targetPort: websecure
EOF

安装 GitLab

安装 postgres

1

	helm  install postgres bitnami/postgresql --set  globalstorageClass=local-path -n dev --create-namespace

创建PostgreSQL密码Secret

1
2


kubectl create secret -n apps-gitlab generic gitlab-postgresql-secret \
  --from-literal=postgresql-password=123456

创建MinIO认证凭据Secret 创建gitlab-minio.yaml。

```yaml
provider: AWS
region: cn-north-1
aws_access_key_id: admin
aws_secret_access_key: minio1234
aws_signature_version: 4
host: minio.minio.svc.cluster.local
endpoint: "http://minio.minio.svc.cluster.local:9000"
path_style: true
```

创建`gitlab-minio.config`。

```cfg
[default]
host_base = minio.minio.svc.cluster.local:9000
host_bucket = minio.minio.svc.cluster.local:9000
use_https = False
signature_v2 = False
access_key = admin
secret_key = minio1234
bucket_location = cn-north-1
multipart_chunk_size_mb = 128
```

创建包含上述两个文件的[Secret](https://kubernetes.io/docs/concepts/configuration/secret/)对象。

```sh

kubectl create secret -n apps-gitlab generic gitlab-minio-secret –from-file=connection=gitlab-minio.yaml –from-file=config=gitlab-minio.config

创建所需的 Buckets

1

mc mb choral-local/gitlab-{artifacts,backups,backups-tmp,caches,dependency-proxy,gitlfs,packages,uploads}

1
2
3
4
5
6
7
8


I have no name!@minio-67bbd967b9-f5fdh:/opt/bitnami/minio-client$ mc mb choral-local/gitlab-{artifacts,backups,backups-tmp,caches,dependency-proxy,gitlfs,packages,uploads}Bucket created successfully `choral-local/gitlab-artifacts`.
Bucket created successfully `choral-local/gitlab-backups`.
Bucket created successfully `choral-local/gitlab-backups-tmp`.
Bucket created successfully `choral-local/gitlab-caches`.
Bucket created successfully `choral-local/gitlab-dependency-proxy`.
Bucket created successfully `choral-local/gitlab-gitlfs`.
Bucket created successfully `choral-local/gitlab-packages`.
Bucket created successfully `choral-local/gitlab-uploads`.

创建helm-gitlab.yaml

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115


registry:
  enabled: false
certmanager-issuer:
  #   # The email address to register certificates requested from Let's Encrypt.
  #   # Required if using Let's Encrypt.
  email: email@example.com
postgresql:
  install: false
certmanager:
  install: false
prometheus:
  install: false
nginx-ingress:
  enabled: false
global:
  edition: ce
  time_zone: Asia/Shanghai
  kas:
    enabled: false
  minio:
    enabled: false
  hosts:
    https: true
    domain: local.choral.io
    gitlab:
      name: code.local.choral.io
  ingress:
    enabled: false
  psql:
    host: postgres-postgresql.dev.svc.cluster.local
    database: gitlab_production
    username: gitlab
    password:
      secret: gitlab-postgresql-secret
      key: postgresql-password
  appConfig:
    lfs:
      bucket: gitlab-gitlfs
      connection:
        secret: gitlab-minio-secret
        key: connection
    artifacts:
      bucket: gitlab-artifacts
      connection:
        secret: gitlab-minio-secret
        key: connection
    uploads:
      bucket: gitlab-uploads
      connection:
        secret: gitlab-minio-secret
        key: connection
    packages:
      bucket: gitlab-packages
      connection:
        secret: gitlab-minio-secret
        key: connection
    dependencyProxy:
      enabled: true
      bucket: gitlab-dependency-proxy
      connection:
        secret: gitlab-minio-secret
        key: connection
    backups:
      bucket: gitlab-backups
      tmpBucket: gitlab-backups-tmp
    defaultProjectsFeatures:
      issues: true
      mergeRequests: true
      wiki: true
      snippets: true
      builds: true
      containerRegistry: false
gitlab:
  webservice:
    registry:
      enabled: false
    resources:
      requests:
        cpu: 150m
  sidekiq:
    registry:
      enabled: false
    resources:
      requests:
        cpu: 200m
  toolbox:
    backups:
      objectStorage:
        config:
          secret: gitlab-minio-secret
          key: config
  gitaly:
    persistence:
      size: 20Gi
gitlab-runner:
  runners:
    privileged: true
    config: |
      [[runners]]
        [runners.kubernetes]
          image = "debian:bullseye"
          privileged = true
          image_pull_secrets = []
        [runners.cache]
          Type = "s3"
          Path = "runners"
          Shared = true
          [runners.cache.s3]
            ServerAddress = "http://minio.minio.svc.cluster.local:9000"
            AccessKey = "admin"
            SecretKey = "minio123"
            BucketName = "gitlab-caches"
            BucketLocation = "cn-north-1"
            Insecure = false      
  # gitlabUrl: http://gitlab-webservice-default:8181

安装 GitLab

使用上一步中创建的配置文件安装 GitLab。

1
2
3


helm upgrade --install gitlab --namespace apps-gitlab \
--values helm-gitlab.yaml \
gitlab/gitlab

采用以上方式的部署的 gitlab，在 webservice 一直报错，进入报错 pod 查看，发现是数据库的问题、

1
2
3


Checking: main
Database Schema - main (gitlab) - current: 0, codebase: 20220413075921
NOTICE: Database has not been initialized yet.

针对这个的解决办法为：修改 psql 的配置如下

1
2
3
4
5
6
7


  psql:
    host: postgres-postgresql.dev.svc.cluster.local
    database: gitlab_test
    username: gitlab_test
    password:
      secret: gitlab-postgresql-secret
      key: postgresql-password

#postgres 密码 export POSTGRES_PASSWORD=$(kubectl get secret –namespace apps-gitlab postgres-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d)

发现 psql，改成数据库名和用户名都改为 gitlab_test

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


 🌈🌈🌈     kubectl run postgres-postgresql-client --rm --tty -i --restart='Never' --namespace dev --image docker.io/bitnami/postgresql:16.3.0-debian-12-r18   --env="PGPASSWORD=$POSTGRES_PASSWORD" \
      --command -- psql --host postgres-postgresql -U postgres -d postgres -p 5432
If you don't see a command prompt, try pressing enter.

postgres=# CREATE USER gitlab_test WITH PASSWORD '123456';
CREATE ROLE
postgres=# CREATE DATABASE gitlab_production OWNER gitlab_test;
ERROR:  database "gitlab_production" already exists
postgres=# CREATE DATABASE gitlab_test OWNER gitlab_test;
CREATE DATABASE
postgres=# ALTER USER gitlab_test with createdb;
ALTER ROLE
postgres=# \c gitlab_production
You are now connected to database "gitlab_production" as user "postgres".
gitlab_production=# CREATE EXTENSION pg_trgm;
N btree_gist;ERROR:  no schema has been selected to create in
gitlab_production=# CREATE EXTENSION btree_gist;
ERROR:  no schema has been selected to create in
gitlab_production=# GRANT ALL PRIVILEGES ON DATABASE gitlab_test to gitlab_test;
IVILEGES ON aGRANT
gitlab_production=# GRANT ALL PRIVILEGES ON all tables in schema public TO gitlab_test;
ERROR:  schema "public" does not exist
gitlab_production=# \c gitlab_test
You are now connected to database "gitlab_test" as user "postgres".
gitlab_test=# CREATE EXTENSION pg_trgm;
NSION btree_gist;CREATE EXTENSION
gitlab_test=# CREATE EXTENSION btree_gist;
CREATE EXTENSION
gitlab_test=# GRANT ALL PRIVILEGES ON DATABASE gitlab_test to gitlab_test;
LL PRIVILEGESGRANT
gitlab_test=# GRANT ALL PRIVILEGES ON all tables in schema public TO gitlab_test;
GRANT
gitlab_test=# \l
                                                                List of databases
       Name        |    Owner    | Encoding | Locale Provider |   Collate   |    Ctype    | ICU Locale | ICU Rules |      Access privileges
-------------------+-------------+----------+-----------------+-------------+-------------+------------+-----------+-----------------------------
 gitlab_production | gitlab      | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =Tc/gitlab                 +
                   |             |          |                 |             |             |            |           | gitlab=CTc/gitlab
 gitlab_test       | gitlab_test | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =Tc/gitlab_test            +
                   |             |          |                 |             |             |            |           | gitlab_test=CTc/gitlab_test
 postgres          | postgres    | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           |
 template0         | postgres    | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres                +
                   |             |          |                 |             |             |            |           | postgres=CTc/postgres
 template1         | postgres    | UTF8     | libc            | en_US.UTF-8 | en_US.UTF-8 |            |           | =c/postgres                +
                   |             |          |                 |             |             |            |           | postgres=CTc/postgres
(5 rows)

创建 gitlab_test 后，再执行升级 gitlab 后，服务正常了。

配置 Ingress 入口规则

创建用于提供 http 和 https 协议访问 Ingress Route 对象。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


cat <<EOF | kubectl apply -f -
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: gitlab-http-web
  namespace: apps-gitlab
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(\`code.local.choral.io\`)
      kind: Rule
      services:
        - name: gitlab-webservice-default
          kind: Service
          port: 8181
  tls:
    secretName: local-choral-io-tls
EOF

创建用于提供SSH协议访问的Ingress Route TCP对象。

```sh
cat <<EOF | kubectl apply -f -
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
  name: gitlab-ssh-shell
  namespace: apps-gitlab
spec:
  entryPoints:
    - git-ssh
  routes:
    - match: HostSNI(\`*\`)
      services:
        - name: gitlab-gitlab-shell
          port: 22
EOF

获取root用户初始密码
1

kubectl get secret --namespace apps-gitlab gitlab-gitlab-initial-root-password \ -o jsonpath="{.data.password}" | base64 -d
获取密码之后可以使用root用户名和密码登录 GitLab（本次演练中访问地址为https://code.local.choral.io/）。

注意这里获取到的密码登不进去，所以采用了，注册来换密码的方式来解决的，注册完成后，进入 postgres 的表中，修改 user 表的 status 为 active 状态即可。

1
2
3
4
5
6
7


查看gitlab-runner

![image-20240111114955837](https://cdn.jsdelivr.net/gh/huangxiaofeng10047/blogimage@main/img/image-20240111114955837.png)

![image-20240111115030059](https://cdn.jsdelivr.net/gh/huangxiaofeng10047/blogimage@main/img/image-20240111115030059.png)

具体.gitlab-ci.yml

设置执行镜像

image: busybox:latest

整个 pipeline 有两个 stage

stages:

build
test

定义全局缓存，缓存的 key 来自分支信息，缓存位置是 vendor 文件夹

cache: key: ${CI_COMMIT_REF_SLUG} paths: - vendor/

before_script:

export
echo “Before script section”

after_script:

echo “After script section”

build1: stage: build tags: - k8s script: - echo “将内容写入缓存” - mkdir -p vendor - echo “build” > vendor/hello.txt

test1: stage: test tags: - k8s script: - echo “从缓存读取内容” - cat vendor/hello.txt

1
2

发现gitlab-runner无法下载源码，通过改coredns，来添加hosts也无法解决问题。

kubectl edit cm -n kube-system coredns #修改

Please edit the object below. Lines beginning with a ‘#’ will be ignored,

and an empty file will abort the edit. If an error occurs while saving this file will be

reopened with the relevant failures.

apiVersion: v1 data: Corefile: | .:53 { errors { } health { lameduck 5s } ready hosts { 10.7.20.242 code.local.choral.io # 这个是添加的域名解释 fallthrough } kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp max_concurrent 1000 } cache 30

    loop
    reload

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19



重试多次无法实现

参考文档为https://blog.51cto.com/u_10272167/2730629

## 最终发现问题可以通过pre_get_sources_script来解决

![image-20240111150817958](https://cdn.jsdelivr.net/gh/huangxiaofeng10047/blogimage@main/img/image-20240111150817958.png)

再次查看runner的job跑的怎么样

![image-20240111150854530](https://cdn.jsdelivr.net/gh/huangxiaofeng10047/blogimage@main/img/image-20240111151111331.png)

## 参考资料

-   [https://docs.gitlab.com/charts/](https://docs.gitlab.com/charts/)
-   https://tiscs.choral.io/notes/k8s-gitlab/
-   https://i4t.com/4373.html
今天遇到报错了

启动 gitlab-runner 报错,通过 k9s 查看报错

1
2
3
4
5
6


 rpc error: code = Unknown desc = failed to exec in container: fa │
│ iled to start exec "4adc11d51a83001df4f04ddaae5ebe005b7cd32167a49d5590606b95766d86e7": OCI runtime exec failed: exec failed: unable to start │
│  container process: error executing setns process: exit status 1: unknown                                                                    │
│   Warning  Unhealthy  102s                kubelet  Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: fa │
│ iled to start exec "45be44a475a9014dcdf3b8829eb288bed7a8a876abad26ab43f0a89060e292bf": OCI runtime exec failed: exec failed: unable to start │
│  container process: error executing setns process: exit status 1: unknown

![[image-20240701161524963.png]]

这是重启以后看到的 ulimit -n 确实变更了。 ![[image-20240701161631034.png]]