Quick Start #

This document provides a quick introduction to using the Flink Kubernetes Operator. Readers of this document will be able to deploy the Flink operator itself and an example Flink job to a local Kubernetes installation.

Prerequisites #

We assume that you have a local installations of the following:

So that the kubectl and helm commands are available on your local system.

For docker we recommend that you have Docker Desktop installed and configured with at least 8GB of RAM. For kubernetes minikube is our choice, at the time of writing this we are using version v1.25.3 (end-to-end tests are using the same version). You can start a cluster with the following command:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


minikube start --kubernetes-version=v1.25.3
😄  minikube v1.28.0 on Darwin 13.0.1
✨  Automatically selected the docker driver. Other choices: hyperkit, ssh
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
🐳  Preparing Kubernetes v1.25.3 on Docker 20.10.20 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

We also recommend k9s as GUI for kubernetes, but it is optional for this quickstart guide.

Deploying the operator #

Install the certificate manager on your Kubernetes cluster to enable adding the webhook component (only needed once per Kubernetes cluster):

1

kubectl create -f https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml

In case the cert manager installation failed for any reason you can disable the webhook by passing --set webhook.create=false to the helm install command for the operator.

Now you can deploy the selected stable Flink Kubernetes Operator version using the included Helm chart:

1
2


helm repo add flink-operator-repo https://downloads.apache.org/flink/flink-kubernetes-operator-<OPERATOR-VERSION>/
helm install flink-kubernetes-operator flink-operator-repo/flink-kubernetes-operator

OPERATOR-VERSION 的版本为 1.9.0 To find the list of stable versions please visit https://flink.apache.org/downloads.html

The Helm chart by default points to the ghcr.io/apache/flink-kubernetes-operator image repository. If you have connectivity issues or prefer to use Dockerhub instead you can use --set image.repository=apache/flink-kubernetes-operator during installation.

You may verify your installation via kubectl and helm:

1
2
3
4
5
6
7


kubectl get pods
NAME READY STATUS RESTARTS AGE
flink-kubernetes-operator-fb5d46f94-ghd8b 2/2 Running 0 4m21s

helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
flink-kubernetes-operator default 1 2022-03-09 17 (tel:12022030917):39:55.461359 +0100 CET deployed flink-kubernetes-operator-1.10-SNAPSHOT 1.10-SNAPSHOT

Submitting a Flink job #

Once the operator is running as seen in the previous step you are ready to submit a Flink job:

1

kubectl create -f https://raw.githubusercontent.com/apache/flink-kubernetes-operator/release-1.8/examples/basic.yaml

You may follow the logs of your job, after a successful startup (which can take on the order of a minute in a fresh environment, seconds afterwards) you can:

1
2
3
4
5
6


kubectl logs -f deploy/basic-example

2022-03-11 21:46:04,458 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 206 (type=CHECKPOINT) @ 1647035164458 for job a12c04ac7f5d8418d8ab27931bf517b7.
2022-03-11 21:46:04,465 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 206 for job a12c04ac7f5d8418d8ab27931bf517b7 (28509 bytes, checkpointDuration=7 ms, finalizationTime=0 ms).
2022-03-11 21:46:06,458 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 207 (type=CHECKPOINT) @ 1647035166458 for job a12c04ac7f5d8418d8ab27931bf517b7.
2022-03-11 21:46:06,483 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 207 for job a12c04ac7f5d8418d8ab27931bf517b7 (28725 bytes, checkpointDuration=25 ms, finalizationTime=0 ms).

To expose the Flink Dashboard you may add a port-forward rule or look the ingress configuration options:

1

kubectl port-forward svc/basic-example-rest 8081

Now the Flink Dashboard is accessible at localhost:8081.

In order to stop your job and delete your FlinkDeployment you can:

1

kubectl delete flinkdeployment/basic-example

install minio

1

helm install minio -f values-test.yaml oci://registry-1.docker.io/bitnamicharts/minio

输出 o/bitnamicharts/minio Pulled: registry-1.docker.io/bitnamicharts/minio:14.6.19 Digest: sha256:2536f58a3618bf117615dc831338e09927816b914af518422ff6df83d6e74896 NAME: minio LAST DEPLOYED: Sun Jul 7 20:25:45 2024 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: minio CHART VERSION: 14.6.19 APP VERSION: 2024.7.4

** Please be patient while the chart is being deployed **

MinIO® can be accessed via port on the following DNS name from within your cluster:

minio.default.svc.cluster.local

To get your credentials run:

export ROOT_USER=$(kubectl get secret –namespace default minio -o jsonpath="{.data.root-user}" | base64 -d) export ROOT_PASSWORD=$(kubectl get secret –namespace default minio -o jsonpath="{.data.root-password}" | base64 -d)

To connect to your MinIO® server using a client:

Run a MinIO® Client pod and append the desired command (e.g. ‘admin info’):

kubectl run –namespace default minio-client
–rm –tty -i –restart=‘Never’
–env MINIO_SERVER_ROOT_USER=$ROOT_USER
–env MINIO_SERVER_ROOT_PASSWORD=$ROOT_PASSWORD
–env MINIO_SERVER_HOST=minio
–image docker.io/bitnami/minio-client:2023.12.23-debian-11-r1 – admin info minio

To access the MinIO® web UI:

Get the MinIO® URL:

You should be able to access your new MinIO® web UI through

http://minio.local/minio/

⚠ SECURITY WARNING: Original containers have been substituted. This Helm chart was designed, tested, and validated on multiple platforms using a specific set of Bitnami and Tanzu Application Catalog containers. Substituting other containers is likely to cause degraded security and performance, broken chart features, and missing environment variables.

Substituted images detected:

docker.io/bitnami/minio:2023.12.23-debian-11-r2
docker.io/bitnami/minio-client:2023.12.23-debian-11-r1
docker.io/bitnami/os-shell:11-debian-11-r93