ClickHouse Operator 快速入门指南在 Kubernetes 上部署、配置与管理 ClickHouse 集群【免费下载链接】clickhouse-operatorAltinity Kubernetes Operator for ClickHouse creates, configures and manages ClickHouse® clusters running on Kubernetes项目地址: https://gitcode.com/GitHub_Trending/cl/clickhouse-operator本文是基于 Altinity clickhouse-operator 仓库官方 docs/quick_start.md 编写的完整快速入门指南。内容涵盖前置条件、四种 Operator 安装方式、从源码构建、以及从单分片单副本 Hello World到持久卷 Pod 模板 自定义用户配置的阶梯式ClickHouseInstallation自定义资源示例。读完本文你将掌握在 Kubernetes 中通过声明式 YAML 创建、连接和定制 ClickHouse 集群的完整实操链路并能理解这些示例背后 Operator 的配置规范化实现原理。目录前置条件ClickHouse Operator 安装Operator 安装过程与验证从源码构建 ClickHouse Operator示例概览与命名空间准备Trivial 示例1 Shard 1 Replica连接 ClickHouse 数据库的三种方式简单持久卷示例自定义 DeploymentPod 与 VolumeClaim 模板自定义 ClickHouse 配置用户、密码、Profile 与文件从源码看配置规范化用户密码与 Secret 引用延伸阅读前置条件在开始之前请确保环境满足以下要求对应官方文档 docs/quick_start.md 的 Prerequisites 章节一个可用的 Kubernetes 集群且版本满足兼容性要求clickhouse-operator0.16.0 之前的版本兼容 Kubernetes1.16至1.22clickhouse-operator0.16.0 及之后的版本兼容 Kubernetes1.25及更高版本。正确配置的kubectl能够访问目标集群并且具备创建 Namespace、CRD、RBAC、Deployment、Service、ConfigMap 等资源的权限。curl安装脚本和在线清单的拉取依赖curl仓库中的安装脚本 deploy/operator-web-installer/clickhouse-operator-install.sh 在启动时会先执行is_curl_available检查若curl不可用则直接退出。ClickHouse Operator 安装官方提供了多条安装路径适用于不同网络环境与定制需求。以下四种方式均由仓库 deploy/ 目录下的清单与脚本支撑你可以直接使用仓库内文件也可以让脚本从远端拉取同一份清单。方式一最简安装kube-system 命名空间如果你接受将 Operator 安装到kube-system命名空间直接应用官方打包好的 Bundle 清单即可。该清单在仓库中的对应文件为 deploy/operator/clickhouse-operator-install-bundle.yamlkubectl apply -f deploy/operator/clickhouse-operator-install-bundle.yaml该 Bundle 由构建脚本 deploy/builder/cat-clickhouse-operator-install-yaml.sh 通过envsubst渲染多个分节模板拼接而成覆盖的内容包括CRD 分节ClickHouseInstallationCHI、ClickHouseInstallationTemplateCHIT、ClickHouseOperatorConfigurationCHOp config、ClickHouseKeeperInstallationCHKRBAC 分节ServiceAccount、ClusterRole / Role 与对应的 Binding环境分节注入 operator 配置与 ClickHouse XML 配置的多个 ConfigMap、默认 SecretDeployment 分节clickhouse-operator与metrics-exporter两个 DeploymentService 分节clickhouse-operator-metrics监控指标 Service。方式二Kubernetes 1.17 之前的兼容安装若集群版本低于1.17请使用基于v1beta1API 的兼容 Bundle对应仓库文件 deploy/operator/clickhouse-operator-install-bundle-v1beta1.yamlkubectl apply -f deploy/operator/clickhouse-operator-install-bundle-v1beta1.yaml方式三自定义安装参数推荐如果希望定制安装命名空间或Operator 镜像等参数请使用专门的安装脚本 deploy/operator-web-installer/clickhouse-operator-install.sh。例如安装到test-clickhouse-operator命名空间curl -s https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator-web-installer/clickhouse-operator-install.sh | OPERATOR_NAMESPACEtest-clickhouse-operator bash脚本会创建并安装 Operator 到显式指定的命名空间OPERATOR_NAMESPACEtest-clickhouse-operator安装过程中脚本会下载若干.yaml与.xml文件并把它们封装进 ConfigMap 后安装到目标命名空间。安装完成后Operator 只监听test-clickhouse-operator命名空间内的kind: ClickHouseInstallation自定义资源。如果不指定OPERATOR_NAMESPACEcurl -s https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator-web-installer/clickhouse-operator-install.sh | bash脚本默认安装到kube-system命名空间并且监听所有可用命名空间中的ClickHouseInstallation资源。该脚本见 deploy/operator-web-installer/clickhouse-operator-install.sh支持以下核心环境变量全部有默认值变量默认值说明OPERATOR_NAMESPACEkube-systemOperator 安装与监听未显式指定时的命名空间METRICS_EXPORTER_NAMESPACE与OPERATOR_NAMESPACE相同metrics-exporter 安装命名空间OPERATOR_VERSION读取远端release文件决定拉取哪一版本的安装模板OPERATOR_IMAGEaltinity/clickhouse-operator:${OPERATOR_VERSION}Operator 镜像OPERATOR_IMAGE_PULL_POLICYAlwaysOperator 镜像拉取策略METRICS_EXPORTER_IMAGEaltinity/metrics-exporter:${OPERATOR_VERSION}metrics-exporter 镜像UPDATEyes已存在旧部署时是否覆盖更新设为no则发现已部署即中止VALIDATE_YAMLtruekubectl apply --validate是否开启TEMPLATE版本化的clickhouse-operator-install-template.yaml远端地址可替换为自定义模板地址MANIFEST空直接指定一份现成清单文件跳过模板渲染仓库内还提供了一个封装脚本 deploy/operator/clickhouse-operator-install.sh它会读取仓库根目录 release 文件中的版本号并默认以kube-system命名空间执行上述安装流程。方式四离线 / 受保护环境安装在无法从互联网运行脚本的受保护环境中可以手动下载模板文件 deploy/operator/clickhouse-operator-install-template.yaml按需编辑后用kubectl应用。也可以直接使用下面的envsubst片段完成变量替换后应用#!/bin/bash # Namespace to install operator into OPERATOR_NAMESPACE${OPERATOR_NAMESPACE:-test-clickhouse-operator} # Namespace to install metrics-exporter into METRICS_EXPORTER_NAMESPACE${OPERATOR_NAMESPACE} # Operators docker image OPERATOR_IMAGE${OPERATOR_IMAGE:-altinity/clickhouse-operator:latest} # Metrics exporters docker image METRICS_EXPORTER_IMAGE${METRICS_EXPORTER_IMAGE:-altinity/metrics-exporter:latest} # Setup clickhouse-operator into specified namespace kubectl apply --namespace${OPERATOR_NAMESPACE} -f ( cat deploy/operator/clickhouse-operator-install-template.yaml | OPERATOR_IMAGE${OPERATOR_IMAGE} \ OPERATOR_NAMESPACE${OPERATOR_NAMESPACE} \ METRICS_EXPORTER_IMAGE${METRICS_EXPORTER_IMAGE} \ METRICS_EXPORTER_NAMESPACE${METRICS_EXPORTER_NAMESPACE} \ envsubst )说明上述示例将原文档中的远端 URL 替换为仓库内相对路径目的是一致地使用仓库内清单。若需从远端获取同版本模板可参考安装脚本中get_file的实现——它优先读取本地文件其次回退到curl/wget见 deploy/operator-web-installer/clickhouse-operator-install.sh。Operator 安装过程与验证以安装到test-clickhouse-operator命名空间为例脚本执行时的典型输出如下对应官方文档展示的安装过程Setup ClickHouse Operator into test-clickhouse-operator namespace namespace/test-clickhouse-operator created customresourcedefinition.apiextensions.k8s.io/clickhouseinstallations.clickhouse.altinity.com configured serviceaccount/clickhouse-operator created clusterrolebinding.rbac.authorization.k8s.io/clickhouse-operator configured service/clickhouse-operator-metrics created configmap/etc-clickhouse-operator-files created configmap/etc-clickhouse-operator-confd-files created configmap/etc-clickhouse-operator-configd-files created configmap/etc-clickhouse-operator-templatesd-files created configmap/etc-clickhouse-operator-usersd-files created deployment.apps/clickhouse-operator created从输出可以看出安装清单的组成部分CRD、ServiceAccount 与 RBAC、监控 Service、以及多个承载 Operator 配置和 ClickHouse XML 片段的 ConfigMap分别对应config/目录下的chi/conf.d、chi/config.d、chi/templates.d、chi/users.d等配置目录见 config/。验证 Operator 是否正常运行kubectl get pods -n test-clickhouse-operatorNAME READY STATUS RESTARTS AGE clickhouse-operator-5ddc6d858f-drppt 1/1 Running 0 1m看到 Pod 的STATUS为Running、READY为1/1即表示安装成功。从源码构建 ClickHouse Operator除了直接安装发布镜像也可以从源码构建。完整流程见官方文档 docs/operator_build_from_sources.md其要点如下构建二进制前置依赖go编译器与go mod包管理器拉取源码后切换到项目根目录执行go mod tidy确保依赖完整构建二进制go build -o ./clickhouse-operator cmd/operator/main.go生成的clickhouse-operator二进制只能运行在 Kubernetes 环境内部。仓库的入口源码位于 cmd/operator/main.go应用逻辑在 cmd/operator/app/main.go。仓库 dev/ 目录下也提供了便捷构建脚本例如 dev/go_build_operator.sh 会编译到dev/bin/clickhouse-operator。构建 Docker 镜像并在 Kubernetes 中使用该流程不需要 Go 工具链但需要 Kubernetes 与 Docker在项目根目录构建镜像docker build -t altinity/clickhouse-operator -f ./dockerfile/operator/Dockerfile ./将镜像导入本地集群以 minikube 为例docker save altinity/clickhouse-operator | (eval $(minikube docker-env) docker load)按本文前述方式安装 Operator并将OPERATOR_IMAGE指向你刚构建的镜像。示例概览与命名空间准备仓库 docs/chi-examples/ 提供了大量开箱即用的ClickHouseInstallation示例覆盖持久卷、副本分片、多集群、滚动更新、分区分布、SSL、监控等数十个场景。下文选取官方快速入门中的五个代表性示例展开。最佳实践为所有组件创建独立命名空间。先创建示例运行命名空间kubectl create namespace test-clickhouse-operatornamespace/test createdTrivial 示例1 Shard 1 Replica首先部署最简的1 分片 1 副本示例清单见 docs/chi-examples/01-simple-layout-01-1shard-1repl.yaml。WARNING该示例没有持久化存储仅适合用作 Hello, world! 级别的功能验证切勿用于任何真实业务。kubectl apply -n test-clickhouse-operator -f docs/chi-examples/01-simple-layout-01-1shard-1repl.yamlclickhouseinstallation.clickhouse.altinity.com/simple-01 created清单内容非常直观定义了一个名为simple的单副本集群同时声明了一个测试用户apiVersion: clickhouse.altinity.com/v1 kind: ClickHouseInstallation metadata: name: simple-01 spec: configuration: users: # printf test_password | sha256sum test_user/password_sha256_hex: 10a6e6cc8311a3e2bcc09bf6c199adecd5dd59408c343e926b129c4914f3cb01 # to allow access outside from kubernetes test_user/networks/ip: - 0.0.0.0/0 clusters: - name: simple关键点解读kind: ClickHouseInstallation是 Operator 监听的自定义资源apiVersion: clickhouse.altinity.com/v1对应安装阶段创建的 CRD用户密码以password_sha256_hex形式提供即printf test_password | sha256sum的输出避免在清单中暴露明文test_user/networks/ip: [0.0.0.0/0]允许从 Kubernetes 集群外部访问该用户未显式指定layout.shardsCount/replicasCount时Operator 默认生成 1 分片 1 副本的布局更完整的字段说明可参考 docs/custom_resource_explained.md。集群创建后需要做两项检查。首先确认 Pod 状态kubectl get pods -n test-clickhouse-operatorNAME READY STATUS RESTARTS AGE chi-b3d29f-a242-0-0-0 1/1 Running 0 10m其次确认 Operator 创建的服务kubectl get service -n test-clickhouse-operatorNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE chi-b3d29f-a242-0-0 ClusterIP None none 8123/TCP,9000/TCP,9009/TCP 11m clickhouse-example-01 LoadBalancer 100.64.167.170 abc-123.us-east-1.elb.amazonaws.com 8123:30954/TCP,9000:32697/TCP 11m可以看到 Operator 为集群生成了两类 Servicechi-b3d29f-a242-0-0ClusterIP为None的 Headless Service暴露 ClickHouse 的8123HTTP、9000Native TCP、9009服务器间通信端口用于集群内部 Pod 间发现与访问clickhouse-example-01LoadBalancer类型作为对外入口示例中映射到 AWS ELB将8123与9000端口暴露到集群外。看到 Pod 处于Running、Service 已创建即表示 ClickHouse 已经成功运行连接 ClickHouse 数据库的三种方式集群就绪后可以通过以下三种方式连接数据库对应官方文档 Connect to ClickHouse Database 章节方式 1通过 EXTERNAL-IP 直连如果上一步kubectl get service输出了EXTERNAL-IP示例中为abc-123.us-east-1.elb.amazonaws.com可以直接连接clickhouse-client -h abc-123.us-east-1.elb.amazonaws.com -u test_user --password test_passwordClickHouse client version 18.14.12. Connecting to abc-123.us-east-1.elb.amazonaws.com:9000. Connected to ClickHouse server version 19.4.3 revision 54416.方式 2从集群内部访问如果没有EXTERNAL-IP可以从 Kubernetes 集群内部通过kubectl exec进入 ClickHouse Pod 执行客户端kubectl -n test-clickhouse-operator exec -it chi-b3d29f-a242-0-0-0 -- clickhouse-clientClickHouse client version 19.4.3.11. Connecting to localhost:9000 as user default. Connected to ClickHouse server version 19.4.3 revision 54416.方式 3kubectl port-forward 端口转发如果本机装有clickhouse-client还可以用端口转发将 Pod 的9000端口映射到本地kubectl -n test-clickhouse-operator port-forward chi-b3d29f-a242-0-0-0 9000:9000 clickhouse-clientClickHouse client version 19.4.3.11. Connecting to localhost:9000 as user default. Connected to ClickHouse server version 19.4.3 revision 54416.简单持久卷示例Trivial 示例没有持久化存储一旦 Pod 重建数据即丢失。在支持**动态卷供应Dynamic Volume Provisioning**的环境例如 AWS中可以使用 PersistentVolumeClaim 为数据与日志分别挂载持久卷。清单见 docs/chi-examples/03-persistent-volume-01-default-volume.yamlapiVersion: clickhouse.altinity.com/v1 kind: ClickHouseInstallation metadata: name: pv-simple spec: defaults: templates: dataVolumeClaimTemplate:>apiVersion: clickhouse.altinity.com/v1 kind: ClickHouseInstallation metadata: name: pv-log spec: configuration: clusters: - name: deployment-pv # Templates are specified for this cluster explicitly templates: podTemplate: pod-template-with-volumes layout: shardsCount: 2 replicasCount: 2 templates: podTemplates: - name: pod-template-with-volumes spec: containers: - name: clickhouse image: clickhouse/clickhouse-server:24.8 volumeMounts: - name:>apiVersion: v1 kind: Secret metadata: name: clickhouse-credentials type: Opaque stringData: testpwduser1: password testpwduser2: 65e84be33532fb784c48129675f9eff3a682b27168c0ea744b2cf58ee02337c5 testpwduser3: 8bd66e4932b4968ec111da24d7e42d399a05cb90bf96f587c3fa191c56c401f8 --- apiVersion: clickhouse.altinity.com/v1 kind: ClickHouseInstallation metadata: name: settings-01 spec: configuration: users: # test user has password specified, while admin user has password_sha256_hex specified test/password: qwerty test/networks/ip: - 127.0.0.1/32 - 192.168.74.1/24 test/profile: test_profile test/quota: test_quota test/allow_databases/database: - dbname1 - dbname2 - dbname3 # reference to namespace/name/field in the secret with plain password testpwduser1/k8s_secret_password: dev/clickhouse-credentials/testpwduser1 # reference to the same namespace as operator is running in/name/field in the secret with sha256 password testpwduser2/k8s_secret_password_sha256_hex: clickhouse-credentials/testpwduser2 testpwduser3/k8s_secret_password_double_sha1_hex: clickhouse-credentials/testpwduser3 # admin use has password_sha256_hex so actual password value is not published admin/password_sha256_hex: 8bd66e4932b4968ec111da24d7e42d399a05cb90bf96f587c3fa191c56c401f8 admin/networks/ip: 127.0.0.1/32 admin/profile: default admin/quota: default # readonly user has password field specified, not password_sha256_hex as admin user above readonly/password: readonly_password readonly/profile: readonly readonly/quota: default profiles: test_profile/max_memory_usage: 1000000000 test_profile/readonly: 1 readonly/readonly: 1 quotas: test_quota/interval/duration: 3600 settings: compression/case/method: zstd disable_internal_dns_cache: 1 files: dict1.xml: | yandex !-- ref to file /etc/clickhouse-data/config.d/source1.csv -- /yandex source1.csv: | a1,b1,c1,d1 a2,b2,c2,d2 clusters: - name: standard layout: shardsCount: 1 replicasCount: 1用户users配置要点多种密码形式test用户直接使用明文password: qwertyadmin用户使用password_sha256_hexSHA-256 十六进制摘要因此明文密码永远不会出现在清单与配置中readonly用户同样使用明文网络访问控制test/networks/ip限制用户可访问的网段127.0.0.1/32与192.168.74.1/24admin/networks/ip同样可配置Profile 与 Quota 绑定test/profile: test_profile、test/quota: test_quota将用户与下面定义的 Profile、Quota 关联admin绑定内置的default数据库白名单test/allow_databases/database声明该用户允许访问的数据库列表dbname1、dbname2、dbname3引用 Kubernetes Secret 中的密码testpwduser1/k8s_secret_password: dev/clickhouse-credentials/testpwduser1—— 引用dev命名空间下clickhouse-credentialsSecret 的testpwduser1字段作为明文密码testpwduser2/k8s_secret_password_sha256_hex: clickhouse-credentials/testpwduser2—— 引用Operator 所在命名空间下同一 Secret 的字段作为SHA-256 摘要密码testpwduser3/k8s_secret_password_double_sha1_hex: clickhouse-credentials/testpwduser3—— 引用字段作为double SHA-1 摘要密码。Profile、Quota 与全局设置profilestest_profile/max_memory_usage: 1000000000限制单查询内存上限约 1GBtest_profile/readonly: 1与readonly/readonly: 1将用户设为只读readonly1禁止修改设置quotastest_quota/interval/duration: 3600定义配额统计周期为 3600 秒settingscompression/case/method: zstd指定压缩算法disable_internal_dns_cache: 1禁用 ClickHouse 内部 DNS 缓存。附加文件filesfiles段允许直接向 Pod 注入额外文件例如字典定义dict1.xml可进一步引用/etc/clickhouse-data/config.d/source1.csv以及数据文件source1.csv。Operator 会把这些文件放置到 ClickHouse 配置目录中从而实现字典、扩展配置等能力的声明式管理。从源码看配置规范化用户密码与 Secret 引用上述扁平化 key: value 语法以及k8s_secret_password系列字段并非仅在 YAML 层面做字符串拼接而是由 Operator 的**配置规范化Normalizer**阶段统一处理。相关实现位于 pkg/model/chi/normalizer/normalizer-configuration-user.go函数normalizeConfigurationUserPassword会依次执行ReplaceSettingsFieldWithSecretFieldValue把k8s_secret_password、k8s_secret_password_double_sha1_hex、k8s_secret_password_sha256_hex解析为对应的密码字段password、password_double_sha1_hex、password_sha256_hex即从 Kubernetes Secret 中取值并回填到用户配置随后按优先级收敛密码字段password_double_sha1_hex优先级最高存在时删除password_sha256_hex与password其次是password_sha256_hex存在时删除明文password如果最终只有明文password规范化逻辑会计算其 SHA-256 摘要写入password_sha256_hex并清理其余密码字段确保下发到 ClickHouse 的 users.xml 中始终以摘要形式呈现密码。这意味着无论你在 CHI 清单中写明文密码、SHA-256 摘要还是 Secret 引用最终落到 ClickHouse 配置的都是经过规范化、去重且安全的密码形态——这正是上面05-settings-01-overview.yaml中各种密码写法的底层原理。延伸阅读更多ClickHouseInstallation示例浏览 docs/chi-examples/ 目录其中包含持久卷扩展、副本分片布局、滚动更新、多集群、地域分布、SSL、监控等 100 余个场景清单自定义资源完整字段说明docs/custom_resource_explained.mdOperator 配置详解命名空间、监听范围、watch 行为等docs/operator_configuration.md安装细节与命名空间隔离策略docs/operator_installation_details.md镜像构建与发布流程docs/operator_build_from_sources.md监控与指标采集安装清单中已包含clickhouse-operator-metricsServicedocs/monitoring_setup.md 与 docs/prometheus_setup.md升级已有 Operator 部署官方文档 docs/operator_upgrade.md。【免费下载链接】clickhouse-operatorAltinity Kubernetes Operator for ClickHouse creates, configures and manages ClickHouse® clusters running on Kubernetes项目地址: https://gitcode.com/GitHub_Trending/cl/clickhouse-operator创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考