Kubernetes Autoscaler Balancer 提案解析用 CRD 控制器在多域部署间统一分配与自动伸缩 Pod 副本【免费下载链接】autoscalerAutoscaling components for Kubernetes项目地址: https://gitcode.com/GitHub_Trending/au/autoscaler导读本文基于 Kubernetes Autoscaler 仓库au/autoscaler中 Balancer 组件的 KEP 设计文档balancer/proposals/balancer.md展开系统讲解 Balancer 的诞生背景、核心 API 设计、两种副本分配策略Priority 与 Proportional以及 Fallback 机制并结合仓库内的 CRD 定义、控制器源码与 Nginx 实战示例给出可直接落地的部署与配置方案。读完本文你将理解如何在多可用区、混合 spot/on-demand 节点等多域场景下用单个 Balancer 对象同时管理多个 Deployment 的副本分布与水平伸缩并掌握其底层策略算法与控制器工作流。背景多域部署下的副本分配难题运行 Kubernetes 工作负载时用户常常希望把 Pod 分散部署到多个域domain中同时保持副本数量的均衡与自动伸缩。这些域可能包括同一区域内的多个云厂商可用区zone保证即使某个可用区故障应用仍能继续运行不同类型的 Kubernetes 节点如 spot/preemptible竞价/抢占式实例与按需实例、不同机器族等。一个单一的 Kubernetes Deployment 要么把放置完全交给调度器极可能导致所有 Pod 涌入同一个域要么只专注于单一域无法达成跨多域分布的目标。PodTopologySpreading 的不足PodTopologySpreading 解决了部分问题但并不彻底它只支持均匀分布even spreading一旦部署发生倾斜就不会主动再平衡它配合 skew 和/或ScheduleAnyway标志本质上只是一个提示当倾斜放置可行且被允许时Cluster Autoscaler 不会被触发用户最终会得到一个倾斜的部署若用户配置严格的拓扑分布约束strict pod topology spreading那么在出问题时部署不会把 Pod 迁移到可用域同时由于可用域过于倾斜部署的扩容也会被完全阻塞。多 Deployment 多 HPA 的不稳定如果追求完全灵活性唯一办法是部署多个 Deployment各自瞄准不同域。但这种方案带来一个重大问题如何一致地对多个 Deployment 进行自动伸缩最简单的思路——为每个 Deployment 配置独立的 HPA——并不稳定由于负载不同、竞态条件等因素可能出现某些域扩容、另一些域缩容的情况。由于 HPA 与 Deployment 之间没有任何关联倾斜状态不会自动修复最终也许能到达一个半均衡状态但不保证一定如此。Balancer多域副本分配的统一控制器正是上述痛点催生了 Balancer 组件。它需要一个能完成以下四件事的组件保持多个 Deployment 对齐例如维持各 Deployment 之间 Pod 数量的固定比例或者把副本全部放入第一个域、溢出部分放入第二个域依此类推响应单个 Deployment 的问题无论是某个可用区宕机还是 spot/preemptible 虚拟机短缺主动尝试再平衡逐步回到期望的布局以单一目标对全部 Deployment 进行自动伸缩同时维持放置策略。什么是 BalancerBalancer 是一个独立控制器运行在用户空间必要时也可运行在控制平面对外暴露一个同样名为 Balancer 的CRD API 对象。每个 Balancer 对象持有对多个 Deployment或其他暴露 Scale 子资源的 Pod 控制对象的引用。Balancer 周期性检查每个目标内运行中与有问题的 Pod 数量与期望副本数、约束及策略进行比较并在某个目标运行过多或过少副本时调整其副本数。为了让 Balancer 自身能够成为HPA 的伸缩目标Balancer 同样对外暴露Scale 子资源——这一设计在 balancer/deploy/crd.yaml 中体现为subresources.scale声明specReplicasPath: .spec.replicas、statusReplicasPath: .status.replicas、labelSelectorPath: .status.selectorHPA 可以直接把scaleTargetRef指向 Balancer。Balancer API 详解KEP 文档给出了完整的 API 类型定义以下是其核心 Go 结构该定义在仓库中的实际实现见 balancer/pkg/apis/balancer.x-k8s.io/v1alpha1/types.go// Balancer is an object used to automatically keep the desired number of // replicas (pods) distributed among the specified set of targets (deployments // or other objects that expose the Scale subresource). type Balancer struct { metav1.TypeMeta // Standard object metadata. // optional metav1.ObjectMeta // Specification of the Balancer behavior. Spec BalancerSpec // Current information about the Balancer. // optional Status BalancerStatus } // BalancerSpec is the specification of the Balancer behavior. type BalancerSpec struct { // Targets is a list of targets between which Balancer tries to distribute // replicas. Targets []BalancerTarget // Replicas is the number of pods that should be distributed among the // declared targets according to the specified policy. Replicas int32 // Selector that groups the pods from all targets together (and only those). // Ideally it should match the selector used by the Service built on top of the // Balancer. All pods selectable by targets selector must match to this selector, // however targets selector dont have to be a superset of this one (although // it is recommended). Selector metav1.LabelSelector // Policy defines how the balancer should distribute replicas among targets. Policy BalancerPolicy } // BalancerTarget is the declaration of one of the targets between which the balancer // tries to distribute replicas. type BalancerTarget struct { // Name of the target. The name can be later used to specify // additional balancer details for this target. Name string // ScaleTargetRef is a reference that points to a target resource to balance. // The target needs to expose the Scale subresource. ScaleTargetRef hpa.CrossVersionObjectReference // MinReplicas is the minimum number of replicas inside of this target. // Balancer will set at least this amount on the target, even if the total // desired number of replicas for Balancer is lower. // optional MinReplicas *int32 // MaxReplicas is the maximum number of replicas inside of this target. // Balancer will set at most this amount on the target, even if the total // desired number of replicas for the Balancer is higher. // optional MaxReplicas *int32 } // BalancerPolicyName is the name of the balancer Policy. type BalancerPolicyName string const ( PriorityPolicyName BalancerPolicyName priority ProportionalPolicyName BalancerPolicyName proportional ) // BalancerPolicy defines Balancer policy for replica distribution. type BalancerPolicy struct { // PolicyName decides how to balance replicas across the targets. // Depending on the name one of the fields Priorities or Proportions must be set. PolicyName BalancerPolicyName // Priorities contains detailed specification of how to balance when balancer // policy name is set to Priority. // optional Priorities *PriorityPolicy // Proportions contains detailed specification of how to balance when // balancer policy name is set to Proportional. // optional Proportions *ProportionalPolicy // Fallback contains specification of how to recognize and what to do if some // replicas fail to start in one or more targets. No fallback happens if not-set. // optional Fallback *Fallback } // PriorityPolicy contains details for Priority-based policy for Balancer. type PriorityPolicy struct { // TargetOrder is the priority-based list of Balancer targets names. The first target // on the list gets the replicas until its maxReplicas is reached (or replicas // fail to start). Then the replicas go to the second target and so on. MinReplicas // is guaranteed to be fulfilled, irrespective of the order, presence on the // list, and/or total Balancers replica count. TargetOrder []string } // ProportionalPolicy contains details for Proportion-based policy for Balancer. type ProportionalPolicy struct { // TargetProportions is a map from Balancer targets names to rates. Replicas are // distributed so that the max difference between the current replica share // and the desired replica share is minimized. Once a target reaches maxReplicas // it is removed from the calculations and replicas are distributed with // the updated proportions. MinReplicas is guaranteed for a target, irrespective // of the total Balancers replica count, proportions or the presence in the map. TargetProportions map[string]int32 } // Fallback contains information how to recognize and handle replicas // that failed to start within the specified time period. type Fallback struct { // StartupTimeout defines how long will the Balancer wait before considering // a pending/not-started pod as blocked and starting another replica in some other // target. Once the replica is finally started, replicas in other targets // may be stopped. StartupTimeout metav1.Duration } // BalancerStatus describes the Balancer runtime state. type BalancerStatus struct { // Replicas is an actual number of observed pods matching Balancer selector. Replicas int32 // Selector is a query over pods that should match the replicas count. This is same // as the label selector but in the string format to avoid introspection // by clients. The string will be in the same format as the query-param syntax. Selector string // Conditions is the set of conditions required for this Balancer to work properly, // and indicates whether or not those conditions are met. // optional // patchMergeKeytype // patchStrategymerge Conditions []metav1.Condition }API 字段与 CRD 校验规则对照 balancer/deploy/crd.yaml 中的 OpenAPI v3 Schema可以发现若干关键校验约束在 types.go 中通过 kubebuilder 标记声明字段类型校验/默认规则说明spec.targets数组必填至少 2 项每个目标必须有name长度 ≥ 1与scaleTargetRefkind namespec.replicasint32必填最小 0Balancer 期望分发的总副本数spec.selectorLabelSelector必填将所有目标的 Pod 归为一组的标签选择器spec.policy.policyNamestring必填取值priority或proportionalspec.policy.priorities.targetOrderstring 数组必填至少 2 项priority 策略的目标顺序spec.policy.proportions.targetProportionsmap必填至少 2 个键proportional 策略的权重映射spec.policy.fallback.startupTimeoutSecondsint32必填若配置 fallback最小 0判定 Pod 启动阻塞的超时秒数target.minReplicasint32可选最小 0未设置时按 0无下限处理target.maxReplicasint32可选最小 0未设置时不设上限spec.replicas与status.replicas、status.selector通过 Scale 子资源暴露这正是 HPA 能够以 Balancer 为伸缩目标的关键所在。部署 Balancer 控制器Balancer 的部署清单位于 balancer/deploy/controller.yaml包含四部分ServiceAccount、ClusterRole、ClusterRoleBinding 与 Deployment。其核心要点如下。RBAC 权限ClusterRole覆盖balancers与balancers/status的读写get/list/watch/patch/updatepods的 get/list/watch用于统计各目标内的 Pod 状态deployments/scale的 get/list/watch/patch/update用于读取和调整各目标的副本数events的创建与记录便于排查控制器行为。控制器 DeploymentapiVersion: apps/v1 kind: Deployment metadata: name: balancer-controller namespace: kube-system spec: replicas: 1 selector: matchLabels: app: balancer-controller template: metadata: labels: app: balancer-controller spec: serviceAccountName: balancer-controller containers: - name: controller image: gcr.io/gke-autoscaling-gcr/balancer:0.1.1 imagePullPolicy: Always args: [-v,4] resources: requests: cpu: 100m部署步骤先安装 CRDkubectl apply -f balancer/deploy/crd.yaml注册balancers.balancer.x-k8s.io这个 Namespaced 资源v1alpha1版本再部署控制器kubectl apply -f balancer/deploy/controller.yaml通过kubectl -n kube-system get pods -l appbalancer-controller确认控制器运行正常。控制器本身采用标准的 informer 限速工作队列模式实现见 balancer/pkg/controller/controller.go使用-v参数控制 klog 日志级别事件会以balancer-controller为事件源写入集群。实战示例一Priority 策略优先填满顺序目标Priority 策略适用于优先使用廉价/首选域容量不足再溢出到备用域的场景例如优先使用 spot 节点池、溢出到按需节点池。完整示例见 balancer/examples/nginx-priority.yamlapiVersion: apps/v1 kind: Deployment metadata: name: nginx-1 labels: app: nginx-1 srv: nginx spec: replicas: 3 selector: matchLabels: app: nginx-1 srv: nginx template: metadata: labels: app: nginx-1 srv: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-2 labels: app: nginx-2 srv: nginx spec: replicas: 3 selector: matchLabels: app: nginx-2 srv: nginx template: metadata: labels: app: nginx-2 srv: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 --- apiVersion: balancer.x-k8s.io/v1alpha1 kind: Balancer metadata: name: nginx spec: replicas: 5 selector: matchLabels: srv: nginx policy: policyName: priority priorities: targetOrder: [nginx-1,nginx-2] fallback: startupTimeoutSeconds: 180 targets: - name: nginx-1 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: nginx-1 minReplicas: 1 maxReplicas: 7 - name: nginx-2 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: nginx-2 minReplicas: 1 --- apiVersion: v1 kind: Service metadata: name: nginx spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: srv: nginx该示例的分配逻辑spec.replicas: 5两个 Deployment 各自minReplicas: 1Balancer 会先把每个目标的minReplicas填满各 1 个剩余 3 个副本按targetOrder: [nginx-1, nginx-2]顺序分配先填nginx-1其maxReplicas: 7不会触顶因此最终形态为nginx-1 4、nginx-2 1若nginx-1中某个副本在 180 秒内未能启动fallback 生效Balancer 会在nginx-2中补一个副本待nginx-1恢复后再缩回。Priority 策略的源码实现位于 balancer/pkg/policy/priority.go 的distributeByPriority函数第一轮为所有目标放置minReplicas从总副本数中扣除若不足replicas 0记录MissingReplicas问题并将余量置 0第二轮按targetOrder依次填充每个目标可容纳max - placement[key]个副本直到replicas耗尽每处理一个目标若其存在NotStartedWithinDeadline 0超过超时仍未启动的副本会把这些被阻塞的副本加回待分配池溢出到后续目标若仍有剩余副本无处安放则记录OverflowReplicas。实战示例二Proportional 策略按比例均衡分布Proportional 策略适用于多可用区均衡承载流量的场景例如在两个可用区之间按 50/50 分配副本。完整示例见 balancer/examples/nginx-proportional.yamlapiVersion: apps/v1 kind: Deployment metadata: name: nginx-1 labels: app: nginx-1 srv: nginx spec: replicas: 3 selector: matchLabels: app: nginx-1 srv: nginx template: metadata: labels: app: nginx-1 srv: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-2 labels: app: nginx-2 srv: nginx spec: replicas: 3 selector: matchLabels: app: nginx-2 srv: nginx template: metadata: labels: app: nginx-2 srv: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 --- apiVersion: balancer.x-k8s.io/v1alpha1 kind: Balancer metadata: name: nginx spec: replicas: 10 selector: matchLabels: srv: nginx policy: policyName: proportional proportions: targetProportions: nginx-1: 50 nginx-2: 50 fallback: startupTimeoutSeconds: 180 targets: - name: nginx-1 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: nginx-1 minReplicas: 1 maxReplicas: 7 - name: nginx-2 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: nginx-2 minReplicas: 1 --- apiVersion: v1 kind: Service metadata: name: nginx spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: srv: nginx该示例的分配逻辑spec.replicas: 10比例nginx-1: 50、nginx-2: 50即 1:1扣除两个目标的minReplicas各 1后剩余 8 个副本按比例分配最终趋近nginx-1 5、nginx-2 5若某个目标达到maxReplicas它会从比例计算中移除剩余副本按更新后的比例在其余目标间重新分配minReplicas的保证与优先级、比例甚至是否出现在targetProportions映射中均无关。Proportional 策略的源码实现位于 balancer/pkg/policy/proportional.godistributeByProportions先放置所有目标的最小副本数再做第一轮比例分配忽略未启动的副本随后统计存在NotStartedWithinDeadline 0的目标把需要补的副本仅分配到无问题not-blocked的目标中实际按比例分发由distributeGroupProportionally完成它采用DHondt 最高平均数法每次迭代计算每个目标proportion / (1 placement[k])作为排名选择排名最高的目标放入 1 个副本直至副本耗尽或所有目标容量用尽sortedKeysWithCapacity对所有尚有容量的目标按键名排序保证算法稳定——每次按相同顺序处理目标避免副本在目标之间来回抖动flapping无论比例策略还是优先级策略目标顺序都先按名称排序这是算法确定性不震荡的关键设计。与 HPA 集成以 Balancer 为伸缩目标Balancer 的一个核心价值是用单一伸缩目标驱动多个 Deployment。由于 Balancer 自身暴露 Scale 子资源可以像 Deployment 一样作为 HPA 的scaleTargetRef。完整示例见 balancer/examples/nginx-priority-hpa.yaml其关键差异是在两个 Deployment 的容器中增加了 CPU requestHPA 计算 CPU 利用率所必需apiVersion: apps/v1 kind: Deployment metadata: name: nginx-1 labels: app: nginx-1 srv: nginx spec: replicas: 3 selector: matchLabels: app: nginx-1 srv: nginx template: metadata: labels: app: nginx-1 srv: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 resources: requests: cpu: 100m --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-2 labels: app: nginx-2 srv: nginx spec: replicas: 3 selector: matchLabels: app: nginx-2 srv: nginx template: metadata: labels: app: nginx-2 srv: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 resources: requests: cpu: 100m --- apiVersion: balancer.x-k8s.io/v1alpha1 kind: Balancer metadata: name: nginx spec: replicas: 5 selector: matchLabels: srv: nginx policy: policyName: priority priorities: targetOrder: [nginx-1,nginx-2] fallback: startupTimeoutSeconds: 180 targets: - name: nginx-1 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: nginx-1 minReplicas: 1 maxReplicas: 7 - name: nginx-2 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: nginx-2 minReplicas: 1 --- apiVersion: v1 kind: Service metadata: name: nginx spec: ports: - port: 80 protocol: TCP targetPort: 80 selector: srv: nginx --- apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: nginx spec: minReplicas: 2 maxReplicas: 10 metrics: - resource: name: cpu target: averageUtilization: 50 type: Utilization type: Resource scaleTargetRef: apiVersion: balancer.x-k8s.io/v1alpha1 kind: Balancer name: nginx这里的协同工作方式是HPA 负责伸缩多少根据 CPU 平均利用率50%在 210 之间调整 Balancer 的spec.replicasBalancer 负责分布到哪收到新的总副本数后按 priority 策略把副本优先分配给nginx-1溢出给nginx-2两个 Deployment 上不再配置各自的 HPA彻底避免多 HPA 之间的竞态与倾斜。控制器工作流从轮询到落地的五个阶段Balancer 控制器的核心协调逻辑在 balancer/pkg/controller/core.go 的ProcessBalancer方法中按五个阶段执行每个阶段失败都会返回带阶段信息的BalancerErrorScaleSubresourcePolling轮询 Scale 子资源遍历spec.targets通过 ScaleClient 获取每个目标的 Scale 对象与 GroupResourcePodListing列出 Pod解析每个目标 Scale 状态中的selector用 Pod lister 列出对应 PodPodLabelsChecking校验 Pod 标签逐一确认这些 Pod 是否匹配 Balancer 的spec.selector不匹配即报错——这保证了 Balancer 统计的副本数与 Service 路由的 Pod 集合一致ApplyingPolicyListing应用策略调用 balancer/pkg/policy/policy.go 的GetPlacement根据policyName分发副本并返回每个目标的目标副本数ReplicaPlacement若配置缺失如 priority 策略未提供priorities或targetOrder会返回incomplete policy definition类错误ReplicaCountSetting设置副本数对每个目标若其当前scale.Spec.Replicas与计算出的目标值不一致则调用UpdateScale进行更新。Pod 状态统计由 balancer/pkg/pods/summary.go 的CalculateSummary完成它是所有策略的输入基础TotalRunning 与 Pending 状态的 Pod 总数Running已运行 Pod 数NotStartedWithinDeadline仍处于 Pending、且创建时间超过超时阈值的 Pod 数——这些 Pod 被判定为启动受阻触发 Fallback 逻辑未设置fallback时内部会使用一个约 50 年的无穷大截止时间infDeadline相当于永不触发回退。控制器的调度侧balancer/pkg/controller/controller.go使用 Balancer informer 的事件回调把对象入队并采用NewFixedItemIntervalRateLimiter(resync)限速工作队列以固定的 resync 周期周期性处理每个 Balancer从而实现周期性检查、主动再平衡的持续收敛。Fallback 机制应对启动受阻的副本Fallback回退是 Balancer 保障可用性的关键机制定义在spec.policy.fallbackstartupTimeoutSeconds是唯一字段最小 0表示 Balancer 在判定某个 Pending/未启动 Pod 为受阻之前等待的时间一旦某个目标内的副本被视为受阻Balancer 会在其他目标中启动额外副本直到受阻副本最终启动后再停止这些多余副本见distributeByPriority与distributeByProportions中基于NotStartedWithinDeadline的 fallback 计算未配置fallback时不做任何回退。这正好回应用了引言中的诉求当某个可用区宕机或 spot 节点池枯竭导致 Pod 长时间无法调度时Balancer 能主动把副本转移到可用域而不是让整个部署停滞或倾斜。小结Balancer 以一个 CRD 对象 一个控制器的轻量形态把**副本总量决策交给 HPA与副本分布决策交给策略**彻底解耦Priority 策略适合首选域优先、容量溢出的成本优化场景Proportional 策略适合跨域均衡承载的高可用场景Fallback保证域级故障时副本可迁移Scale 子资源 Selector 约束保证 Balancer 能与 Service、HPA 无缝协作。如果需要在同一项目中继续深入建议阅读以下文件设计文档balancer/proposals/balancer.mdAPI 类型定义balancer/pkg/apis/balancer.x-k8s.io/v1alpha1/types.goCRD 与部署清单balancer/deploy/crd.yaml、balancer/deploy/controller.yaml实战示例balancer/examples/nginx-priority.yaml、balancer/examples/nginx-proportional.yaml、balancer/examples/nginx-priority-hpa.yaml策略算法balancer/pkg/policy/policy.go、balancer/pkg/policy/priority.go、balancer/pkg/policy/proportional.go控制器实现balancer/pkg/controller/core.go、balancer/pkg/controller/controller.go【免费下载链接】autoscalerAutoscaling components for Kubernetes项目地址: https://gitcode.com/GitHub_Trending/au/autoscaler创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
