New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dynamic Kubelet Configuration Proposal #29459
Conversation
cc @kubernetes/sig-node |
## Motivation | ||
|
||
The Kubelet is currently configured via command-line flags. This is painful for a number of reasons: | ||
- It makes it difficult to change the way Kubelets are configured in a running cluster, without tearing the cluster down. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well tearing down is not necessary. It depends on the config management system that is being used.
I'd say that it requires changing the kubelet startup configuration which is often tedious.
cc/ @mikedanese |
e7608dd
to
a548857
Compare
|
||
- Add a well-defined, versioned configmap object for Kubelet configuration. | ||
- Add/Remove/Update global and per-node Kubelet config on the fly via the API server. | ||
- Provide a REST API to get the config currently in use by a given Kubelet. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not place the configuration and the source of the configuration in NodeStatus
?
I imagine a workflow where users can pull a new config object from a separate git repo, and have a controller apply the new config to nodes in a controlled manner, by waiting for a set of nodes to come up with the new configuration and be healthy, before proceeding with applying the new config to other nodes.
Bad config changes can be reverted via git based workflow on top of this functionality.
Tracking an example implementation of some basic functionality here: #30090. |
## Motivation | ||
|
||
The Kubelet is currently configured via command-line flags. This is painful for a number of reasons: | ||
- It makes it difficult to change the way Kubelets are configured in a running cluster, because it is often tedious to change the Kubelet startup configuration. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
without a configuration management solution.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ansible or salt can do that for you. You will have problems when you update kubelet configuration in apiserver which gets reflected in a kubelet vs. re-running ansible/running salt. With ansible, it is quote easy to skip configuration tasks (skip-tags
flag) once the cluster is deployed and the ansible is re-run. So the new kubelet configuration does not get touched (still, need to keep that in mind). With salt, not sure if you can do anything here unless you stop the salt, change salt (e.g. skip configuration) and start salt.
Saying, some folks can be running ansible/salt already. When you start configuring the kubelet dynamically by default, you can get into trouble.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a good point. We should warn people about this in the documentation for the feature. With the current design, it would be pretty easy to avoid dynamically configuring the Kubelet -- just choose not to post kubelet-<node-name>
configmaps in kube-system
.
Alpha implementation of the Dynamic Kubelet Configuration feature. See the proposal doc in kubernetes#29459.
Alpha implementation of the Dynamic Kubelet Configuration feature. See the proposal doc in kubernetes#29459.
Automatic merge from submit-queue (batch tested with PRs 50016, 49583, 49930, 46254, 50337) Alpha Dynamic Kubelet Configuration Feature: kubernetes/enhancements#281 This proposal contains the alpha implementation of the Dynamic Kubelet Configuration feature proposed in ~#29459~ [community/contributors/design-proposals/dynamic-kubelet-configuration.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/dynamic-kubelet-configuration.md). Please note: - ~The proposal doc is not yet up to date with this implementation, there are some subtle differences and some more significant ones. I will update the proposal doc to match by tomorrow afternoon.~ - ~This obviously needs more tests. I plan to write several O(soon). Since it's alpha and feature-gated, I'm decoupling this review from the review of the tests.~ I've beefed up the unit tests, though there is still plenty of testing to be done. - ~I'm temporarily holding off on updating the generated docs, api specs, etc, for the sake of my reviewers 😄~ these files now live in a separate commit; the first commit is the one to review. /cc @dchen1107 @vishh @bgrant0607 @thockin @derekwaynecarr ```release-note Adds (alpha feature) the ability to dynamically configure Kubelets by enabling the DynamicKubeletConfig feature gate, posting a ConfigMap to the API server, and setting the spec.configSource field on Node objects. See the proposal at https://github.com/kubernetes/community/blob/master/contributors/design-proposals/dynamic-kubelet-configuration.md for details. ```
This change is