Hi @zmerlynn , do you need any help with this issue ? If yes, which work items could I work on ? Thanks !

This is a P1 and needs an owner. @zmerlynn if you are planning to drive it, please assign to yourself, otherwise it sounds like @rsokolowski is interested in starting on it.

I think a chunk of this work is actually on Dawn, but let me figure out how to piece it out.

cc @dchen1107

I think the first bullet, reporting the actual version somewhere, is probably somewhere on the node team. It seems like it could hook in with the status update mechanism for #4562 so that we're just doing one status update, but we don't necessarily need to update that often.

The second bullet (keeping track of all the versions) is ostensibly the NodeController.

The third bullet, having apiserver only speak the minimum, I'm not sure where it fits: ideally in a rolling upgrade scenario, you'd have some interconnection from NodeController to the apiserver that specifies the minimum, and when the last node reports in, an event fires back to apiserver. I'm not sure what follows this pattern right now.

The last bullet is waiting on the merge of #4833, and is otherwise, I think a testing and policy bullet. Which is complicated in its own right, but doesn't need a lot of upfront design.

This is awesome, @zmerlynn—thanks for nailing down the version requirements and writing out the 1.0 upgrade plan in understandable English.

I'm going to be removing the upgrade parts of this to shove them in the other rollups (master #6075 and node #6079). That way, this can be specifically the versioning rollup and we can split the work nicely into three separate issues. I think I've covered everything upgrade-related in those, but I really liked the clear description of upgrades for 1.0 you wrote in this, so I'm going to transfer that to one or both of them.

Let me know if I missed something in this transition, and sorry if I confused anyone's issue linking by doing this!

• assigning @zmerlynn as the overall owner to drive, even if individual work items are handled across teams

Field gate proposal: https://docs.google.com/document/d/1wuoSqHkeT51mQQ7dIFhUKrdi3-1wbKrNWeIL4cKb9zU/edit#

Note also that the concept of "profiles" is being discussed, for multiple purposes.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@zmerlynn - can you triage this issue?

/remove-lifecycle stale
/lifecycle frozen

I'd like to keep this open. We have a number of challenges around cluster lifecycle that I'd like to consider holistically:

• Field-level versioning: Field-level versioning #34508
• HA masters mangle data: HA-Kubernetes mangles data when running mixed apiserver versions (upgrades) #46073
• Storage upgrade: Storage upgrade mechanism #52185
• Teardown: Need an API call for "teardown all external resources" #4630
• Run levels: Control plane bootstrapping order AKA we need a run-level concept #54522
• I don't remember which issue covers resource stranding upon rollback.
• Add-on management: kube-addon-update revamp #23233

cc @jagosan

Operations:

• Turn up
• Upgrade
• Downgrade
• Teardown

Strawman upgrade sequence:

1. Upgrade masters (apiserver, controller manager, scheduler)
2. Enable new APIs
3. Upgrade addons (caveats: don't remove old DaemonSets, maybe support API version discovery)
4. Disable old APIs
5. Upgrade nodes

Default storage versions can't be updated until after step 1.

@bgrant0607 why is this frozen ? it seems it was originally created for 1.0 in 2015, but has lot of relevant details that could become a documentation link or KEP for how clusters should be upgraded , but not sure i have seen any of those documentation.

@krmayankk It's fine to close this issue, though AFAIK, neither the upgrade/downgrade sequencing, nor lifecycle stages, nor teardown operations, nor the other proposals mentioned above (#4855 (comment)) have been implemented. More specific issues, such as #54522 have been filed.

/close

@bgrant0607: Closing this issue.