Description
What happened?
PodResourcesAPI provide a List() endpoint which reports about all the resources that consumed by pods and containers on the node.
The problem is that pods which are in terminal phase (i.e. are in Failed or Succeeded status) are reported as well. about The internal managers reassign resources assigned to pods in terminal phase, so PodResources should ignore them, because they can still be in used.
What did you expect to happen?
PodResources should ignore and not reports about resources which are in used by pods which are in terminal phase.
How can we reproduce it (as minimally and precisely as possible)?
Provided a new test-case that demonstrates the exact problem and can be used as a reproducer: #119402
Anything else we need to know?
The docs that describes PodResourceAPI are not refer explicitly to whether List() should ignore terminal pod's resources or not, meaning it might not be a bug at all.
OTOH, internal managers reassign resources assigned to pods in terminal phase so it make sense to not account them.
Kubernetes version
Kubernetes v1.26.2
Cloud provider
OS version
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output hereInstall tools
Container runtime (CRI) and version (if applicable)
Related plugins (CNI, CSI, ...) and versions (if applicable)
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Milestone
Relationships
Development
Participants
Activity
ffromani commentedon Jul 19, 2023
/sig node
SergeyKanzhelev commentedon Jul 19, 2023
/triage accepted
carlory commentedon Jul 21, 2023
/assign
ffromani commentedon Jul 27, 2023
I was thinking about this issue for a bit, and the root cause is probably that there is not a (good enough) formal spec for the API itself so the implementation is pretty much the reference.
We should probably improve here but not sure how (e.g. where to describe more formally the API, perhaps just in the docs?)
That said, my thoughts about the ideal semantics are:
SergeyKanzhelev commentedon Jul 28, 2023
I wonder if pod resources API may report the same set of resources used by two pods - one succeeded and one newly scheduled in a single response?
@ffromani should we try to summarize shortcomings at the page you listed as a starting point? We can start with the list of edge cases like:
I like the idea that podresources API is driven by the fact whether resources are actually in use, not by the Pod status" Thus the resource manager behavior ideally needs to be documented.
65 remaining items