New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
A better story of container metrics for runtime integration #27097
Comments
/c @kubernetes/sig-node |
+1 |
cc @kubernetes/sig-rktnetes |
One thing to consider is that this will make adding additional metrics significantly more difficult. We may want to distinguish between required and nice-to-have metrics. It also depends on our future monitoring story, which is still up in the air (e.g. will these metrics be surfaced in the summary API and collected by Heapster? I suppose even if we went with a standalone solution, it could implement the client interface to collect the metrics). |
+1 for a set of core metrics. As for nice-to-have metrics, we can either 1) allow runtime to return empty stats if it doesn't support it, or 2) go with the standalone solution, in which case it's up to runtime to integrate with heapster or other components. @timstclair @dchen1107 , what's the status about custom metrics and monitoring in general? |
I don't think any progress has been made since we decided not to prioritize it for 1.3, and I haven't seen anything about it on 1.4. @dchen1107 do you have any more context? |
As @timstclair mentioned in google/cadvisor#1394 (comment), it'd be great to keep Windows in mind when defining this interface. |
FYI @DirectXMan12 |
@timstclair @yujuhong the next action item for this is to analyze & refactor the current cAdvisor usage in kubelet, is that correct? |
/cc @ConnorDoyle |
Yes, that'd give us a better idea of what metrics we need to support the current implementation. |
We've had a lot of discussions around separating "core" metrics that are required for internal kubernetes functionality from monitoring metrics that are user facing, and would be used for monitoring, alerting, etc. With this split, the CRI would only need to provide the core metrics (other metrics would be collected through alternate channels). I'd really like to make this split before defining an API for it, but I think this is still a ways off. |
Do HPA metrics count as "required for internal kubernetes functionality"? How do you define what "core" is, especially if we open up the HPA to allow referring to arbitrary metrics? Do we say "by default, the only metrics available in your cluster to scale on are CPU, memory, etc, and for anything else you need to write your own solution to gather and push to Heapster"? |
@DirectXMan12 And this is why I think it's a ways off :) In all seriousness though, HPA metrics are definitely part of the discussion. There needs to be a way to ingest them, whether or not they're a part of the core metrics exported by the runtime is another question. |
Just brainstorming, but how committed are folks to making metrics part of the CRI? Based on the above it sounds like: "HPA requires collecting some metrics, those metrics can come from multiple sources, one likely metrics source is the container runtime." Would it make sense to define a metrics source interface and then let container runtimes optionally implement it? |
To clarify:
|
I think we should come up with a story for how that's going to happen. At the end of the day, it does not necessarily have to involve the CRI, but it may (I do like the idea of an optional interface, as @ConnorDoyle suggested), so I think we should work it out. I'm wary of a scenario where every kubernetes cluster/platform has a different method for allowing users to expose "service" metrics, which would make applications not as easily portable between clusters. |
This is one possible route, but it requires kubelet figuring out whether runtime supports metrics (i.e., optional features) and whether kubelet's metrics library (a.k.a. cadvisor) supports the type of the runtime (e.g., hypervisor-based). It may make more sense to just push this down to the runtime. |
I had interpreted the suggestion as something like defining a well-known interface that had two different actions: a) fetching metrics So Kubelet can a) ask the container runtime for metrics, and b) say "by the way, expect to see 'service' metrics in format X in location Y for these pods in these places" (where "in location Y" might be "on port Z" or something like that). The container runtime can then choose to implement those two pieces however it wants. Runtimes would be free to use cAdvisor for this purpose, but would not be obligated to do so; use of cAdvisor would be behind the the metrics interface, so that Kubelet would not know about cAdvisor at all. Fetching core metrics would be a separate interface (that would be mandatory), but all the metrics would be presented together by Kubelet for collection by Heapster. |
Or even more simply, configure Kubelet to look to one or more endpoints as metrics sources. One of them might happen to be a process that also implements the CRI interface. With gRPC you can stack multiple services. If guaranteeing system metrics are present is a concern, maybe the system metrics source endpoint could be a separate config value. |
Agreed.
This is debatable. A separate monitoring pipeline could allow more flexibility without changing the kubelet stats api.
If kubelet is going to be the hub of all custom metrics, this interface/api should be completely separate from the CRI. I think we've all agreed that core metrics should be part of CRI, and we should aim for finishing that first. |
This is my first message on Kubernetes community forum so: Hello all 👋 I think that from the point of view of telemetry solutions having an interface to integrate with Kubelet (to provide core and/or non-core metrics) would be a great solution. |
Splitting cAdvisor into a part that handles core metrics and is compiled into Kubelet, and a part that runs as a separate pod and handles everything else (including custom metrics) and is interchangeable with other collection agents, seems reasonable to me. I didn't fully understand why container vs. pod was the criteria you are using to split non-core metrics vs. core metrics, but it sounds like it makes the CRI simpler, and I don't know enough to object. :) In terms of what the scheduler needs,
I assume that (2) will actually come through the separate monitoring pipeline, not from the core cAdvisor? In that case it is not quite correct to say that the core metrics provides all the information needed for scheduling. |
@thockin re: #27097 (comment)
@davidopp re: #27097 (comment)
|
Users specify resource requests and limits per container. Thus, container-level metrics will be needed for vertical auto-scaling, resource estimation, and predictive horizontal auto-scaling. Presumably HPA currently use container-level metrics, as well. |
@bgrant0607, the conclusion drawn from this discussion is that container metrics should be handled in a separate pipeline (standalone cadvisor + heapster), where kubelet may not be involved. |
@yujuhong I'm fine with the metrics being surfaced via another component. I was just pointing out that the data will be required. |
We've decided to punt the monitoring issues to after v1.5, in the hope that the monitoring pipeline will be more well-defined by then. |
@yujuhong can you please involve me in the planning for the monitoring aspect of the container runtime interface? i would like to represent the windows-aspect of this work. thanks! |
Automatic merge from submit-queue (batch tested with PRs 45809, 46515, 46484, 46516, 45614) CRI: add methods for container stats **What this PR does / why we need it**: Define methods in CRI to get container stats. **Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: Part of kubernetes/enhancements#290; addresses kubernetes#27097 **Special notes for your reviewer**: This PR defines the *minimum required* container metrics for the existing components to function, loosely based on the previous discussion on [core metrics](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/core-metrics-pipeline.md) as well as the existing cadvisor/summary APIs. Two new RPC calls are added to the RuntimeService: `ContainerStats` and `ListContainerStats`. The former retrieves stats for a given container, while the latter gets stats for all containers in one call. The stats gathering time of each subsystem can vary substantially (e.g., cpu vs. disk), so even though the on-demand model preferred due to its simplicity, we’d rather give the container runtime more flexibility to determine the collection frequency for each subsystem*. As a trade-off, each piece of stats for the subsystem must contain a timestamp to let kubelet know how fresh/recent the stats are. In the future, we should also recommend a guideline for how recent the stats should be in order to ensure the reliability (e.g., eviction) and the responsiveness (e.g., autoscaling) of the kubernetes cluster. The next step is to plumb this through kubelet so that kubelet can choose consume container stats from CRI or cadvisor. **Alternatively, we can add calls to get stats of individual subsystems. However, kubelet does not have the complete knowledge of the runtime environment, so this would only lead to unnecessary complexity in kubelet.* **Release note**: ```release-note Augment CRI to support retrieving container stats from the runtime. ```
#45614 added methods in CRI to get container stats. We didn't have time to plumb this through kubelet, so CRI runtimes cannot use it yet. I will file a separate issue for finishing that. |
Currently, kubelet gets container stats/metrics through cadvisor and expose them through the /stats endpoint.
This forces container runtimes to integrate with cadvisor first, before they can properly integrate with kubelet.
Now that we are defining the new container runtime interface for kubelet (#22964), we should define container metrics and associated methods in the new interfaces so that runtimes can integrate directly with kubelet. This will greatly reduce the integration pain.
Before doing that, we should audit existing cadvisor usage in kubelet and break down the cadvisor interface for different purposes.
/cc @timstclair @vishh @mtaufen
The text was updated successfully, but these errors were encountered: