New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for troubleshooting distroless containers #27140
Comments
Maybe just use shared PID namespaces in Docker 1.12 (and rkt's equivalent, if/when it's there): moby/moby#10163 You'd create a new container with a debug image (or one of many) and attach it to the existing pod. |
Ah, that would be perfect! I see that #1615 exists for sharing a PID namespace in a pod, which also led me to discover this issue may very well be a dupe of #24957 (though @smarterclayton only mentions debugging as one of many possible uses). Before I close this issue as a dupe, I just want to make sure I understand how it would work. I see two possibilities --
Though not as useful as a debug container, something that could work in the interim would be to attach a volume to a running container, if that were possible. (In 1.2 it doesn't seem to be: spec: Forbidden: pod updates may not change fields other than |
Today, it's not possible to mutate a running pod. Attach connects to an existing container (if the Stdin and/or TTY flags were set). Another option beyond those you mention above would be to allow adhoc container execution inside of a pod that mounts the majority of the namespaces of another container. I.e.:
Scheduling a new pod doesn't convey to the runtime that C should be cleaned up if 1/A is removed (which we want, probably). However, in the flow above C would have to share resources with A so there are various edge cases around guaranteed pods having no more CPU to spare for C. Another option would be to implement container volumes #831 and mount your debug tools into your bare bones image. That's not adhoc though, so it doesn't quite fit your use case. |
General observation: if we're going to make it possible to add containers to pods, we should do it in a general way, not just to cover this use case. I can't imagine we're going to implement adding containers to pods soon though. |
I've gotten a lot of great suggestions so far, so I thought I'd summarize them:
I'm not yet familiar with the kubernetes implementation, but the volume solutions strike me as the easiest to implement and those requiring mutation of pods the most difficult. The solutions that extend I don't like coupling this functionality to the node, so I'll start down the road of writing a proposal to support |
The idea of running a "toolbox" container that is not part of a pod, but has the same namespaces as the pod is cute, but a little scary and opaque. If we allowed adding a container to a pod, it could just be a shiny form of that. Another option might be to include a container in the pod that is "inactive" until later activated. |
How far do you get if you have your build system produce "debug" and "stripped" versions of your container image ("debug" has a base image with a debugger, and "stripped" does not)? Normally, your service just has "stripped" containers, but if there is an active bug, you deploy one or more extra containers for that service which are "debug images", and wait for the problem to happen again. Doesn't help if the problem only happens once, but who has time to debug that kind of problem anyhow? |
The key question I think is whether you do this by making a new pod, and pods stay immutable, or if we make pods mutable. |
@thockin I'll grant you scary, but I'm not yet willing to concede that pulling in a different userland for a binary is any more opaque than your run-of-the-mill I'd be happy with an inactive container as long as it (optionally) automatically updates prior to activating so I'm not tied to an old debug image. The downside here is the extra boilerplate config in every pod I run. |
I should note that we've had very good feedback from the "debug" command On Fri, Jul 8, 2016 at 7:16 PM, verb notifications@github.com wrote:
|
Debug also leads naturally to other hooks like On Fri, Jul 8, 2016 at 7:31 PM, Clayton Coleman ccoleman@redhat.com wrote:
|
@erictune Certainly workable, but I'm concerned about some downsides:
Even if I can't think of it right now, I'm confident there will eventually be a reason that I'll need to connect to a stripped container and it seems like a bit of a gap in the toolbox. |
Vaguely related to #831 |
/assign @verb |
Issues go stale after 90d of inactivity. Prevent issues from auto-closing with an If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or |
Until this support lands, I've been using this to debug: |
/priority important-longterm |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle stale |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle stale |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle rotten |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
This issue has become a sort of tracking issue for the overall problem of troubleshooting distroless containers, but tracking issues are a poor fit for k/k: the discussion dies down and it just into fighting lifecycle. This issue motivated two enhancements: kubernetes/enhancements#277 (Ephemeral containers) and kubernetes/enhancements#1441 (kubectl debug). Let's track these efforts in their feature issues and consider this k/k issue closed. |
I favor running distroless container images in production, usually built from SCRATCH with the binary and bare minimum of libraries. This has a number of benefits and works well the vast majority of the time, but it's difficult on the occasion where I need to troubleshoot something in production.
It'd be nice if there was a way to enable some sort of debug mode for a running pod where a set of tools could be attached that I could then exec within the container.
I'm willing to do the work to develop this feature, if necessary, but I'm not exactly sure what the next steps should be. I'm opening this issue to find out if anyone is working on something similar and/or get early guidance on how this could fit into the Kubernetes road map.
If it's the case that no one else is working on something similar and such a feature would be welcomed, my expectation is the next step would be to write a design proposal.
Tracking issue: kubernetes/enhancements#277
KEP: https://git.k8s.io/enhancements/keps/sig-node/20190212-ephemeral-containers.md
PRs:
api/api-rules/violation_exceptions.list
The text was updated successfully, but these errors were encountered: