Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for troubleshooting distroless containers #27140

Closed
6 of 9 tasks
verb opened this issue Jun 9, 2016 · 37 comments
Closed
6 of 9 tasks

Support for troubleshooting distroless containers #27140

verb opened this issue Jun 9, 2016 · 37 comments
Assignees
Labels
lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@verb
Copy link
Contributor

verb commented Jun 9, 2016

I favor running distroless container images in production, usually built from SCRATCH with the binary and bare minimum of libraries. This has a number of benefits and works well the vast majority of the time, but it's difficult on the occasion where I need to troubleshoot something in production.

It'd be nice if there was a way to enable some sort of debug mode for a running pod where a set of tools could be attached that I could then exec within the container.

I'm willing to do the work to develop this feature, if necessary, but I'm not exactly sure what the next steps should be. I'm opening this issue to find out if anyone is working on something similar and/or get early guidance on how this could fit into the Kubernetes road map.

If it's the case that no one else is working on something similar and such a feature would be welcomed, my expectation is the next step would be to write a design proposal.

Tracking issue: kubernetes/enhancements#277
KEP: https://git.k8s.io/enhancements/keps/sig-node/20190212-ephemeral-containers.md
PRs:

@therc
Copy link
Member

therc commented Jun 13, 2016

Maybe just use shared PID namespaces in Docker 1.12 (and rkt's equivalent, if/when it's there): moby/moby#10163

You'd create a new container with a debug image (or one of many) and attach it to the existing pod.

@verb
Copy link
Contributor Author

verb commented Jun 13, 2016

Ah, that would be perfect! I see that #1615 exists for sharing a PID namespace in a pod, which also led me to discover this issue may very well be a dupe of #24957 (though @smarterclayton only mentions debugging as one of many possible uses).

Before I close this issue as a dupe, I just want to make sure I understand how it would work. I see two possibilities --

  1. We attach a new container to a running pod. I assume this would look something like a kubectl attach, but if I try to do this in 1.2 I get the error spec.containers: Forbidden: pod updates may not add or remove containers. (am I doing this incorrectly?)
  2. We create a new pod using the PID namespace of the target pod we want to debug, in which case we would want support for that in the implementation of shared PID namespaces.

Though not as useful as a debug container, something that could work in the interim would be to attach a volume to a running container, if that were possible. (In 1.2 it doesn't seem to be: spec: Forbidden: pod updates may not change fields other than containers[*].image or spec.activeDeadlineSeconds)

@smarterclayton
Copy link
Contributor

Today, it's not possible to mutate a running pod. Attach connects to an existing container (if the Stdin and/or TTY flags were set).

Another option beyond those you mention above would be to allow adhoc container execution inside of a pod that mounts the majority of the namespaces of another container. I.e.:

  1. Have pod 1 with containers A and B
  2. Launch a transient container C that attaches to container A inside of pod 1:
    1. C shares the pid namespace of A
    2. C mounts the filesystem of A at /mnt/attach (or similar)
    3. C also joins other namespaces of A as necessary, esp. cgroup (or pod cgroup)

Scheduling a new pod doesn't convey to the runtime that C should be cleaned up if 1/A is removed (which we want, probably). However, in the flow above C would have to share resources with A so there are various edge cases around guaranteed pods having no more CPU to spare for C.

Another option would be to implement container volumes #831 and mount your debug tools into your bare bones image. That's not adhoc though, so it doesn't quite fit your use case.

@davidopp
Copy link
Member

General observation: if we're going to make it possible to add containers to pods, we should do it in a general way, not just to cover this use case.

I can't imagine we're going to implement adding containers to pods soon though.

@j3ffml j3ffml added the team/ux label Jun 28, 2016
@verb
Copy link
Contributor Author

verb commented Jul 8, 2016

I've gotten a lot of great suggestions so far, so I thought I'd summarize them:

  1. Volume solutions make binaries available for kubectl exec inside of a running container
    1. Allow mutation of a running container (via kubectl patch or equivalent) to add volumes. This would be especially useful in conjunction with Image volumes and container volumes #831 to support container images as volumes.
    2. Implicitly create a unique, empty volume attached to every container that's run. Tools can be unpacked into this volume at debug time, perhaps using an existing package solution like nix or SNAP. (credit: @thockin)
  2. Container solutions run a new container process in shared namespaces. The debug container image (obviously) brings its own filesystem, so the filesystem roots (including volume mounts) of other containers must be made available to the debug container somehow.
    1. Extend kubectl exec to allow executing commands from arbitrary container images which are then automatically fetched and run within the existing pod cgroup/pid/net/etc namespaces. (e.g. kubectl exec 123456-7890 -X gcr.io/debug-image -i -t -- bash)
    2. Allow permanent mutation of a running pod to add a new container using the standard shared pod namespaces (requires at least Shared PID and UTS namespaces #1615 for a pod shared PID namespace).
    3. Support creating a new, standalone pod that uses the same namespace as the target pods.
  3. Node solutions (for lack of a better term) are independent of the pod/container constructs.
    1. Support running binaries provided by the node inside the container namespaces (Make it possible to shell into any container, even if it has no shell #10834). This could be an extension of kubectl as well (e.g. kubectl exec 123456-7890 -H /bin/busybox -i -t)

I'm not yet familiar with the kubernetes implementation, but the volume solutions strike me as the easiest to implement and those requiring mutation of pods the most difficult.

The solutions that extend kubectl exec "feel" the most natural to me as a user. The backend steps (read: docker commands to run) seem conceptually straightforward, but there may be some technical hurdles (e.g. I'm getting ptrace not permitted when attempting to strace across container images, but that's happening within the same image as well).

I don't like coupling this functionality to the node, so I'll start down the road of writing a proposal to support kubectl exec of arbitrary container images (2i in the above list).

@thockin
Copy link
Member

thockin commented Jul 8, 2016

The idea of running a "toolbox" container that is not part of a pod, but has the same namespaces as the pod is cute, but a little scary and opaque.

If we allowed adding a container to a pod, it could just be a shiny form of that.

Another option might be to include a container in the pod that is "inactive" until later activated.

@erictune
Copy link
Member

erictune commented Jul 8, 2016

How far do you get if you have your build system produce "debug" and "stripped" versions of your container image ("debug" has a base image with a debugger, and "stripped" does not)?

Normally, your service just has "stripped" containers, but if there is an active bug, you deploy one or more extra containers for that service which are "debug images", and wait for the problem to happen again.

Doesn't help if the problem only happens once, but who has time to debug that kind of problem anyhow?

@erictune
Copy link
Member

erictune commented Jul 8, 2016

The key question I think is whether you do this by making a new pod, and pods stay immutable, or if we make pods mutable.

@verb
Copy link
Contributor Author

verb commented Jul 8, 2016

@thockin I'll grant you scary, but I'm not yet willing to concede that pulling in a different userland for a binary is any more opaque than your run-of-the-mill kubectl exec. Both seem solvable.

I'd be happy with an inactive container as long as it (optionally) automatically updates prior to activating so I'm not tied to an old debug image. The downside here is the extra boilerplate config in every pod I run.

@smarterclayton
Copy link
Contributor

I should note that we've had very good feedback from the "debug" command
(which spawns a copy of the pod with a shell, or other command, as the
target). It doesn't address the "I created a pod that has nothing useful
in it to debug", but it does address the bulk of the practical "w.t.h. is
this not starting up correctly" problems.

On Fri, Jul 8, 2016 at 7:16 PM, verb notifications@github.com wrote:

@thockin https://github.com/thockin I'll grant you scary, but I'm not
willing to concede that pulling in a different userland for a binary is any
more opaque than your run-of-the-mill kubectl exec. Both seem solvable.

I'd be happy with an inactive container as long as it (optionally)
automatically updates prior to activating so I'm not tied to an old debug
image. The downside here is the extra boilerplate config in every pod I run.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#27140 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABG_p2uVpzBsO1U8z8rTGXRCBPFQimVuks5qTtpXgaJpZM4IyRHU
.

@smarterclayton
Copy link
Contributor

Debug also leads naturally to other hooks like
"--attach-and-mount=IMAGENAME" for getting access to debug tools.

On Fri, Jul 8, 2016 at 7:31 PM, Clayton Coleman ccoleman@redhat.com wrote:

I should note that we've had very good feedback from the "debug" command
(which spawns a copy of the pod with a shell, or other command, as the
target). It doesn't address the "I created a pod that has nothing useful
in it to debug", but it does address the bulk of the practical "w.t.h. is
this not starting up correctly" problems.

On Fri, Jul 8, 2016 at 7:16 PM, verb notifications@github.com wrote:

@thockin https://github.com/thockin I'll grant you scary, but I'm not
willing to concede that pulling in a different userland for a binary is any
more opaque than your run-of-the-mill kubectl exec. Both seem solvable.

I'd be happy with an inactive container as long as it (optionally)
automatically updates prior to activating so I'm not tied to an old debug
image. The downside here is the extra boilerplate config in every pod I run.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#27140 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABG_p2uVpzBsO1U8z8rTGXRCBPFQimVuks5qTtpXgaJpZM4IyRHU
.

@verb
Copy link
Contributor Author

verb commented Jul 8, 2016

@erictune Certainly workable, but I'm concerned about some downsides:

  1. We'd want use the same image for QA as in prod, but then how do we QA the debug image? What if the stripped and debug images differ somehow?
  2. We're coupled with the debug environment we had at build time rather than the ones available at debug time
  3. The goal is minimal dependencies, but I'd have to include the debug tools (and all of their ongoing updates) as dependencies, which would be hard to track. (i.e. I need to apply a security update for an image that's not currently running but could be running if I needed to debug something in the future)

Even if I can't think of it right now, I'm confident there will eventually be a reason that I'll need to connect to a stripped container and it seems like a bit of a gap in the toolbox.

@pwittrock pwittrock added priority/backlog Higher priority than priority/awaiting-more-evidence. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Jul 18, 2016
@bgrant0607
Copy link
Member

Vaguely related to #831

@verb
Copy link
Contributor Author

verb commented Sep 12, 2017

/assign @verb

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 5, 2018
@yujuhong yujuhong removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 5, 2018
@kfox1111
Copy link

Until this support lands, I've been using this to debug:
https://github.com/pnnl-miscscripts/miscscripts/tree/master/bin

@verb
Copy link
Contributor Author

verb commented Oct 25, 2019

/priority important-longterm
/remove-priority backlog

@k8s-ci-robot k8s-ci-robot added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Oct 25, 2019
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 27, 2020
@verb
Copy link
Contributor Author

verb commented Jan 28, 2020

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 28, 2020
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 27, 2020
@verb
Copy link
Contributor Author

verb commented Apr 27, 2020

/remove-lifecycle stale

@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 27, 2020
@warmchang
Copy link
Contributor

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label May 28, 2020
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 26, 2020
@verb
Copy link
Contributor Author

verb commented Aug 27, 2020

This issue has become a sort of tracking issue for the overall problem of troubleshooting distroless containers, but tracking issues are a poor fit for k/k: the discussion dies down and it just into fighting lifecycle.

This issue motivated two enhancements: kubernetes/enhancements#277 (Ephemeral containers) and kubernetes/enhancements#1441 (kubectl debug). Let's track these efforts in their feature issues and consider this k/k issue closed.

@verb verb closed this as completed Aug 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Development

No branches or pull requests