GCE e2e build/test passed for commit 96d3e5462e5d292c76efcea9fad174790323a0f1.

• Build Log
• Test Artifacts
• Internal Jenkins Results

GCE e2e build/test passed for commit bf40ff2d4d6f8ded560161ead6027118e18204a8.

• Build Log
• Test Artifacts
• Internal Jenkins Results

cc @brendandburns

GCE e2e build/test passed for commit 00fcbb013a76f4b689c11fce236be0cf61337134.

• Build Log
• Test Artifacts
• Internal Jenkins Results

GCE e2e build/test passed for commit 6d21bdcaba1b5b3d2b90f20af23ee3b931924a06.

• Build Log
• Test Artifacts
• Internal Jenkins Results

GCE e2e build/test failed for commit 763719084c5c777e746b5afaea51ffe87594681d.

• Build Log
• Test Artifacts
• Internal Jenkins Results

GCE e2e build/test passed for commit 919dbf60516ebbd831a1a0d38e2dc99bce63d345.

• Build Log
• Test Artifacts
• Internal Jenkins Results

This would also enhance the apparent responsiveness of UIs in some cases (openshift/origin#3781). As just one example in OpenShift, a deployment's outcome is tied closely to the status of a Pod, and by proxy its containers; but no matter how fast the deployment container returns, the apparent running time can be inflated artificially by up to 10 seconds depending on where we land in the pod polling.

@yujuhong On the way back home, I realized that cadvisor only support 4 types events: containerCreation, containerDeletion, OOM, and OOMKilled, and none of them are the key event you want or sufficient enough for you to make any decision:

• containerCreation doesn't mean we can start a container.
• containerDeletion is only fired when container GC removes dead container
• The only termination event you have is OOMKilled when reach cgroup's memory limit
• When sys oom and kill a random process, you won't have container information. This is known issue in cAdvisor, but I don't think it is easy to fix.
• Other failures cause a termination of container, there is no event generated.

From above, you can see we pretty much cannot rely on cAdvisor's event stream at all, and I don't think there is an easy to add those missing events to cadvisor. Here is the code defined event:
https://github.com/google/cadvisor/blob/2a022a4a74f4f994c1ae7684de7655aaa0a898ba/info/v1/container.go#L521

Let's just stick on docker events for now. For rkt case, we could invoke relist operation all the time until they have event stream support.

cc/ @kubernetes/goog-node

Adding such strong dependency on cadvisor to provide information about lifecycle might block adding new container technologies in the future ( one might want to add for example lxc, qemu, ) and dependency for those drivers on something that is just metering component does not sound very good.

That being said I prefer option 1 for source of PLEG events ( Each runtime provides events by itself).

I believe the intention is to abstract out container runtimes using the cadvisor interface. Since cadvisor is already trying to handle container runtimes as first class entities, it seems natural to use it as an abstraction layer.
+1 to @dchen1107's arguments against using raw cadvisor events. May be we can extend cadvisor to support docker events?

This would also enhance the apparent responsiveness of UIs in some cases (openshift/origin#3781). As just one example in OpenShift, a deployment's outcome is tied closely to the status of a Pod, and by proxy its containers; but no matter how fast the deployment container returns, the apparent running time can be inflated artificially by up to 10 seconds depending on where we land in the pod polling.

This should definitely improve the response latency provided that kubelet is not already overwhelmed.

May be we can extend cadvisor to support docker events?

I am not sure this is a good idea though, but could be convinced. Docker is one implementation of container runtime, rkt is another one. To extend cadvisor to support docker event, we have to complicate cadvisor to make it composable too.

We will have to abstract out events from multiple container runtimes in
kubelet anyways. I assume your argument is whether cAdvisor is the right
abstraction layer?

On Thu, Aug 20, 2015 at 11:35 AM, Dawn Chen notifications@github.com
wrote:

May be we can extend cadvisor to support docker events?
I am not sure this is a good idea though, but could be convinced. Docker
is one implementation of container runtime, rkt is another one. To extend
cadvisor to support docker event, we have to complicate cadvisor to make it
composable too.

—
Reply to this email directly or view it on GitHub
#12802 (comment)
.

Some clarifications since we (me, @vishh and @dchen1107) discussed this offline :-)

@yujuhong On the way back home, I realized that cadvisor only support 4 types events: containerCreation, containerDeletion, OOM, and OOMKilled, and none of them are the key event you want or sufficient enough for you to make any decision:

containerCreation doesn't mean we can start a container.

containerCreation means that we tried to start a container. It may fail to start because of other reasons (image, etc), but we'll receive a containerDeletion in those cases.

@vishh brought up a valid point that this would make kubelet interprets "container starts" differently today. I agree with that, but personally I think this does not affect how kubelet works internally, and is not that critical.

containerDeletion is only fired when container GC removes dead container
The only termination event you have is OOMKilled when reach cgroup's memory limit
When sys oom and kill a random process, you won't have container information. This is known issue in cAdvisor, but I don't think it is easy to fix.
Other failures cause a termination of container, there is no event generated.

The conclusion from the discussion is that cAdvisor would send out a Deletion event if a container terminates because cgroups cleanup upon pid 1 exiting (thanks @vishh for pointing that out). I can verify this since I have tested my prototype by docker stop containers. kubelet reacts to that immediately.

From above, you can see we pretty much cannot rely on cAdvisor's event stream at all, and I don't think there is an easy to add those missing events to cadvisor. Here is the code defined event:
https://github.com/google/cadvisor/blob/2a022a4a74f4f994c1ae7684de7655aaa0a898ba/info/v1/container.go#L521

Let's just stick on docker events for now. For rkt case, we could invoke relist operation all the time until they have event stream support.

We will have to abstract out events from multiple container runtimes in
kubelet anyways. I assume your argument is whether cAdvisor is the right
abstraction layer?

I agree that we need an abstraction for events across different container runtimes.

cadvisor provides the bare-minimum event stream for kubelet to work, and we can move faster with it for now. That's why I chose cadvisor as a starting point. I think it's worth giving it a try since it should be easy to swap out the container event watcher.

That said, if the consensus is to use docker event, I'd be okay with that.

I think we need to define the event stream interface in kubelet. Whether we implement it using cadvisor or docker event stream should not matter.

👍

@yujuhong Can you update this based on our current roadmap, design and implemention.

GCE e2e test build/test passed for commit 919dbf60516ebbd831a1a0d38e2dc99bce63d345.

• Build Log
• Test Artifacts
• Internal Jenkins Results

GCE e2e build/test failed for commit 4faba2a3957276992366ecf02969ff57cb7f87af.

• Build Log
• Test Artifacts
• Internal Jenkins Results

@yujuhong Can you update this based on our current roadmap, design and implemention.

@dchen1107, I updated the proposal. Please take a look, thanks!

GCE e2e test build/test passed for commit 501ced30a43a23e34227b5868affce0447d01a80.

• Build Log
• Test Artifacts
• Internal Jenkins Results

GCE e2e test build/test passed for commit b8b532b.

• Build Log
• Test Artifacts
• Internal Jenkins Results

@k8s-bot test this issue: #IGNORE

Tests have been pending for 24 hours

GCE e2e test build/test passed for commit b8b532b.

• Build Log
• Test Artifacts
• Internal Jenkins Results

LGTM

Automatic merge from submit-queue