OK, will do! This is indeed necessary for future work. :)

+1 for starting with the Generic PLEG.
One thing to consider is the resource overhead of periodic relisting on the
docker daemon.

On Wed, Nov 4, 2015 at 6:12 PM, taotao notifications@github.com wrote:

OK, will do! This is indeed necessary for future work. :)

—
Reply to this email directly or view it on GitHub
#16831 (comment)
.

Now that we've added both a generic PLEG and a cache. We should re-evaluate the benefit of adopting the docker event stream and whether we should prioritize it.

Here is some benchmark result for this:
https://docs.google.com/document/d/1_mmIFY6MU8SovilagiAFXXwNSBHzdJTCdSj_8KdgZVg/edit?usp=sharing

I am familiar with the docker side of events. With a little guidance I can help do the work in the kube code.

/cc @rrati fyi to keep on the radar.

@MHBauer We are redefining the container runtime interface now #22964.
At least now, event is not in our short plan, because not all runtimes support event now, e.g. rkt. And according to our measurement, event stream does improve the performance, but the current polling based implementation is enough for now.

/cc @yujuhong

@MHBauer that's great.

As @Random-Liu pointed out, event stream will improve performance and resource usage for kubelet, but since 1) not all runtimes support this and 2) the benefits significant enough, we didn't prioritize switching to the event stream.

The problem is a little bit more complicated than simply interfacing with the docker event stream. If you are interested, here is some background.

Kubelet has a Pod Lifecycle Event Generator, PLEG that is compatible with all runtimes. PLEG does basically two things:

1. periodically polls docker (via docker ps -a) and compare the current container list with the last one. If there is any change, it generates an event.
2. In addition to generating an event, it also does docker inspect calls and updates an in-memory pod status cache.

Kubelet has a per-pod goroutine (aka a pod worker) checking a pod periodically. Every time a worker syncs a pod, it gets the current status from the pod status cache to avoid hitting docker directly. Before the next sync can start, the worker will need to wait until all the side-effects (i.e., events) from the previous sync have been observed by PLEG and the in-memory cache has been updated accordingly. This is important because otherwise the worker may create the same container again! To enforce this rule, PLEG records a timestamp on each relisting, to indicate how fresh the cache is. The worker will block until the cache is newer than its last sync.

If we want to use switch to docker event stream today, we'll need to

1. add a docker-specific PLEG.
2. handle the working syncing problem.

(2) is required because if kubelet doesn't receive any events over a period of time, it'd not know if there are truly no events, or the events are simply delayed. This is not the case for docker ps -a with an absolute timestamp to show how fresh the cache is. This issue can be handled by recording expected events, which requires modifying a non-trivial amount of kubelet code.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

This issue can be adapted to evaluate whether we need to support event stream in CRI. It is very low priority right now unless the need can be identified clearly.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close