Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Long time for starting container when many pods are scheduled. #12099

Closed
thereallukl opened this issue Jul 31, 2015 · 4 comments
Closed

Long time for starting container when many pods are scheduled. #12099

thereallukl opened this issue Jul 31, 2015 · 4 comments
Assignees
@yujuhong
Labels
area/kubelet priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/scalability Categorizes an issue or PR as relevant to SIG Scalability.
Milestone
v1.2-candidate

Comments

@thereallukl
Copy link

I am trying to start about 50 pods on a minion, and it is taking about 2-3 minutes to start first pause container and over 15 minutes to start all containers for all pods. In docker logs I can see that kubelet is constantly listing running pods, but it doesn't start missing ones. Additionally in the same time I cannot list images, docker images is taking minutes to return anything ( after all pods are created, it starts working again ). I presume that kubelet is creating more requests to docker than daemon can handle.

I am using:
docker 1.7.1
kubernetes 0.19.3
kernel: 4.0.9-coreos
@thereallukl thereallukl mentioned this issue Jul 31, 2015
Closed
@fgrzadkowski
Copy link
Contributor

@wojtek-t @dchen1107

@wojtek-t
Copy link
Member

wojtek-t commented Aug 3, 2015

Currently Kubelet doesn't support running more that 30 pods per node. So this is not surprising that Kubelet cannot handle it now.
IIUC, Kubelet team is looking into using Docker event stream instead of relisting everything frequently. This should help a lot.
So we should be able to support more than 30 pods per node in the future (hopefully 1.1 release), but it's not the case now.

@yujuhong yujuhong added area/kubelet sig/scalability Categorizes an issue or PR as relevant to SIG Scalability. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Aug 3, 2015
@yujuhong yujuhong self-assigned this Aug 3, 2015
@dchen1107
Copy link
Member

Using event stream is currently targeted for our 1.1 release, so that we could improve kubelet / docker performance and scalability to some degree.

@dchen1107 dchen1107 added this to the v1.1 milestone Aug 4, 2015
@yujuhong yujuhong mentioned this issue Aug 11, 2015
Closed
@mbforbes mbforbes added the priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. label Aug 16, 2015
@dchen1107 dchen1107 modified the milestones: v1.2, v1.1 Sep 15, 2015
@bgrant0607 bgrant0607 modified the milestones: v1.2, v1.2-candidate Oct 9, 2015
@yujuhong
Copy link
Contributor

We decided to not go with docker event stream for v1.2, and save that for v1.3 or beyond. That said, we've still made rather significant changes to kubelet, so that it won't hammer docker with numerous concurrent requests.

With HEAD (b3bc741) and docker v1.9.1, the time to start 50 pause pods (and wait until they are all running and ready) is ~1min. If you loosen the apiserver client QPS limit, the time becomes around 40 to 50s.

I am going to close this issue. If you still see long startup time in batch creation, feel free to open another issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yujuhong yujuhong

Labels
area/kubelet priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/scalability Categorizes an issue or PR as relevant to SIG Scalability.
Projects
None yet
Milestone
v1.2-candidate
Development

No branches or pull requests
7 participants
@mbforbes @bgrant0607 @dchen1107 @yujuhong @wojtek-t @fgrzadkowski @thereallukl