We've been discussing it in a few places. Etcd 0.6 has proposed a pageable internal data structure which would potentially allow us to that efficiently. It's probably a ways out.

Some links I dug up for etcd-io/etcd#860:
http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api#pagination
https://blog.apigee.com/detail/restful_api_design_can_your_api_give_developers_just_the_information
https://cloud.google.com/appengine/docs/python/datastore/queries#Python_Query_cursors

Discussion in the store proposal: etcd-io/etcd#1634 (comment)

----- Original Message -----

Some links I dug up for etcd-io/etcd#860:
http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api#pagination
https://blog.apigee.com/detail/restful_api_design_can_your_api_give_developers_just_the_information
https://cloud.google.com/appengine/docs/python/datastore/queries#Python_Query_cursors

---

Reply to this email directly or view it on GitHub:
#2349 (comment)

We should try to use the same approach for paging for metrics, logs, events, lists, etc.
cc @vishh @timstclair @dchen1107

I'd prefer to go down the (prefix for start, next page token, limit) route
rather than pure numeric paging. Numeric paging doesn't really help
clients much and is a pain to optimize at the limit.

On Tue, Nov 17, 2015 at 1:21 PM, Brian Grant notifications@github.com
wrote:

We should try to use the same approach for paging for metrics, logs,
events, lists, etc.
cc @vishh https://github.com/vishh @timstclair
https://github.com/timstclair @dchen1107 https://github.com/dchen1107

—
Reply to this email directly or view it on GitHub
#2349 (comment)
.

@smarterclayton I agree. Best practice for Google APIs is to allow list request to specify a max size and page token (empty for first page), with each page returning an opaque next page token. To ensure opacity, page tokens are typically base64 encoded, and often contain structured data.

Yes, I think the internal rules around this are good and we should use them.

@bgrant0607 @smarterclayton I am getting a bit worried that someone is going to run a huge cluster and this will turn into an emergency overnight, but it's a fair amount of work, so I think we want to have it in before it becomes an emergency. Does it make sense to target this for 1.3? @krousey is interested in adding this to API server.

I wanted to wait for etcd3, because I suspect with protobuf + etcd we will
push back the urgency of this by a few months.

On Tue, Feb 2, 2016 at 1:29 PM, Daniel Smith notifications@github.com
wrote:

@bgrant0607 https://github.com/bgrant0607 @smarterclayton
https://github.com/smarterclayton I am getting a bit worried that
someone is going to run a huge cluster and this will turn into an emergency
overnight, but it's a fair amount of work, so I think we want to have it in
before it becomes an emergency. Does it make sense to target this for 1.3?
@krousey https://github.com/krousey is interested in adding this to API
server.

—
Reply to this email directly or view it on GitHub
#2349 (comment)
.

@smarterclayton How much work is there left to do on protobuf?

• Merge the serializer implementation
• Add dev tooling (protoc in an image to run hack/update-generated-protobuf)
• Finish the watch implementation refactor (to support multiple types of
  streaming protocol), which is mostly complete
• Determine a versioning policy on protobuf schema (internal only for 1.3,
  experimental, etc)
• Review the generated protobuf schema by an expert and the details of the
  streaming protocol
• Give clients command line flags to ask for protobuf (kubelet, controllers)
• Stress test the system with protobuf on
• Fix bugs

A few weeks of effort altogether and the necessary soak time

On Tue, Feb 2, 2016 at 11:58 PM, Brian Grant notifications@github.com
wrote:

@smarterclayton https://github.com/smarterclayton How much work is
there left to do on protobuf?

—
Reply to this email directly or view it on GitHub
#2349 (comment)
.

Chunking is now supported via the api