Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using DNS names in Endpoints #13358

Closed
erez-rabih opened this issue Aug 30, 2015 · 33 comments
Closed

Using DNS names in Endpoints #13358

erez-rabih opened this issue Aug 30, 2015 · 33 comments
Labels
kind/design Categorizes issue or PR as related to design. priority/backlog Higher priority than priority/awaiting-more-evidence.

Comments

@erez-rabih
Copy link

Currently it is only possible to add IP addresses to Endpoints in Kubernetes.
What if I need to add a DNS name instead of IP address as a service endpoint?

The classic use case is AWS RDS instances which provide different Databases as a service.
They provide an endpoint which is a host name and not an IP address.

Is there a way to workaround this limitation?

@lavalamp lavalamp added team/api priority/backlog Higher priority than priority/awaiting-more-evidence. kind/design Categorizes issue or PR as related to design. labels Aug 31, 2015
@smarterclayton
Copy link
Contributor

I'd like to add a DNS to a service so that skydns would respond with a CNAME to the provided DNS as well. It's not quite headless and it's not quite setting the endpoints, but it's a variant of this use case.

@smarterclayton
Copy link
Contributor

The CNAME in many cases is superior to setting endpoints, because you can resolve to a destination DNS address for TLS.

@smarterclayton
Copy link
Contributor

@ncdc re: service linking as we discussed.

@thockin
Copy link
Member

thockin commented Sep 2, 2015

Brendan sent #11838 wherein we discussed a bunch of reasons why a hostname
in an endpoints list is a scary idea. In looking at that PR now, it looks
like some compromise was reached to store a resolved IP rather than the
hostname, but I still don't like it.

Can you explain what this issue is about relative to that PR?

On Tue, Sep 1, 2015 at 4:49 PM, Clayton Coleman notifications@github.com
wrote:

@ncdc https://github.com/ncdc re: service linking as we discussed.


Reply to this email directly or view it on GitHub
#13358 (comment)
.

@erez-rabih
Copy link
Author

@thockin Storing a resolved IP could not always be achieved.
For example, AWS ELBs are guaranteed to stay under the same host name, but their resolved IP may change and in fact some of the core functionality of ELBs in AWS lays in the fact that they resolve to different IPs upon different requests.
Currently, all external services that relay on dynamic IP resolving, cannot be integrated with Kubernetes cluster.

@thockin
Copy link
Member

thockin commented Sep 2, 2015

They can be used under their real names. They can not be aliased into
Kubernetes. That is a desireable feature, but I don't think a hostname
endpoint is the way to implement it.

On Tue, Sep 1, 2015 at 10:38 PM, erez notifications@github.com wrote:

@thockin https://github.com/thockin Storing a resolved IP could not
always be achieved.
For example, AWS ELBs are guaranteed to stay under the same host name, but
their resolved IP may change and in fact some of the core functionality of
ELBs in AWS lays in the fact that they resolve to different IPs upon
different requests.
Currently, all external services that relay on dynamic IP resolving,
cannot be integrated with Kubernetes cluster.


Reply to this email directly or view it on GitHub
#13358 (comment)
.

@smarterclayton
Copy link
Contributor

I'd prefer to have a hostname reference service. Name "lb" resolves to "
amazon-elb13482.aws.com". I think tls is still solvable although more
annoying.

On Sep 2, 2015, at 2:02 AM, Tim Hockin notifications@github.com wrote:

They can be used under their real names. They can not be aliased into
Kubernetes. That is a desireable feature, but I don't think a hostname
endpoint is the way to implement it.

On Tue, Sep 1, 2015 at 10:38 PM, erez notifications@github.com wrote:

@thockin https://github.com/thockin Storing a resolved IP could not
always be achieved.
For example, AWS ELBs are guaranteed to stay under the same host name, but
their resolved IP may change and in fact some of the core functionality of
ELBs in AWS lays in the fact that they resolve to different IPs upon
different requests.
Currently, all external services that relay on dynamic IP resolving,
cannot be integrated with Kubernetes cluster.


Reply to this email directly or view it on GitHub
<
#13358 (comment)

.


Reply to this email directly or view it on GitHub
#13358 (comment)
.

@Arjun2016
Copy link

I want to use AWS RDS instance which provide different Databases (Ex: postgres ) as a service. Now i want to add this service as an endpoint in kubernetes , but unfortunately the endpoint of RDS is a DNS name . The endpoint yaml does not allow the dns name in the ip field .

kind: "Endpoints"
apiVersion: "v1"
metadata:
name: "ext-postgres-service"
subsets:

addresses:
  -
    ip: "postgres.xxxxx.us-west-2.rds.amazonaws.com"
ports:
  -
    port: 5432

name: "postgres"

Is there a workaround this ?

@paralin
Copy link
Contributor

paralin commented Mar 31, 2016

Very interested in seeing updates to this.

Our use case: we want to have a namespace to test one specific component against external components. So we have http://api-external, http://api-internal. We want those to CNAME to http://api-external.dev.oursite.com/ for example. Because that is again CNAMEd to an AWS ELB, we cannot simply put an IP address for this, it won't work correctly.

@trinitronx
Copy link

As @Arjun2016 mentions, for certain AWS services (e.g.: RedShift, RDS, Elasticsearch Service, etc...) Amazon only provides a CNAME for accessing the database with no guarantee that the backend cluster IPs will remain the same. This is by design, as RDS is supposed to be able to dynamically scale up or down, and database cluster IPs will change over time with upgrades & scaling events. The only guarantee is that the CNAME or A record for the cluster will remain the same. It's usually given in the form: db-cluster-name.abcdefghi123.us-east-1.redshift.amazonaws.com. or <db-cluster-name>.<unique-string>.<region>.<rds_service>.amazonaws.com. For Elasticsearch, the form is: search-<cluster-name>-<unique-string>.<region>.es.amazonaws.com.

Only being able to tie an external endpoint to an IP restricts access from pods to services external to a Kubernetes cluster which are only guaranteed to be accessible via DNS names. Without the ability to use DNS names as an Endpoint, it enforces the use of workarounds such as setting up an external proxy host or ELB plus an Elastic IP, or other potentially fragile and overly complex solutions which may introduce additional points of failure.

Not sure what the design reasoning is behind only allowing IPs (and I'm sure there is probably some good reason), but clearly from a user standpoint, it's a necessary feature in certain circumstances. I'm interested in seeing the discussion continue on this.

@smarterclayton
Copy link
Contributor

smarterclayton commented Apr 4, 2016 via email

@trinitronx
Copy link

@smarterclayton: As far as Amazon has commented on load balancing RDS instances, it's currently not possible to load balance SQL requests to read replicas via an ELB, or via the CNAME (AFAIK). Not sure how they expect you to even be able to appropriately scale a read-heavy RedShift cluster without this, but that's what they give the customer. They recommend using HAProxy or "roll your own solution" for load balancing, however again there is no guarantee that the cluster node IPs will remain the same after scaling or upgrade events. In fact, their RedShift FAQ says:

Q: Will my data warehouse cluster remain available during scaling?

The existing data warehouse cluster remains available for read operations while a new data warehouse cluster gets created during scaling operations. When the new data warehouse cluster is ready, your existing data warehouse cluster will be temporarily unavailable while the canonical name record of the existing data warehouse cluster is flipped to point to the new data warehouse cluster. This period of unavailability typically lasts only a few minutes, and will occur during the maintenance window for your data warehouse cluster, unless you specify that the modification should be applied immediately. Amazon Redshift moves data in parallel from the compute nodes in your existing data warehouse cluster to the compute nodes in your new cluster. This enables your operation to complete as quickly as possible.

So, as it seems, they want you to access it solely via the single CNAME record. That is, unless you roll your own HAProxy solution and either manually or dynamically update the backend nodes when a cluster scaling event happens. It also appears that the CNAME record given resolves differently inside and outside of an AWS VPC. So, ideally... you'd want to resolve this CNAME via the AmazonProvidedDNS server (usually located at last IP octet of .2 within a VPC CIDR range). The CNAME itself is a bit unclear what it points to publicly, but points to the cluster "Leader" node inside a VPC. That is to say (in RedShift at least):

  • Both Inside & Outside your Amazon VPC: the CNAME resolves to an A record pointing to an EC2-like Public DNS name: ec2-12-34-56-78.compute-1.amazonaws.com.
  • Outside your Amazon VPC: ec2-12-34-56-78.compute-1.amazonaws.com. A record resolves to a Publicly routable IP (e.g.: 12.34.56.78) (in my experiment with RedShift, what lives at this IP was not obvious and was not listed as any of the listed cluster nodes... seems that it is probably an Elastic IP or some other instance that RDS sets up somewhere not manageable by the customer. It just magically gets you to a cluster node, presumably the leader)
  • Inside your Amazon VPC: ec2-12-34-56-78.compute-1.amazonaws.com. A record resolves to a Private IP inside the VPC CIDR range (e.g.: 10.123.4.5 perhaps if the example VPC range was: 10.123.0.0/16)
    • The Private IP that was given in my RedShift experiment turned out to be the "Leader" node's Private IP as listed on the RedShift Cluster's "Details" page under "Node IP Addresses:"

@smarterclayton
Copy link
Contributor

xref #13748 for service CNAME to another DNS address

@trinitronx
Copy link

To complicate matters further... with services such as Amazon ElasticSearch provide a CNAME, but also require the use of an access policy for security. However, the access policy choices are:

  • Resource-based access policy:
    • Use the standard AWS policy syntax to apply an access policy via a Principal and Resource
    • Requests must be signed (apps using the standard ElasticSearch API will not work unless you use their SDK or sign your own requests)
  • IP-based policy:
    • Allow specific IP addresses access to ElasticSearch API. This can be configured to allow anonymous access, which enables you to submit unsigned requests (use standard ElasticSearch API-capable apps)
  • IAM user and role-based access policies:
    • Use IAM-based users and roles for RBAC
    • Again: Requests must be signed (apps using the standard ElasticSearch API will not work unless you use their SDK or sign your own requests)

So the options are slim for connecting to it unless you find something that uses their SDK to sign ES requests, roll your own, or use an IP-based security policy for anonymous REST access. Again, this comes with it's own complexities if you're using an Auto Scaling Group for Kubernetes worker nodes... as they'll change over time for scaling or self-healing events. Role-based would work if you add the Role as an "Instance Profile" to the AutoScaling Group Launch Configuration, but this requires that you find something that'll sign requests (not easy at this point).

So, that pretty much leaves the "simplest" solution to use IP-based access for ES. So, for connecting up Kubernetes fluentd to AWS ElasticSearch, one use case for an external Service + Endpoint here would be:

  • Set up an EC2 instance (or tier of instances behind an ELB) with an Elastic IP to proxy requests to ElasticSearch via CNAME
  • Set up the Elastic IP as an allowed IP in the ElasticSearch security policy to allow anonymous REST access.
  • Set up a Kubernetes Service + Endpoint that can be used by pods internally for the ElasticSearch service. Set up an internal cluster name for the Elastic IP so fluentd can proxy requests to ElasticSearch
  • Set up a DaemonSet to run fluentd on every worker node for container log shipping. Configure the pods to use the Kubernetes Service name for ElasticSearch.

@therc
Copy link
Member

therc commented Apr 4, 2016

@thockin
Copy link
Member

thockin commented Apr 7, 2016

Is this a dup of #13748 - it's pretty darn close - can we flatten them?

@thockin
Copy link
Member

thockin commented Jul 5, 2016

I'm closing this as a dup, #13748 has more info

@thockin thockin closed this as completed Jul 5, 2016
@metal3d
Copy link
Contributor

metal3d commented May 29, 2017

I don't know why this issue is closed. It's not the same as #13748 because the title is about Endpoints.

I've got a situation where Endpoints object with IP addresses is not suffisent and the need of dns resolution is important: glusterfs-endpoint

In my case, my servers are behind routers that use NAT. So my endpoint to gluster are private ips (10.0.0.0/8) but after a restart, some server can change private IP. So, my endpoints are all wrong.

I've got private domain name (*.priv.scaleway.com) that are fixed. If Endpoins could use DNS instead, my problem is resolved.

Note: I'm using OpenShift 1.5.0

@smarterclayton
Copy link
Contributor

@metal3d this is probably a different bug then - it sounds like you need a way to control how gluster resolves those endpoints by having gluster look at the service, rather than just the endpoints? Or do you need gluster to refresh endpoints more frequently?

@metal3d
Copy link
Contributor

metal3d commented May 29, 2017

@smarterclayton I don't know if it's a bug :) I only find one way to assign gluster endpoints to persistent volumes: using endpoint object, and the only examples that I see are using IP addresses.
I sometime need to reboot gluster nodes (for different reasons) and ip are changing frequently. So at this time, I need to launch a Ansible playbook that removes every gluster enfpoints (because kubernetes ask to set endpoints in each namespaces) to recreate them with correct ip addresses.

At this time, glusterfs is the only solution I can use on my servers to be able to replicate data, and endpoints are not easy to administrate. Replicate endpoints in namespaces + no solution to use DNS instead of IP complicate a bit the management

PS: excuse my english

@metal3d
Copy link
Contributor

metal3d commented May 29, 2017

PS: I'm checking how to use storageclass that should resolve my problem

@paralin
Copy link
Contributor

paralin commented May 29, 2017

Why not just point everything at a clusterip on the service?

@metal3d
Copy link
Contributor

metal3d commented May 29, 2017

@paralin because persistent volume want a endpoint reference, and gluster is not installed via kubernetes/openshift, it's on different servers. Or I didn't understand what you mean

@thockin
Copy link
Member

thockin commented May 30, 2017

Endpoints can not hold hostnames. The spillover of complexity is not viable.

@WegDamit
Copy link

WegDamit commented Sep 4, 2018

When using docker edge on mac with kubernetes and hyperkit there is no IP for the host,
see Docker Docs

so when accessing an external Service via endpoint (eg. a db) on the host the workaround for docker is using "host.docker.internal",
which is not possible in k8s as the endpoint definition only accepts an IP.

What now?

@thockin
Copy link
Member

thockin commented Sep 17, 2018 via email

@onitake
Copy link

onitake commented Jan 18, 2019

ExternalName sounds like the best solution to this problem to me - if resolving the host name once is ok.
If you need something that resolves hosts dynamically, then you probably need a more complex setup than a simple endpoint could give you anyway.

@astorath
Copy link

@thockin @onitake
ExternalName does not provide any means to change destination port, this is crucial in some setups (like several DB instances on one machine).

@onitake
Copy link

onitake commented Apr 10, 2019

@astorath This is correct. ExternalName is implemented as a CNAME record on the cluster DNS, not via forwarding. If you want to change the port, you have to do it in the application.

@lukasmrtvy
Copy link

What about EndpointSlice ?

apiVersion: v1
kind: Service
metadata:
  name: k8s-host
  namespace: default
spec:
  ports:
  - protocol: TCP
    targetPort: 9090
    port: 80
---
apiVersion: discovery.k8s.io/v1beta1
kind: EndpointSlice
metadata:
  name: k8s-host
  namespace: default
  labels:
    kubernetes.io/service-name: k8s-host
addressType: FQDN 
ports:
  - port: 9090
endpoints:
  - addresses:
      - "k8s.host"

@matzegebbe
Copy link

@lukasmrtvy Does the EndpointSlice work for you? I still get (<error: endpoints "k8s-host" not found>) in my ingress

@MikeCongdon1
Copy link

MikeCongdon1 commented Apr 16, 2021

EndpointSlice doesn't work for me (I'm trying to access the local server / baremetal running a mariadb instance).

---
kind: "Service"
apiVersion: "v1"
metadata:
  name: mariadb
spec:
  ports:
    - name: mariadb
      protocol: "TCP"
      port: 3306
      targetPort: 3306
      nodePort: 0
---
apiVersion: discovery.k8s.io/v1beta1
kind: EndpointSlice
metadata:
  name: mariadb
  labels:
    kubernetes.io/service-name: "mariadb"
ports:
  - name: http
    protocol: TCP
    port: 3306
addressType: FQDN
endpoints:
  - addresses:
    - "localhost.localdomain"

doesn't work (also doesn't seem to work with my full FQDN instead of localhost...)
but:

---
kind: "Service"
apiVersion: "v1"
metadata:
  name: mariadb
spec:
  ports:
    - name: mariadb
      protocol: "TCP"
      port: 3306
      targetPort: 3306
      nodePort: 0
---

kind: "Endpoints"
apiVersion: "v1"
metadata:
  name: mariadb 
subsets:
  - addresses:
      - ip: x.x.0.108
    ports:
      - port: 3306
        name: mariadb 

works fine --- only, my IP for the endpoint will need to be something else for my fail-over server (which is x.x.1.108) instead.

Am I doing something wrong with my EndpointSlice config there? It looked like what the docs said to me.

@wu0407
Copy link

wu0407 commented Jun 21, 2021

maybe kube-proxy not support Endpointslice of FQDN

EndpointSlice doesn't work for me (I'm trying to access the local server / baremetal running a mariadb instance).

---
kind: "Service"
apiVersion: "v1"
metadata:
  name: mariadb
spec:
  ports:
    - name: mariadb
      protocol: "TCP"
      port: 3306
      targetPort: 3306
      nodePort: 0
---
apiVersion: discovery.k8s.io/v1beta1
kind: EndpointSlice
metadata:
  name: mariadb
  labels:
    kubernetes.io/service-name: "mariadb"
ports:
  - name: http
    protocol: TCP
    port: 3306
addressType: FQDN
endpoints:
  - addresses:
    - "localhost.localdomain"

doesn't work (also doesn't seem to work with my full FQDN instead of localhost...)
but:

---
kind: "Service"
apiVersion: "v1"
metadata:
  name: mariadb
spec:
  ports:
    - name: mariadb
      protocol: "TCP"
      port: 3306
      targetPort: 3306
      nodePort: 0
---

kind: "Endpoints"
apiVersion: "v1"
metadata:
  name: mariadb 
subsets:
  - addresses:
      - ip: x.x.0.108
    ports:
      - port: 3306
        name: mariadb 

works fine --- only, my IP for the endpoint will need to be something else for my fail-over server (which is x.x.1.108) instead.

Am I doing something wrong with my EndpointSlice config there? It looked like what the docs said to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/design Categorizes issue or PR as related to design. priority/backlog Higher priority than priority/awaiting-more-evidence.
Projects
None yet
Development

No branches or pull requests