Description
Is your feature request related to a problem? Please describe.
We are using grpc-xds and need a cluster name attirbute to be logged in our metrics for more granularity on how are our requests are flowing between the system.
What we have figured out till now?
https://github.com/grpc/grpc-java/blob/b6947de95a5b4d46da8ce5cd2bc50f89ade9db08/core/src/main/java/io/grpc/internal/GrpcAttributes.java#L42C58-L42C58
Hence in a client interceptor of GRPC one can do:
getAttributes().get(GrpcAttributes.ATTR_CLIENT_EAG_ATTRS).get(InternalXdsAttributes.ATTR_CLUSTER_NAME)
In the above example I am avoiding null checks.
Let me also highlight in brief what this attirbute is and why this is an important piece of information. With istio and grpc-xds we can use grpc as a client side load-balancer and with a single kubernetes cluster endpoint like
grpc-server.istio-poc.svc.cluster.local
We can create multiple subsets using a destination rule / virtual service like
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: grpc-server-destination
namespace: istio-poc
spec:
host: grpc-server.istio-poc.svc.cluster.local
subsets:
- name: server1
labels:
server: server-1
- name: server2
labels:
server: server-2
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: grpc-server-route
namespace: istio-poc
spec:
hosts:
- grpc-server.istio-poc.svc.cluster.local
http:
- match:
- headers:
SERVER:
exact: "server-1"
route:
- destination:
host: grpc-server.istio-poc.svc.cluster.local
subset: server1
- match:
- headers:
SERVER:
exact: "server-2"
route:
- destination:
host: grpc-server.istio-poc.svc.cluster.local
subset: server2
What the above is doing is basically, when the header is server-1 then request goes to pod with label as server-1 and when the header passed is server-2 then it goes to pod with labels as server-2
The above example may look like a very specific use case, but it is just an example the config can be very complicated based on path, locality (region, zone), staging - canary and many other conditions basically your cluster can be split behind a single endpoint and grpc-xds (acting as client side load balancer) decides the right servers to connect using the destination rules and virtual services.
But when we see metrics net_peer_name in all the above cases we will be seeing grpc-server.istio-poc.svc.cluster.local while internally they are subdividing.
The actual information is available in the above cluster name attribute that I mentioned as follows:
outbound|9000|server1|grpc-server.istio-poc.svc.cluster.local
or
outbound|9000|server2|grpc-server.istio-poc.svc.cluster.local
and helps to further segregates for the end-user. Because sometimes the complexity of the topology may yield in benefit to keeping only one endpoint but one might want to view the granularities separately.
Describe the solution you'd like
So we understand the flow that first this needs to be added as an attribute to the GRPC Call span and then it needs to be extarcted while adding metrics.
The main problem here is adding attribute to the current span by the agent, either my all application have to add this instrumentation manually or I can contribute to the opentelemetry repository create a new instrumentation with grpc-xds and when this is present a set of interceptor and labels can be added as attribute to the current grpc span.
Then I can use a custom attributes view in my opentelemetry extension module and view this as label, because I am not sure if everyone else will require this as label in their own setup.
Is there a better approach and how do I decide what to put in extension and contribute to the repo?
Describe alternatives you've considered
No response
Additional context
No response