ALTS: GRPCLB LoadBalancer had an error

[veblush]

When running GCS benchmark over DirectPath with gRPC 1.34.0-SNAPSHOT, it failed with the error below. It has been working with gRPC 1.33.1.

What version of gRPC-Java are you using?

gRPC 1.34.0-SNAPSHOT

What is your environment?

Linux Debian/10

What did you expect to see?

Connecting to the backend using ALTS.

What did you see instead?

GRPCLB Error

Exception in thread "main" io.grpc.StatusRuntimeException: UNAUTHENTICATED: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
Stream to GRPCLB LoadBalancer had an error
        at io.grpc.Status.asRuntimeException(Status.java:533)
        at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:648)
        at io.grpc.gcs.GrpcClient.makeMediaRequest(GrpcClient.java:179)
        at io.grpc.gcs.GrpcClient.startCalls(GrpcClient.java:106)
        at io.grpc.gcs.TestMain.main(TestMain.java:52)

Steps to reproduce the bug

java -jar $HOME/grpc-gcp-java/end2end-test-examples/gcs/libs_1.34.0/gcs-1.0-SNAPSHOT.jar --client=grpc --bkt=$GCS_BUCKET --obj=$GCS_OBJECT --method=read --dp=true --calls=1

[veblush]

1.34.0_error.txt is a debug log of gRPC.

[sanjaypujare]

There is an earlier error in the log. Is that related or is that also seen with 1.33.1

Nov 19, 2020 2:25:10 AM io.grpc.grpclb.GrpclbNameResolver resolveBalancerAddresses
FINE: Balancer resolution failure
javax.naming.OperationNotSupportedException: DNS service refused [response code 5]; remaining name '_grpclb._tcp.metadata.google.internal.'
	at com.sun.jndi.dns.DnsClient.checkResponseCode(DnsClient.java:663)
	at com.sun.jndi.dns.DnsClient.isMatchResponse(DnsClient.java:578)
	at com.sun.jndi.dns.DnsClient.doUdpQuery(DnsClient.java:426)
	at com.sun.jndi.dns.DnsClient.query(DnsClient.java:211)
	at com.sun.jndi.dns.Resolver.query(Resolver.java:81)
	at com.sun.jndi.dns.DnsContext.c_getAttributes(DnsContext.java:434)
	at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:235)
	at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:141)
	at com.sun.jndi.toolkit.url.GenericURLDirContext.getAttributes(GenericURLDirContext.java:103)
	at javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:142)
	at io.grpc.internal.JndiResourceResolverFactory$JndiRecordFetcher.getAllRecords(JndiResourceResolverFactory.java:216)
	at io.grpc.internal.JndiResourceResolverFactory$JndiResourceResolver.resolveSrv(JndiResourceResolverFactory.java:133)
	at io.grpc.grpclb.GrpclbNameResolver.resolveBalancerAddresses(GrpclbNameResolver.java:79)
	at io.grpc.grpclb.GrpclbNameResolver.doResolve(GrpclbNameResolver.java:62)
	at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:318)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[sanjaypujare]

@dapengzhang0 any idea what changes in grpclb could have caused this?

[jiangtaoli2016]

@apolcyn
It may relate to Identity bound token enforced by gRPC LB.
The problem is only on a particular version of grpc?

[apolcyn]

I think the '_grpclb._tcp.metadata.google.internal.' query failure is expected, and just happens because we have SRV queries enabled (so the inner ALTS channel will also make the SRV query). However, metadata.google.internal. isn't a load balanced name, so that query should fail.

+1 that this seems likely related to ID bound token. Can we check what the target name (https://.com/grpc/grpc/blob/e6e6be4b0b12562e38083181aee27438d8258b5a/src/proto/grpc/gcp/handshaker.proto#L104) is that is passed to the ALTS handshake service?

[veblush]

I haven't seen this issue with gRPC Java prior to 1.34. I just checked that gRPC 1.33.1 doesn't have this issue. So commits after v1.33.1 most likely caused this.

[jiangtaoli2016]

I don't think Id bound token rollout in grpclb touches grpc code. It is all backend enabling. Let me check with Jianing who is responsible for Id bound token.

[apolcyn]

Although ID bound token rollout itself doesn't touch grpc code, if I'm correct it does newly require the client to pass ALTS target names and RPC authority headers on the LB channel properly, which might have changed at the client

[jiangtaoli2016]

Confirmed with Jianing, rollout only change ESF config and fallback code is still there.

I haven't seen this issue with gRPC Java prior to 1.34. I just checked that gRPC 1.33.1 doesn't have this issue. So commits after v1.33.1 most likely caused this.

Based on this, it looks like some changes in gRPC code caused the failure.

[dapengzhang0]

There wasn't much change to grpclb/alts between 1.33.1 and 1.34. I suspect #7613 or #7589 which was added to 1.34 is causing the problem.

[ejona86]

This should have been a release blocker for v1.34 since it seems to be a serious regression.

gRPCLB doesn't provide any call credentials to the server. If the server now requires it, that would definitely break gRPC. But why would older versions succeed? That's the important question.

Looking at the logs, it appears that the gRPCLB request includes a JWT. That's surprising. I wonder if the swap to ChannelCredentials somehow added it.

[ejona86]

This is a regression caused by ChannelCredentials. ManagedChannelImpl injects the call credentials to the existing CallCredentialsApplyingTransportFactory which is used by both normal and OOB connections.