com.example.spanner.SpannerSampleIT: testSample failed

[flaky-bot]

Note: #2412 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

---

commit: ed3aedf
buildURL: Build Status, Sponge
status: failed

Test output

com.google.cloud.spanner.SpannerException: RESOURCE_EXHAUSTED: Operation with name "projects/java-docs-samples-testing/instances/default-instance/databases/restored-8c2576f0-d59e-4290-9/operations/_auto_op_a94c5424b94b8592" failed with status = GrpcStatusCode{transportCode=RESOURCE_EXHAUSTED} and message = The instance projects/java-docs-samples-testing/instances/default-instance has too many database splits to complete the operation.  Please add more nodes and try again.
	at com.example.spanner.SpannerSampleIT.runSample(SpannerSampleIT.java:61)
	at com.example.spanner.SpannerSampleIT.testSample(SpannerSampleIT.java:314)
Caused by: com.google.cloud.spanner.SpannerException: RESOURCE_EXHAUSTED: Operation with name "projects/java-docs-samples-testing/instances/default-instance/databases/restored-8c2576f0-d59e-4290-9/operations/_auto_op_a94c5424b94b8592" failed with status = GrpcStatusCode{transportCode=RESOURCE_EXHAUSTED} and message = The instance projects/java-docs-samples-testing/instances/default-instance has too many database splits to complete the operation.  Please add more nodes and try again.

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

When run at the same commit (ed3aedf), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

[skuruppu]

@olavloite if you could please take a look, that would be great.

[olavloite]

@skuruppu Could you (or someone else) supply a list of the databases that are currently on this test instance?

I've been going through all the places where a database is created in the Java samples, but I have not been able to find any path where that database will not be deleted at the end of the test case. The error does however seem to indicate that there are too many databases on the instance, but I cannot verify that those databases are actually from the Java samples.

If the instance that is used for testing the Java samples is only used to test these samples, we could also add a 'delete all databases on this instance' to the tests.

[skuruppu]

Sorry @olavloite, I don't seem to have permission to list spanner instances so I can't see the databases. @lesv would you be able to help us with this?

I doubt that any other tests are using this particular spanner instance so I think it's safe to have some code that deletes stale databases (say > 4 hours old) at test startup time. @lesv please confirm if you think it's safe to do this.

[lesv]

Yes - but do it in a way that tolerates failure as the code might be running in several instances at the same time. (for example Java 8, Java 11 for several PR's)