test: delete stale test databases by olavloite · Pull Request #4133 · GoogleCloudPlatform/java-docs-samples ·

Delete stale test databases that have the same name as those being created by this test.

spanner/cloud-client/src/test/java/com/example/spanner/SpannerSampleIT.java

kurtisvg · 2020-11-04T17:46:09Z

spanner/cloud-client/src/test/java/com/example/spanner/SpannerSampleIT.java

+for (Database db : dbClient.listDatabases(instanceId).iterateAll()) {
+if (db.getCreateTime().getSeconds() > 0 && TimeUnit.HOURS
+.convert(now.getSeconds() - db.getCreateTime().getSeconds(), TimeUnit.SECONDS) > 24) {
+if (pattern.matcher(toComparableId(baseDbId, db.getId().getDatabase())).matches()) {


Would it be less fragile to just use:
if db.getId.getDatabase().startsWith(baseDbID)?
Otherwise any adjustments to the formatForTest UUID will break this without any immediate feedback.

Well, yes, that would be possible, but I do worry about doing that: The method will delete all databases on the instance that meet the criteria, so if someone were to execute these tests on a random instance and the spanner.sample.database happens to be set to a fairly generic value (something like 'test' or 'sample'), it could potentially delete more databases than it should.
This is also only intended to be a fallback in case of unexpected test failures. The databases that are created by the tests are automatically deleted at the end of the test run, unless the test run is killed.

@kurtisvg Could you supply me with a list of the IDs of the databases that are currently present on the test instance? The build on Java 11 claims to be successful. I would have expected that to have deleted all stale test databases on the instance, but we are currently seeing a lot of build failures that seem to be caused by too many databases still being present on that instance. See for example #4175.

Project: Java-docs-samples-testing
Region: us-central1
ID: default-instance w/ 91 databases
bjfjf-e6ce3e71-2221-4000-a | 0.12% | 92.96 KB | | df-spanner-read-it-292649216 | 0.21% | 0 B | | jfodx-f432c184-5c6b-4d14-b | 0.16% | 80.46 KB | | mysample-09631a0d-0107-4fc5-a | 0.09% | 1.81 KB | | mysample-0b340a88-d096-4261-9 | 0.16% | 1.81 KB | | mysample-13337208-538b-4392-b | 0.12% | 1.81 KB | | mysample-14472e40-9349-4b8f-9 | 0.10% | 1.81 KB | | mysample-17263438-c62c-4ef0-9 | 0.21% | 1.81 KB | | mysample-17caeba3-ca10-4808-8 | 0.12% | 1.81 KB | | mysample-1a13079a-c6d6-4694-a | 0.21% | 212 B | | mysample-1d4c2207-f1aa-4556-a | 0.17% | 1.08 KB | | mysample-2878ac26-95f0-4348-b | 0.19% | 1.1 KB | | mysample-2b863578-1bab-410c-b | 0.10% | 1.81 KB | | mysample-32dc29c8-f074-4bdb-9 | 0.15% | 1.81 KB | | mysample-3d64a361-75da-4ea7-9 | 0.12% | 1.81 KB | | mysample-439adf53-4419-46ae-9 | 0.13% | 1.81 KB | | mysample-476db395-aa6e-4624-9 | 0.16% | 1.81 KB | | mysample-486d8bac-4077-4c11-8 | 0.09% | 1.81 KB | | mysample-4b940c0b-fffc-4caa-a | 0.17% | 1.81 KB | | mysample-4ca24702-bd50-41a4-a | 0.13% | 474 B | | mysample-4ceafb69-cd43-492a-b | 0.11% | 1.81 KB | | mysample-50918327-f321-4516-8 | 0.17% | 762 B | | mysample-6635ddaf-2bda-43ed-b | 0.11% | 1.81 KB | | mysample-6f48aa6b-159a-4a7a-8 | 0.23% | 1.81 KB | | mysample-70ea21c8-4a55-40d0-a | 0.10% | 732 B | | mysample-73af71b0-b212-4794-b | 0.10% | 603 B | | mysample-7819d309-ef65-4eab-a | 0.11% | 1.81 KB | | mysample-7cbe0849-f091-437e-9 | 0.14% | 1.81 KB | | mysample-7ead3ac1-f28a-43b5-8 | 0.18% | 729 B | | mysample-807e0e6a-9afd-46da-b | 0.08% | 1.81 KB | | mysample-82dc45c0-952d-467b-a | 0.10% | 1.81 KB | | mysample-84e5b267-4ff7-409e-8 | 0.08% | 1.81 KB | | mysample-860840f6-e13c-499d-a | 0.16% | 1.81 KB | | mysample-8aabfc92-4838-479b-b | 0.18% | 0 B | | mysample-90037d1b-cc70-4f75-b | 0.19% | 1.81 KB | | mysample-9dbe20a4-d4a0-49a7-a | 0.14% | 1.43 KB | | mysample-ad1162e9-205b-4274-8 | 0.13% | 550 B | | mysample-af602369-6897-445e-9 | 0.12% | 651 B | | mysample-b22cc310-fd3c-4db0-b | 0.12% | 1.4 KB | | mysample-bb4fb4cd-7317-4b31-9 | 0.43% | 708 B | | mysample-c22adce8-0466-4781-9 | 0.13% | 1.81 KB | | mysample-c62bcee2-17b3-4423-9 | 0.08% | 1.81 KB | | mysample-c8fdfb9b-021f-4ab1-8 | 0.11% | 1.81 KB | | mysample-d744d126-1405-4a66-a | 0.12% | 1.81 KB | | mysample-daf852ef-98f9-4d54-a | 0.19% | 815 B | | mysample-def739ce-d1ce-4e5e-a | 0.10% | 1.81 KB | | mysample-e185f237-e813-4cbf-b | 0.17% | 1.81 KB | | mysample-ebd1c05e-8ddf-420b-b | 0.22% | 354 B | | mysample-fce9edf0-0bb2-4d5c-b | 0.12% | 1.81 KB | | occvu-0a80960f-697d-4609-b | 0.14% | 97.46 KB | | quickstart-db | 0.14% | 89 B | | restored-004abec7-52a9-4918-a | 0.12% | 1.86 KB | | restored-0b340a88-d096-4261-9 | 0.12% | 1.81 KB | | restored-12bd398b-0972-4ab4-b | 0.12% | 1.86 KB | | restored-1d4c2207-f1aa-4556-a | 0.11% | 1.86 KB | | restored-213afcab-89df-4219-a | 0.24% | 1.86 KB | | restored-2878ac26-95f0-4348-b | 0.15% | 1.86 KB | | restored-2aafc9b2-72ab-4aec-b | 0.19% | 1.86 KB | | restored-2b863578-1bab-410c-b | 0.11% | 1.81 KB | | restored-2f9a1547-fe1e-48f3-9 | 0.11% | 1.86 KB | | restored-3060273a-5185-470a-8 | 0.08% | 1.86 KB | | restored-3bb24fe6-7fb7-44af-b | 0.15% | 1.86 KB | | restored-4b05449a-4c8f-4769-b | 0.16% | 1.86 KB | | restored-4b2de289-6000-463d-8 | 0.07% | 1.86 KB | | restored-50918327-f321-4516-8 | 0.24% | 1.86 KB | | restored-55e86d1a-b7cd-4af2-b | 0.11% | 1.86 KB | | restored-61a6eb6b-b9a6-4b65-a | 0.12% | 1.86 KB | | restored-77c6e8b1-f5b3-4ca8-a | 0.12% | 1.86 KB | | restored-790f8c0f-3106-4d53-8 | 0.13% | 1.86 KB | | restored-79a3039b-9774-4ad3-a | 0.16% | 1.86 KB | | restored-7cbe0849-f091-437e-9 | 0.09% | 1.81 KB | | restored-7e1e2fdd-4e75-4c2c-8 | 0.12% | 1.86 KB | | restored-7ead3ac1-f28a-43b5-8 | 0.32% | 1.86 KB | | restored-8055ce7a-b316-4414-9 | 0.10% | 1.86 KB | | restored-82dc45c0-952d-467b-a | 0.10% | 1.81 KB | | restored-9dbe20a4-d4a0-49a7-a | 0.25% | 1.86 KB | | restored-a90ec3d8-9d3e-4f89-b | 0.12% | 1.86 KB | | restored-af602369-6897-445e-9 | 0.17% | 1.86 KB | | restored-afcf21cf-e927-4c6b-9 | 0.15% | 1.86 KB | | restored-bb4fb4cd-7317-4b31-9 | 0.09% | 1.86 KB | | restored-c22adce8-0466-4781-9 | 0.14% | 1.81 KB | | restored-c62bcee2-17b3-4423-9 | 0.11% | 1.81 KB | | restored-c8fdfb9b-021f-4ab1-8 | 0.13% | 1.81 KB | | restored-d744d126-1405-4a66-a | 0.11% | 1.81 KB | | restored-daf852ef-98f9-4d54-a | 0.12% | 1.86 KB | | restored-e185f237-e813-4cbf-b | 0.12% | 1.81 KB | | restored-mysample-16ae8df9-7ec | 0.10% | 1.81 KB | | sejzn-3d2384a5-1344-4bff-8 | 0.18% | 18.46 KB | | vzdqp-da80840c-38c5-430b-a | 0.16% | 65.96 KB | | wfwyd-f251339f-d247-4031-b | 0.14% | 24.46 KB | | zfidi-cecbc595-7e56-4ee9-9 | 0.07% | 109.46 KB |

There is also samples-926dcd35-e183-4b8c-a in asia w/ 0 databases.

Maybe it would be better to remove this logic from the test all together, and instead just put it in a Cloud Function that runs periodically to clean up stale resources in the project.

@lesv could you delete the existing databases? We are seeing errors when running the samples test in our other repository as well (we are in the process of migrating the samples from this repository to the other one - https://.com/googleapis/java-spanner/tree/master/samples)

There is also samples-926dcd35-e183-4b8c-a in asia w/ 0 databases.
As far as I'm concerned, that one may be deleted. I can't see that it is used by anything in the Java samples.

Thanks for the list of the existing databases. That helped a lot.

@kurtisvg PTAL

Java 8 is failing

Java 8 is failing

It fails with the following error:

com.google.cloud.spanner.SpannerException:
RESOURCE_EXHAUSTED: com.google.api.gax.rpc.ResourceExhaustedException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Administrative requests' and limit 'Administrative requests per minute'

Java 8 is failing

It fails with the following error:

com.google.cloud.spanner.SpannerException:
RESOURCE_EXHAUSTED: com.google.api.gax.rpc.ResourceExhaustedException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Administrative requests' and limit 'Administrative requests per minute'

For those, we were asked by the Spanner team to retry the tests. In Node.js, we retry the admin requests with some exponential backoff to get around this problem.

Java 8 is failing
It fails with the following error:
com.google.cloud.spanner.SpannerException:
RESOURCE_EXHAUSTED: com.google.api.gax.rpc.ResourceExhaustedException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Quota exceeded for quota metric 'Administrative requests' and limit 'Administrative requests per minute' 
For those, we were asked by the Spanner team to retry the tests. In Node.js, we retry the admin requests with some exponential backoff to get around this problem.

The Java client has a built-in option to throttle the number of admin requests. See #4178

We will need to do this fix in https://.com/googleapis/java-spanner/tree/master/samples, since we are in the progress of moving samples to that repository.

test: delete stale test databases

e851b3d

google-cla bot added the cla: yesThis human has signed the Contributor License Agreement.label Nov 3, 2020

olavloite added 3 commits November 4, 2020 08:19

test: actually delete databases

c4cffc8

fix: fix NPE

db8c168

fix: only remove stale databases once

0872b64

olavloite marked this pull request as ready for review November 4, 2020 10:51

olavloite requested review from a team, skuruppu and lesv November 4, 2020 10:51

olavloite changed the title ~~[WIP] test: delete stale test databases~~ test: delete stale test databases Nov 4, 2020

blunderbuss-gcf bot assigned kurtisvg Nov 4, 2020

kurtisvg suggested changes Nov 4, 2020
View reviewed changes

fix: reuse baseDbId

0816514

olavloite added the kokoro:force-runAdd this label to force Kokoro to re-run the tests.label Nov 4, 2020

kokoro-team removed the kokoro:force-runAdd this label to force Kokoro to re-run the tests.label Nov 4, 2020

skuruppu requested a review from thiagotnunes November 5, 2020 10:31

lesv approved these changes Nov 6, 2020
View reviewed changes

thiagotnunes approved these changes Nov 6, 2020
View reviewed changes

olavloite added 8 commits November 6, 2020 10:08

fix: also delete stale restored databases

8d14233

test: force logging of non-deleted db

34efac7

fix: log non-deleted databases

008f634

fix: show age in hours

78b593e

fix: drop dbs with length at least 20

49d14ee

fix: drop dbs with length at least 20

dc0ba7a

fix: delete all restored db's that have a long name

619674d

fix: use different patterns for sample and restored

b98b43e

fix: remove deleted check as it is no longer needed

37c8816

olavloite requested a review from kurtisvg November 6, 2020 11:14

kurtisvg approved these changes Nov 6, 2020
View reviewed changes

lesv merged commit f8a99e0 into GoogleCloudPlatform:master Nov 6, 2020

olavloite deleted the spanner-delete-stale-databases branch November 13, 2020 07:20

olavloite mentioned this pull request Nov 13, 2020
fix: delete stale sample databases googleapis/java-spanner#622
Merged

Uh oh!

Uh oh!

Uh oh!

kurtisvg Nov 4, 2020

Uh oh!

olavloite Nov 4, 2020

Uh oh!

olavloite Nov 5, 2020

Uh oh!

lesv Nov 5, 2020

Uh oh!

lesv Nov 5, 2020

Uh oh!

kurtisvg Nov 5, 2020

Uh oh!

thiagotnunes Nov 6, 2020

Uh oh!

olavloite Nov 6, 2020

Uh oh!

olavloite Nov 6, 2020

Uh oh!

olavloite Nov 6, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Conversation

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!