Conversation

olavloite

Reduces the probability of RESOURCE_EXHAUSTED errors during tests by making the GetOperation method retry errors with this code with an exponential backoff. The GetOperation method is called repeatedly for long-running operations by a polling future. These calls also count towards the max 5 admin requests per second.

Fixes #733

Reduces the probability of RESOURCE_EXHAUSTED errors during tests by making the
GetOperation method retry errors with this code with an exponential backoff.
The GetOperation method is called repeatedly for long-running operations by a
polling future. These calls also count towards the max 5 admin requests per second.

Fixes #733
@olavloiteolavloite requested a review from a team as a code owner December 13, 2020 19:06
@product-auto-labelproduct-auto-label bot added the api: spannerIssues related to the googleapis/java-spanner API.label Dec 13, 2020
@google-clagoogle-cla bot added the cla: yesThis human has signed the Contributor License Agreement.label Dec 13, 2020
Comment on lines 457 to 461
callSettings =
callSettings
.toBuilder()
.setRetryableCodes(codes)
.build();

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
callSettings =
callSettings
.toBuilder()
.setRetryableCodes(codes)
.build();
callSettings = callSettings.toBuilder().setRetryableCodes(codes).build();

ImmutableSet.<StatusCode.Code>builderWithExpectedSize(
callSettings.getRetryableCodes().size() + 1)
.addAll(callSettings.getRetryableCodes())
.add(StatusCode.Code.RESOURCE_EXHAUSTED)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be preferable if this could be changed in the generated code, but I don't know whether that is possible considering this comes from the generic com.google.longrunning.stub.OperationsStub.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems we would need this only for fixing the tests (correct me if I am wrong). If that is the case, changing production behaviour to fix those might not be the best path forward. Could we inject different behaviour in test only.

On another note, do we still need this change now that the tests seem to be working as expected?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated the PR to only change this behavior if SpannerOptions#setAutoThrottleAdministrativeRequests has been set. This means that it will only be enabled for (integration) tests.

@codecov

Codecov Report

Merging #734 (76bcbec) into master (aa701f5) will decrease coverage by 0.11%.
The diff coverage is 18.51%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #734      +/-   ##
============================================
- Coverage     85.15%   85.03%   -0.12%     
- Complexity     2562     2563       +1     
============================================
  Files           142      143       +1     
  Lines         13960    13983      +23     
  Branches       1331     1335       +4     
============================================
+ Hits          11887    11890       +3     
- Misses         1513     1532      +19     
- Partials        560      561       +1     
Impacted FilesCoverage ΔComplexity Δ
...rpcDatabaseAdminStubWithCustomCallableFactory.java0.00% <0.00%> (ø)0.00 <0.00> (?)
...m/google/cloud/spanner/spi/v1/GapicSpannerRpc.java81.35% <20.00%> (-2.04%)81.00 <0.00> (ø)
...ud/spanner/SessionPoolAsyncTransactionManager.java87.30% <0.00%> (+1.58%)13.00% <0.00%> (+2.00%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aa701f5...76bcbec. Read the comment docs.

@thiagotnunesthiagotnunes merged commit cd946d7 into master Dec 16, 2020
@thiagotnunesthiagotnunes deleted the reduce-resource-exhausted-errors branch December 16, 2020 22:54
@thiagotnunesthiagotnunes mentioned this pull request Dec 17, 2020
thiagotnunes pushed a commit that referenced this pull request May 6, 2021
…#734)

* fix: reduce the probability of RESOURCE_EXHAUSTED errors during tests

Reduces the probability of RESOURCE_EXHAUSTED errors during tests by making the
GetOperation method retry errors with this code with an exponential backoff.
The GetOperation method is called repeatedly for long-running operations by a
polling future. These calls also count towards the max 5 admin requests per second.

Fixes #733

* fix: use default retry settings

* chore: run formatter

* fix: only retry RESOURCE_EXHAUSTED when throttling requests
ansh0l pushed a commit to ansh0l/java-spanner that referenced this pull request Nov 10, 2022
This is an auto-generated regeneration of the .pb.go files by
cloud.google.com/go/internal/gapicgen. Once this PR is submitted, genbot will
update the corresponding PR to depend on the newer version of go-genproto, and
assign reviewers. Whilst this or any regen PR is open in go-genproto, genbot
will not create any more regeneration PRs. If all regen PRs are closed,
gapicgen will create a new set of regeneration PRs once per night.

If you have been assigned to review this PR, please:

- Ensure that CI is passing. If it's failing, it requires your manual attention.
- Approve and submit this PR if you believe it's ready to ship. That will prompt
genbot to assign reviewers to the google-cloud-go PR.

Corresponding google-cloud-go PR: googleapis/google-cloud-go#5245

Changes:

chore: regenerate API index

  Source-Link: googleapis/googleapis@8d996c9

feat: sync with latest API updates * Refresh documentation * Messages related to terminal point service were removed * The already deprecated available_capacity field was removed * Route information added to UpdateTrip
  PiperOrigin-RevId: 417686942
  Source-Link: googleapis/googleapis@f7cdb41
rajatbhatta pushed a commit to rajatbhatta/java-spanner that referenced this pull request Nov 17, 2022
rajatbhatta pushed a commit to rajatbhatta/java-spanner that referenced this pull request Nov 17, 2022
🤖 I have created a release *beep* *boop*
---


### [2.5.10](googleapis/java-spanner-jdbc@v2.5.9...v2.5.10) (2022-02-09)


### Dependencies

* update dependency com.google.cloud:google-cloud-spanner-bom to v6.18.0 ([googleapis#734](googleapis/java-spanner-jdbc#734)) ([52f407a](googleapis/java-spanner-jdbc@52f407a))

---
This PR was generated with [Release Please](https://.com/googleapis/release-please). See [documentation](https://.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on . Already have an account? Sign in to comment
api: spannerIssues related to the googleapis/java-spanner API.cla: yesThis human has signed the Contributor License Agreement.
None yet

Successfully merging this pull request may close these issues.

spanner.it.ITDatabaseAdminTest: updateDdlRetry failed