Conversation

olavloite

Setting a timeout value for Partitioned DML would not be respected if the timeout value was higher than the timeout value set for the ExecuteSql RPC on the SpannerStub. Lower timeout values would be respected.

Fixes #199

@googlebotgooglebot added the cla: yesThis human has signed the Contributor License Agreement.label May 13, 2020
@olavloiteolavloite requested a review from skuruppu May 13, 2020 16:18
Comment on lines +345 to +346
.setInitialRpcTimeout(options.getPartitionedDmlTimeout())
.setMaxRpcTimeout(options.getPartitionedDmlTimeout())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are PDML requests not meant to be retried? I'm just curious why the initial and max timeouts are the same.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question.

I would say it's a a matter of probabilities and a little bit opinion:

  1. PDML is designed to accept long running update statements. That means that we should not automatically retry the statement if it takes longer than X time, as it would be impossible to find a good global value for X, as we don't know whether the statement is taking a long time because of a network problem or because the statement itself is taking a long time. Re-sending a long-running statement that is still running on the server would only hurt performance. My opinion is that for a PDML statement the chance that the statement is actually taking a long time is more probable than a network problem.
  2. The PDML documentation also clearly states that the update statement should be idempotent. This means that we should automatically retry it if the server is unavailable.

I think we should consider adding an additional method to the public API which would allow the users to specify a lower timeout for a specific statement, in addition to the current global setting. That would give the user more control over the timeout setting for statements that are known to take less time than the global setting.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added an extra test case to ensure that PDML is retried on UNAVAILABLE.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick debugging and fix @olavloite.

Setting a timeout value for Partitioned DML would not be respected if the timeout
value was higher than the timeout value set for the ExecuteSql RPC on the SpannerStub.
Lower timeout values would be respected.

Fixes #199
@olavloiteolavloite merged commit 13cb37e into master May 14, 2020
@olavloiteolavloite deleted the pdml-timeout branch May 14, 2020 06:45
gcf-merge-on-green bot pushed a commit that referenced this pull request May 19, 2020
🤖 I have created a release \*beep\* \*boop\* 
---
## [1.55.0](https://www..com/googleapis/java-spanner/compare/v1.54.0...v1.55.0) (2020-05-19)


### Features

* mark when a Spanner client is closed ([#198](https://www..com/googleapis/java-spanner/issues/198)) ([50cb174](https://www..com/googleapis/java-spanner/commit/50cb1744e7ede611758d3ff63b3df77a1d3682eb))


### Bug Fixes

* make it possible to override backups methods ([#195](https://www..com/googleapis/java-spanner/issues/195)) ([2d19c25](https://www..com/googleapis/java-spanner/commit/2d19c25ba32847d116194565e67e1b1276fcb9f8))
* Partitioned DML timeout was not always respected ([#203](https://www..com/googleapis/java-spanner/issues/203)) ([13cb37e](https://www..com/googleapis/java-spanner/commit/13cb37e55ddfd1ff4ec22b1dcdc20c4832eee444)), closes [#199](https://www..com/googleapis/java-spanner/issues/199)
* partitionedDml stub was not closed ([#213](https://www..com/googleapis/java-spanner/issues/213)) ([a2d9a33](https://www..com/googleapis/java-spanner/commit/a2d9a33fa31f7467fc2bfbef5a29c4b3f5aea7c8))
* reuse clientId for invalidated databases ([#206](https://www..com/googleapis/java-spanner/issues/206)) ([7b4490d](https://www..com/googleapis/java-spanner/commit/7b4490dfb61fbc81b5bd6be6c9a663b36b5ce402))
* use nanos to prevent truncation errors ([#204](https://www..com/googleapis/java-spanner/issues/204)) ([a608460](https://www..com/googleapis/java-spanner/commit/a60846043dc0ca47e1970d8ab99380b6d725c7a9)), closes [#200](https://www..com/googleapis/java-spanner/issues/200)


### Dependencies

* update dependency com.google.cloud:google-cloud-shared-dependencies to v0.3.1 ([#190](https://www..com/googleapis/java-spanner/issues/190)) ([ad41a0d](https://www..com/googleapis/java-spanner/commit/ad41a0d4b0cc6a2c0ae0611c767652f64cfb2fb7))
---


This PR was generated with [Release Please](https://.com/googleapis/release-please).
rajatbhatta pushed a commit to rajatbhatta/java-spanner that referenced this pull request Nov 17, 2022
* chore: updated versions.txt [ci skip]

* chore: updated samples/pom.xml [ci skip]

* chore: updated samples/install-without-bom/pom.xml [ci skip]

* chore: updated samples/snippets/pom.xml [ci skip]

* chore: updated pom.xml [ci skip]

* chore: updated samples/snapshot/pom.xml

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply..com>
Sign up for free to join this conversation on . Already have an account? Sign in to comment
cla: yesThis human has signed the Contributor License Agreement.
None yet

Successfully merging this pull request may close these issues.

Experienced '504 Deadline Exceeded' error when tried to delete large number rows by partitioned dml