[gb-8]

Background

The integration test ClientScriptExecutionCanBeCancelledWhenRetriesAreEnabled.DuringStartScript_ForListeningTentacle_ThatIsRetryingTheRpc_AndConnecting_ScriptExecutionCanBeCancelled sometimes times out after 120s. It happened about once in 100 runs, most recently on Amazon Linux 2023 in build 24907377, which failed Full Chain 13.0.4673. The failure is tracked in LEV-2242. LEV-869, for the same test on Amazon Linux 2, is very likely the same problem.

The cause is a race in the client's cancellation handling, not the infrastructure.

1. StartScript is retrying because Tentacle is offline and every attempt fails with a connecting HalibutClientException.
2. Normally the cancellation lands during Polly's sleep between retries. The sleep throws TaskCanceledException, RpcCallRetryHandler catches it and rethrows the last real exception (the connecting exception). StartScript then correctly concludes that the request never connected and exits without calling CancelScript.
3. If the cancellation lands just as a sleep finishes, Polly throws a plain OperationCanceledException (from ThrowIfCancellationRequested) instead. RpcCallRetryHandler's catch filter only matched TaskCanceledException, so the OperationCanceledException escaped, hiding the connecting exception.
4. In StartScript's catch, IsConnectionException() returned false. The client took the safe "script may have started" path and called CancelScript. CancelScript runs with CancellationToken.None and the full retry duration (1 hour in the test), against a Tentacle that is offline, so the call hangs.

This is a real client bug, not only a test bug. A real client cancelling at that moment against an offline Tentacle would keep retrying CancelScript for the whole retry duration instead of stopping promptly.

Results

RpcCallRetryHandler now catches OperationCanceledException. TaskCanceledException derives from it, so the existing behaviour is kept. The last real exception is now rethrown wherever in the retry loop the cancellation is observed.

I added RpcCallRetryHandlerFixture.ThrowsTheLastExceptionWhenCancelledWithAnOperationCanceledExceptionDuringARetry. It fails without the fix (it receives System.OperationCanceledException) and passes with it.

Before

From the failed build: cancellation was requested at 00:00:12.613, StartScript was never attempted a third time, and the client went into CancelScriptAsync retries until the test timed out:

00:00:12.6135230 [TentacleClient] Info "An error occurred communicating with Tentacle. This action will be retried after 1 seconds. Retry attempt 1. ..."   <- CancelScript
...
00:01:57.6282676 [TentacleClient] Info "... Retry attempt 16. Retries will be performed for up to 3494 seconds."
Test exceeded Timeout value of 120000ms

After

Cancellation during a StartScript that is still trying to connect now always surfaces the connecting exception. Script execution is cancelled without calling CancelScript.

How to review this PR

Quality ✔️

• Check the one-line change to the catch filter in RpcCallRetryHandler. It now also catches an OperationCanceledException that is not a TaskCanceledException. One case to think about: an action that throws such an exception without the caller cancelling, after an earlier retry failed, now surfaces the last real exception. TaskCanceledException already behaved this way.
• All 23 RpcCallRetryHandlerFixture tests pass locally. I haven't run the full client test project or the integration tests locally; CI covers those.

Pre-requisites

• I have read How we use GitHub Issues for help deciding when and where it's appropriate to make an issue.
• I have considered informing or consulting the right people, according to the ownership map.
• I have considered appropriate testing for my change.

---

Generated by Claude Code

[CLAassistant]

All committers have signed the CLA.

[akirayamamoto]

Interesting edge case, LGTM.