Skip to content

fix(go): retry transient Cloud API read failures - #535

Open
h4x3rotab wants to merge 1 commit into
mainfrom
fix/rate-limited-read-retries
Open

h4x3rotab wants to merge 1 commit into
mainfrom
fix/rate-limited-read-retries

Conversation

@h4x3rotab

@h4x3rotab h4x3rotab commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Terraform environment updates could fail immediately when their GetCVMInfo read received HTTP 429, even though the Go SDK already had retry handling for writes.

Route bodyless GET/HEAD requests through the existing doWithRetry helper. Reuse its retry budget, Retry-After, exponential backoff and context cancellation. The production fix adds 13 lines, with no new dependencies or configuration.

The virtual-clock regression covers GET/HEAD recovery after a 60-second quota wait with a 30-second HTTP timeout, operation deadline cancellation, and no new automatic POST retries. Retain the upstream-sync CI fixes: Go race testing, Ubuntu 24.04 for Playwright's WebKit ABI, and the repository's pinned Bun version in CLI CI.

The entire PR is 91 added lines and 5 deleted lines across five files, including tests, changelog and CI. JavaScript and Python are unchanged.

Validation:

  • go test -race ./... and go vet ./... pass.
  • The new read regression fails against the original request path and passes with this fix.
  • Terraform provider TestCVMEnvCreateSurvivesReadThrottling passes with race detection, including exactly one encrypted environment update.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

📋 Check Results

✨ JS SDK - Code Formatting

Show format check results
✓ No formatting issues found

🔍 JS SDK - TypeScript Type Check

Show type check output
$ tsc --noEmit

🧪 JS SDK - Test Results

Show test output
$ vitest --run --exclude '**/*.e2e.test.ts'

�[1m�[30m�[46m RUN �[49m�[39m�[22m �[36mv4.1.12 �[39m�[90m/home/runner/work/phala-cloud/phala-cloud/js�[39m

 �[32m✓�[39m src/utils/errors.test.ts �[2m(�[22m�[2m42 tests�[22m�[2m)�[22m�[32m 34�[2mms�[22m�[39m
 �[32m✓�[39m src/client.test.ts �[2m(�[22m�[2m47 tests�[22m�[2m)�[22m�[32m 21�[2mms�[22m�[39m
�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mStandard Version�[2m > �[22m�[2mshould add compose hash successfully with default parameters
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mStandard Version�[2m > �[22m�[2mshould handle custom timeout
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mStandard Version�[2m > �[22m�[2mshould use custom schema when provided
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mStandard Version�[2m > �[22m�[2mshould return raw data when schema is false
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mStandard Version�[2m > �[22m�[2mshould throw when custom schema validation fails
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mStandard Version�[2m > �[22m�[2mshould work with wallet client authentication
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mStandard Version�[2m > �[22m�[2mshould work with both clients provided
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mStandard Version�[2m > �[22m�[2mshould skip prerequisite checks when configured
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mStandard Version�[2m > �[22m�[2mshould use retry mechanism when enabled
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mStandard Version�[2m > �[22m�[2mshould handle progress callbacks
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mSafe Version�[2m > �[22m�[2mshould return success result when operation succeeds
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mSafe Version�[2m > �[22m�[2mshould work with custom schema
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mSafe Version�[2m > �[22m�[2mshould return raw data when schema is false
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mSafe Version�[2m > �[22m�[2mshould work without parameters
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mSafe Version�[2m > �[22m�[2mshould work with empty parameters object
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mSchema Flexibility�[2m > �[22m�[2mshould allow extra fields in transaction receipt for forward compatibility
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mSchema Flexibility�[2m > �[22m�[2mshould handle ComposeHashAdded event when present
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mType Inference�[2m > �[22m�[2mshould infer correct types for default schema
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mType Inference�[2m > �[22m�[2mshould infer correct types for custom schema
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mType Inference�[2m > �[22m�[2mshould infer unknown type when schema is false
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mSafe Version Type Inference�[2m > �[22m�[2mshould infer correct SafeResult types for default schema
�[22m�[39m[]

�[90mstdout�[2m | src/actions/blockchains/add_compose_hash.test.ts�[2m > �[22m�[2maddComposeHash�[2m > �[22m�[2mSafe Version Type Inference�[2m > �[22m�[2mshould infer correct SafeResult types for custom schema
�[22m�[39m[]

 �[32m✓�[39m src/actions/blockchains/add_compose_hash.test.ts �[2m(�[22m�[2m34 tests�[22m�[2m)�[22m�[32m 69�[2mms�[22m�[39m
 �[32m✓�[39m src/utils/define-action.test.ts �[2m(�[22m�[2m24 tests�[22m�[2m)�[22m�[32m 23�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/blockchains/deploy_app_auth.test.ts �[2m(�[22m�[2m27 tests�[22m�[2m)�[22m�[32m 36�[2mms�[22m�[39m
 �[32m✓�[39m src/utils/define-action.type.test.ts �[2m(�[22m�[2m20 tests�[22m�[2m)�[22m�[32m 20�[2mms�[22m�[39m
 �[32m✓�[39m src/types/cvm_id.test.ts �[2m(�[22m�[2m53 tests�[22m�[2m)�[22m�[32m 15�[2mms�[22m�[39m
 �[32m✓�[39m src/utils/get_mr_config_id.test.ts �[2m(�[22m�[2m28 tests�[22m�[2m)�[22m�[32m 25�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/provision_cvm_compose_file_update.test.ts �[2m(�[22m�[2m29 tests�[22m�[2m)�[22m�[32m 24�[2mms�[22m�[39m
 �[32m✓�[39m src/utils/hostname.test.ts �[2m(�[22m�[2m53 tests�[22m�[2m)�[22m�[32m 11�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/provision_cvm.test.ts �[2m(�[22m�[2m26 tests�[22m�[2m)�[22m�[32m 21�[2mms�[22m�[39m
 �[32m✓�[39m src/parse_dotenv.test.ts �[2m(�[22m�[2m71 tests�[22m�[2m)�[22m�[32m 10�[2mms�[22m�[39m
 �[32m✓�[39m src/version-inference.runtime.test.ts �[2m(�[22m�[2m20 tests�[22m�[2m)�[22m�[32m 45�[2mms�[22m�[39m
 �[32m✓�[39m src/redirect-security.test.ts �[2m(�[22m�[2m48 tests�[22m�[2m)�[22m�[32m 254�[2mms�[22m�[39m
 �[32m✓�[39m src/create-client.parity.test.ts �[2m(�[22m�[2m12 tests�[22m�[2m)�[22m�[32m 36�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/watch_more_contract.test.ts �[2m(�[22m�[2m20 tests�[22m�[2m)�[22m�[32m 53�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/get_current_user.test.ts �[2m(�[22m�[2m14 tests�[22m�[2m)�[22m�[32m 15�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/patch_cvm.test.ts �[2m(�[22m�[2m18 tests�[22m�[2m)�[22m�[32m 18�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/watch_cvm_state.test.ts �[2m(�[22m�[2m9 tests�[22m�[2m)�[22m�[32m 73�[2mms�[22m�[39m
 �[32m✓�[39m src/version-inference.type.test.ts �[2m(�[22m�[2m18 tests�[22m�[2m)�[22m�[32m 33�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/commit_cvm_compose_file_update.test.ts �[2m(�[22m�[2m12 tests�[22m�[2m)�[22m�[32m 13�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/watch_contract.test.ts �[2m(�[22m�[2m11 tests�[22m�[2m)�[22m�[32m 40�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/get_cvm_compose_file.test.ts �[2m(�[22m�[2m11 tests�[22m�[2m)�[22m�[32m 15�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/get_cvm_info.test.ts �[2m(�[22m�[2m11 tests�[22m�[2m)�[22m�[32m 24�[2mms�[22m�[39m
 �[32m✓�[39m src/client.extend.type.test.ts �[2m(�[22m�[2m8 tests�[22m�[2m)�[22m�[32m 5�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/placement-identifiers.test.ts �[2m(�[22m�[2m15 tests�[22m�[2m)�[22m�[32m 26�[2mms�[22m�[39m
 �[32m✓�[39m src/types/app_compose.test.ts �[2m(�[22m�[2m16 tests�[22m�[2m)�[22m�[32m 23�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/kms/get_app_env_encrypt_pubkey.test.ts �[2m(�[22m�[2m9 tests�[22m�[2m)�[22m�[32m 16�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/cvm_lifecycle_versioning.test.ts �[2m(�[22m�[2m12 tests�[22m�[2m)�[22m�[32m 18�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/get_cvm_list.test.ts �[2m(�[22m�[2m9 tests�[22m�[2m)�[22m�[32m 25�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/commit_cvm_provision.test.ts �[2m(�[22m�[2m8 tests�[22m�[2m)�[22m�[32m 19�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/get_cvm_status_batch.test.ts �[2m(�[22m�[2m7 tests�[22m�[2m)�[22m�[32m 10�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/get_available_nodes.test.ts �[2m(�[22m�[2m7 tests�[22m�[2m)�[22m�[32m 16�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/kms/get_kms_list.test.ts �[2m(�[22m�[2m8 tests�[22m�[2m)�[22m�[32m 16�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/apps/app_allowlist_versioning.test.ts �[2m(�[22m�[2m8 tests�[22m�[2m)�[22m�[32m 21�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/get_cvm_containers_stats.test.ts �[2m(�[22m�[2m7 tests�[22m�[2m)�[22m�[32m 10�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/kms/get_kms_info.test.ts �[2m(�[22m�[2m8 tests�[22m�[2m)�[22m�[32m 16�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/get_cvm_create_resources.test.ts �[2m(�[22m�[2m4 tests�[22m�[2m)�[22m�[32m 18�[2mms�[22m�[39m
 �[32m✓�[39m src/utils/verify-webhook.test.ts �[2m(�[22m�[2m11 tests�[22m�[2m)�[22m�[32m 4�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/get_cvm_stats.test.ts �[2m(�[22m�[2m8 tests�[22m�[2m)�[22m�[32m 13�[2mms�[22m�[39m
 �[32m✓�[39m src/client.extend.test.ts �[2m(�[22m�[2m6 tests�[22m�[2m)�[22m�[32m 10�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/confirm_cvm_patch.test.ts �[2m(�[22m�[2m9 tests�[22m�[2m)�[22m�[32m 9�[2mms�[22m�[39m
 �[32m✓�[39m src/utils/validate-parameters.test.ts �[2m(�[22m�[2m9 tests�[22m�[2m)�[22m�[32m 6�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/ssh_keys/create_ssh_key.test.ts �[2m(�[22m�[2m9 tests�[22m�[2m)�[22m�[32m 13�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/update_cvm_envs.test.ts �[2m(�[22m�[2m6 tests�[22m�[2m)�[22m�[32m 9�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/shutdown_cvm.test.ts �[2m(�[22m�[2m8 tests�[22m�[2m)�[22m�[32m 12�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/restart_cvm.test.ts �[2m(�[22m�[2m8 tests�[22m�[2m)�[22m�[32m 14�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/start_cvm.test.ts �[2m(�[22m�[2m8 tests�[22m�[2m)�[22m�[32m 13�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/stop_cvm.test.ts �[2m(�[22m�[2m8 tests�[22m�[2m)�[22m�[32m 11�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/refresh_cvm_instance_ids.test.ts �[2m(�[22m�[2m4 tests�[22m�[2m)�[22m�[32m 15�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/refresh_cvm_instance_id.test.ts �[2m(�[22m�[2m5 tests�[22m�[2m)�[22m�[32m 11�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/upgrade_prelaunch_script.test.ts �[2m(�[22m�[2m4 tests�[22m�[2m)�[22m�[32m 13�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/ssh_keys/update_cvm_ssh_keys.test.ts �[2m(�[22m�[2m5 tests�[22m�[2m)�[22m�[32m 10�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/ssh_keys/delete_ssh_key.test.ts �[2m(�[22m�[2m8 tests�[22m�[2m)�[22m�[32m 8�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/ssh_keys/sync_github_ssh_keys.test.ts �[2m(�[22m�[2m6 tests�[22m�[2m)�[22m�[32m 9�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/delete_cvm.test.ts �[2m(�[22m�[2m8 tests�[22m�[2m)�[22m�[32m 10�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/get_prelaunch_script_upgrade_status.test.ts �[2m(�[22m�[2m3 tests�[22m�[2m)�[22m�[32m 7�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/apps/get_app_attestation.test.ts �[2m(�[22m�[2m3 tests�[22m�[2m)�[22m�[32m 7�[2mms�[22m�[39m
 �[32m✓�[39m src/utils/as-hex.test.ts �[2m(�[22m�[2m9 tests�[22m�[2m)�[22m�[32m 5�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/get_cvm_state.test.ts �[2m(�[22m�[2m4 tests�[22m�[2m)�[22m�[32m 8�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/update_cvm_visibility.test.ts �[2m(�[22m�[2m2 tests�[22m�[2m)�[22m�[32m 8�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/ssh_keys/get_cvm_ssh_keys.test.ts �[2m(�[22m�[2m3 tests�[22m�[2m)�[22m�[32m 12�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/kms/list_kms_contracts.test.ts �[2m(�[22m�[2m2 tests�[22m�[2m)�[22m�[32m 8�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/apps/create_app_instance.test.ts �[2m(�[22m�[2m2 tests�[22m�[2m)�[22m�[32m 8�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/workspaces/list_workspaces.test.ts �[2m(�[22m�[2m4 tests�[22m�[2m)�[22m�[32m 7�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/kms/list_kms_contract_nodes.test.ts �[2m(�[22m�[2m2 tests�[22m�[2m)�[22m�[32m 9�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/api_tokens/revoke_current_api_token.test.ts �[2m(�[22m�[2m3 tests�[22m�[2m)�[22m�[32m 6�[2mms�[22m�[39m
 �[32m✓�[39m src/utils/get_compose_hash.test.ts �[2m(�[22m�[2m18 tests�[22m�[2m)�[22m�[32m 7�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/replicate_cvm.test.ts �[2m(�[22m�[2m2 tests�[22m�[2m)�[22m�[32m 9�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/apps/app_launch_token.test.ts �[2m(�[22m�[2m3 tests�[22m�[2m)�[22m�[32m 8�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/kms/get_kms_contract.test.ts �[2m(�[22m�[2m2 tests�[22m�[2m)�[22m�[32m 9�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/get_available_os_images.test.ts �[2m(�[22m�[2m1 test�[22m�[2m)�[22m�[32m 5�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/cvm_annotations.test.ts �[2m(�[22m�[2m2 tests�[22m�[2m)�[22m�[32m 8�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/update_cvm_resources.test.ts �[2m(�[22m�[2m1 test�[22m�[2m)�[22m�[32m 5�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/get_cvm_init_scripts.test.ts �[2m(�[22m�[2m1 test�[22m�[2m)�[22m�[32m 4�[2mms�[22m�[39m
 �[32m✓�[39m src/actions/cvms/get_cvm_user_config.test.ts �[2m(�[22m�[2m1 test�[22m�[2m)�[22m�[32m 4�[2mms�[22m�[39m

�[2m Test Files �[22m �[1m�[32m76 passed�[39m�[22m�[90m (76)�[39m
�[2m      Tests �[22m �[1m�[32m982 passed�[39m�[22m�[90m (982)�[39m
�[2m   Start at �[22m 09:06:16
�[2m   Duration �[22m 8.37s�[2m (transform 1.50s, setup 0ms, import 14.67s, tests 1.50s, environment 6ms)�[22m

📝 JS SDK - Lint Check

Show lint results
✓ No linting issues found

🌐 JS SDK - Browser Compatibility

Show browser test results

🌐 Browser Compatibility Report

Browser compatibility tests completed across:

  • ✓ Chromium
  • ✓ Firefox
  • ✓ WebKit (Safari)

The SDK has been verified to work in modern browser environments.


Check run: https://github.com/Phala-Network/phala-cloud/actions/runs/42590767483

@h4x3rotab h4x3rotab changed the title fix(go): retry throttled reads within the operation deadline fix(sdk): align read retries across Go, JavaScript and Python Oct 26, 2026
@h4x3rotab h4x3rotab changed the title fix(sdk): align read retries across Go, JavaScript and Python fix(go): retry transient Cloud API read failures Nov 9, 2026
Route bodyless GET/HEAD requests through the existing retry helper. Cover quota waits longer than the HTTP timeout, deadline cancellation, and unchanged POST retry behavior. Keep CI compatible with the current runner images and pinned Bun version.
@h4x3rotab
h4x3rotab force-pushed the fix/rate-limited-read-retries branch from 4990544 to b3a1f2f Compare November 9, 2026 09:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant