Repository navigation
Conversation
Add URL query parameters that set bucket-wide defaults for the parallel-download options, so applications that open buckets by URL can enable and tune parallel downloads without changing Download call sites: - allow_parallel_download (bool) - download_part_size (bytes) - download_concurrency (int) The part size and concurrency are driver-local: each driver parses the params into its Options and uses them in DownloadAt when the per-call ReaderOptions leaves them zero. allow_parallel_download needs the portable layer, since the opt-in gate lives in Bucket.Download, so add an optional driver.ParallelDownloadDefaulter interface that Download consults; the per-call ReaderOptions.AllowParallelDownload is ORed with the driver default (the URL default can enable parallel download but a per-call false cannot turn off a default-enabled bucket). Implemented for s3blob, gcsblob, and azureblob, with URLOpener tests and a portable-layer test for the defaulter. Implements google#3818
Contributor
Author
|
Closing in favor of the programmatic |
Contributor
Author
|
Superseded by #3820 ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #3818.
What
Adds URL query parameters to set bucket-wide defaults for the parallel-download options, so applications that open buckets by URL can enable and tune parallel downloads without touching
Downloadcall sites:allow_parallel_download(bool)download_part_size(bytes)download_concurrency(int)e.g.
s3://bucket?region=us-west-2&allow_parallel_download=true&download_part_size=16777216&download_concurrency=8Implemented for s3blob, gcsblob, and azureblob.
How
Two tiers, as discussed in #3818:
download_part_size/download_concurrency— driver-local. Each driver parses the params into itsOptions, stores them, and uses them inDownloadAtwhen the per-callReaderOptionsleaves them zero. No portable-layer change.allow_parallel_download— portable layer. The opt-in gate is inBucket.Download, which doesn't consult the driver, so this adds an optionaldriver.ParallelDownloadDefaulterinterface (DefaultAllowParallelDownload() bool) thatDownloadqueries.Override semantics (please confirm)
Per-call
ReaderOptionswins for the ints (non-zero overrides the URL default). For the bool, the per-call value is ORed with the driver default, so the URL default can only enable parallel download; a per-callAllowParallelDownload: falsecannot turn off a default-enabled bucket. This sidesteps the bool unset-vs-false ambiguity. The alternative (apply the default only whenopts == nil) is easy to switch to if preferred — flagging it since #3818 left it open.Tests
TestDownloaderAtDefault) covers the driver default enabling parallel download with no per-call opt-in, and the no-default fallback.go vet/gofmtclean; fullblob, s3blob, gcsblob, and azureblob test suites pass (including the recorded conformance tests).Docs
Each driver's package doc "URLs" section lists the three new params.