Repository navigation
Conversation
…titions_with_specs
get_partitions_with_specs skipped the hive.metastore.limit.partition.request
guard that every regular partition listing RPC enforces (get_partitions,
get_partitions_by_names, get_partitions_with_auth, get_partitions_by_filter,
get_partitions_by_expr, get_partitions_ps, get_partitions_ps_with_auth).
Enforce it:
- Fail fast before the partition fetch using the same cheap count queries
the regular listing RPCs use, mapped per filter mode:
- filter spec unset -> getNumPartitionsByFilter("")
- BY_NAMES -> the requested partition names count (as get_partitions_by_names does)
- BY_VALUES -> getNumPartitionsByPs (same makePartNameMatcher predicate as the fetch)
- BY_EXPR -> getNumPartitionsByFilter on the joined expressions (same grammar)
- Keep a check on the fetched result size as the final backstop (covers the
count/fetch TOCTOU window and unknown filter modes)
- Add get_partitions_with_specs to the config description
Default (-1) behavior is unchanged; the guard only applies when the operator
configures the limit.
Related: HIVE-30195 (table-level filter hook check gap in the same RPC)
Co-authored-by: Cursor <cursoragent@cursor.com>
Member
Author
|
cc @dengzhhu653 |
Co-authored-by: Cursor <cursoragent@cursor.com>
|
dengzhhu653
reviewed
Oct 19, 2026
| List<Partition> partitions = getMS() | ||
| .getPartitionSpecsByFilterAndProjection(table, request.getProjectionSpec(), | ||
| request.getFilterSpec()); | ||
| GetPartitionsHandler.checkLimitNumberOfPartitions(getConf(), tableName, partitions.size()); |
Member
There was a problem hiding this comment.
we don't need to check again at here
Member
There was a problem hiding this comment.
Can you move this implementation to GetPartitionsHandler if it's ok? so it shares the same pre check and post check and the fetch limit validation.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
get_partitions_with_specs(introduced by HIVE-20306) skipped thehive.metastore.limit.partition.requestguard that every regular partition listing RPC enforces — the config's own description enumerates the 8 covered RPCs and this one was absent. The thrift request has no client-side max parameter, so a single call made the metastore materialize all matching partitions server-side before responding.HiveServer2 paths (
PartitionIterablebatched iteration, SHOW TABLE EXTENDED, EXCHANGE PARTITION, repl bootstrap, view rewrite analysis) and direct HMS clients (e.g. Spark) use this API, so an operator-configured guard was silently bypassed on it.What this change does
HMSHandler.get_partitions_with_specs, before the expensive fetch, mirroring what the regular listing RPCs already do (checkLimitNumberOfPartitionsByFilter/checkLimitNumberOfPartitionsByPs), mapped per filter mode:rs.getNumPartitionsByFilter(cat, db, tbl, "")BY_NAMES→ the requested partition names count (same asget_partitions_by_names)BY_VALUES→rs.getNumPartitionsByPs(identical predicate: sameMetaStoreUtils.makePartNameMatcheron both the SQL and JDO paths)BY_EXPR→rs.getNumPartitionsByFilteron the joined expressions (samePartFilterExprUtilgrammar the projection path parses)add_partitions) and unknown filter modes.get_partitions_with_specsto theLIMIT_PARTITION_REQUESTconfig description.firePreEvent(pre-event listeners) still runs before the limit checks.exceedsPartitionFetchLimit/checkLimitNumberOfPartitionswere madepublic static(taking an explicitConfiguration) so the handler flow andget_partitions_with_specsshare one implementation. The check stays at the RPC layer on purpose: the same store method is also used internally by the column-stats path, so the guard must not become a store-level rule.Behavior changes (only when the limit is configured; default
-1= disabled is unchanged)MetaException(same message / config name) as the regular listing RPCs.BY_NAMESnow uses the strict requested-count semantics, mirroringget_partitions_by_names: requesting more names than the limit fails even if fewer of them exist.BY_VALUES/BY_EXPRrequests pay one extra count query per request;BY_NAMESis free.PartitionSpecresponses count as partition objects — response size and DB work scale with the number of matched partitions regardless of projection.Tests
TestGetPartitionsUsingProjectionAndFilterSpecs(server started withLIMIT_PARTITION_REQUEST=100), new tests against a 101-partition table:...ExceedsPartitionLimit...AtPartitionLimit...ByNamesExceeds......ByNamesRequestedCountExceeds......ByValuesExceeds......ByExprExceeds......ExceedsPartitionLimitWithJDOAll 32 tests pass, plus
TestRemoteHiveMetaStore#testGetPartitionsWithSpec,TestEmbeddedHiveMetaStore#testGetPartitionsWithSpec, andTestHmsServerAuthorization(authorization failure still precedes the limit check).