Repository navigation
feat!: 🎸 move middleware queries to the redesigned SubQuery schema - #1694
prashantasdeveloper wants to merge 17 commits into
Conversation
Types come from the indexer image 966b4a8c (PolymeshAssociation/polymesh-subquery, docs/architecture). The processing script no longer adds CreatedAt members to an OrderBy enum whose entity already has a createdAt column, which produced duplicate members for IndexerAnomaly and IndexOrigin. The SDK does not compile against these types yet; the following commits move each query and its mapping code over. BREAKING CHANGE: 🧨 the SDK requires a middleware running the redesigned SubQuery schema. Re-exported middleware enums and OrderBy enums follow the new schema (for example CREATED_BLOCK_ID_* becomes CREATED_EVENT_ID_*).
Entities now reference the event that created or last updated them (createdEvent, updatedEvent) instead of carrying a block and an event index. middlewareEventToEventIdentifier takes that event, and eventIdentifierFields is the selection every query uses for it. The middleware never indexes NoData or investor uniqueness claims, so the deprecated claim filter only has NoType left to drop.
…w schema Asset, claim, trusted issuer, custom claim type, distribution payment, multisig, portfolio, authorization and investment queries select createdEvent and updatedEvent, order by CREATED_EVENT_ID_*, and filter on the *Id columns that replaced string relations. External agent queries read assetAgents, assetAgentHistories and assetAgentActions. Authorizations sent to an Account are filtered and mapped by its SS58 address (toAccountId) instead of its public key. An investment is filtered by its offering's id, <offering asset id>/<offering number>. AssetPermissions.enabledAt now looks up this Identity's agent row. It used to return when the Asset's most recent agent was added, whichever Identity that was. BREAKING CHANGE: 🧨 AssetPermissions.getOperationHistory takes AssetAgentActionsOrderBy instead of TickerExternalAgentActionsOrderBy, and OrderBy values follow the new schema (CREATED_BLOCK_ID_* is now CREATED_EVENT_ID_*).
Legs select their holders as ids (fromId, fromPortfolioId, fromAccountId and the to* columns), and a portfolio holder is read from its <did>/<number> id. Settlement filters match portfolio ids and identity ids, historical instructions filter parties and legs by identity id, and a leg's addresses are matched with contains. Instruction events, instructions and affirmations select createdEvent and updatedEvent. Portfolio movements are read from assetTransactions: internal transfers that no instruction made. NFT movements are listed as NFT legs. Affirmation rows exist for every holder from the instruction's creation, as Pending or Affirmed. getAffirmations lists the holders, without mediators, and says which portfolio each Identity affirms for, as the chain path does. Adds changelogs/v32.0.0-next.md. BREAKING CHANGE: 🧨 Instruction.getAffirmations lists every portfolio and Account holder, including those still Pending, once per portfolio, and never the mediators; a rejection is no longer reported as an affirmation status.
polyxTransactions is gone. polyxEntries has one row per side of a movement, so an Account's or an Identity's history filters on accountId or identityId alone, with no from/to OR. Each HistoricPolyxTransaction is the change one movement made to one Account: account and identity, the counterparty if there is one, a positive amount with its direction (Credit or Debit), the movement kind and the balance pool. src/types re-exports PolyxPool, MovementKind, EntryDirection, PolyxEntriesOrderBy and NftsOrderBy, and no longer BalanceTypeEnum or NftHoldersOrderBy. BREAKING CHANGE: 🧨 HistoricPolyxTransaction replaces fromAccount/fromIdentity and toAccount/toIdentity with account/identity and counterpartyAccount/counterpartyIdentity, and type: BalanceTypeEnum with pool: PolyxPool, kind: MovementKind and direction: EntryDirection. getPolyxTransactions takes PolyxEntriesOrderBy. BalanceTypeEnum and NftHoldersOrderBy are no longer exported.
Extrinsics no longer have paramsTxt. args holds the call's arguments as an object keyed by argument name, with integers as decimal strings and Accounts as SS58 addresses, or null when the indexer could not decode them. BREAKING CHANGE: 🧨 ExtrinsicData.params is the call's arguments keyed by name (Record<string, unknown> | null) instead of an array.
nftHolders is gone. getHeldNfts reads a page of the Identity's assetHolders rows for NFT collections, then every NFT it holds in them from nfts. AssetHolders.get reads a page of the collection's holders, then every NFT they hold. Both page through nfts a hundred at a time, since the middleware returns no more than 100 rows to a request without saying so; a nested holdings connection would stop there. BREAKING CHANGE: 🧨 Identity.getHeldNfts takes AssetHoldersOrderBy instead of NftHoldersOrderBy for orderBy and the deprecated order.
The middleware no longer has eventArg0..eventArg3. getEventByIndexedArgs and getEventsByIndexedArgs take refs, what the event involves in any argument (identity:, account:, asset:, ticker: or portfolio: followed by its id), and args, argument values keyed by field name, both matched with contains. The event type is still required, which also narrows the unindexed args filter. BREAKING CHANGE: 🧨 getEventByIndexedArgs and getEventsByIndexedArgs take refs and args instead of eventArg0, eventArg1 and eventArg2.
… form In the 7.0 form an extrinsic permission maps each pallet name to its extrinsics as these, except or whole. Only pallets given as these were read, so a pallet given whole, or with exceptions, was dropped. The redesigned middleware writes every row in this form, including permissions set before 7.0, which used to be read through the legacy form.
The middleware stores dates in UTC and serialises them with no zone marker, so new Date() read them as local time and shifted them by the reader's offset. middlewareDateToDate reads them as UTC. It is used for event and block dates, instruction trade and value dates, authorization expiries, mediator affirmation expiries and distribution payment dates. Extrinsic block dates already appended the marker themselves.
Mocks select createdEvent and updatedEvent instead of a block and an event index, legs and affirmations carry id columns, portfolio movements come from assetTransactions, POLYX history from polyxEntries, and dates are strings as the middleware sends them. New tests cover per-holder affirmations, NFT holders paged past the middleware's 100 row limit, event refs and args, and the POLYX entry mapping.
The redesigned middleware stores an instruction's and a venue's chain sequence number zero padded to 10 digits, so ordering by id is numeric, and a filter has to use the padded form. Instruction and Venue filtered by the bare number and matched nothing: Instruction.details reported the instruction as not yet processed, and its affirmations, legs, events and a venue's instructions came back empty. bigNumberToMiddlewareSequenceId pads them.
Covers an Account holder on a leg, an NFT movement between portfolios and one between Accounts, a POLYX entry with no module, an empty page of NFT holders, a holder with no NFTs returned, and the NFT paging helper reading past one page and stopping at an empty one.
Types come from the final schema (polymesh-subquery docs/architecture 9d182a69, release/indexer-redesign 4a2b79d3). It drops StakingPosition, Validator, AssetPreApproval, EvmAccountMapping and CorporateActionDefaultConfig, some CheckpointSchedule and Proposal fields, and the one-to-one reverse fields on Account and CorporateAction. No SDK query reads any of them.
params.proposals is nullable in the middleware: bridge proposals have none, and from polymesh-subquery e9662b7c neither does a proposal whose extrinsic has no single proposing call for that MultiSig. getHistoricalProposals read it unconditionally, so one such row threw and lost the whole page. txTag and args are now null for that row. Also documents that an EventIdentifier for a seeded row has eventIndex -1, which is how the middleware now marks its synthetic seed event. BREAKING CHANGE: 🧨 HistoricalMultiSigProposal.txTag and args can be null.
|
| } | ||
| return { | ||
| party: new Identity({ did: identity as string }, context), | ||
| party: new Identity({ did: identityId }, context), | ||
| // eslint-disable-next-line @typescript-eslint/no-non-null-assertion | ||
| portfolio: middlewarePortfolioToPortfolio( | ||
| portfolioIdStringToPortfolio(portfolioId!), | ||
| context | ||
| ), | ||
| status: middlewareAffirmStatusToAffirmationStatus(status), | ||
| }; |
There was a problem hiding this comment.
Bug: getAffirmations() throws for an instruction with a claimed off-chain receipt.
When a receipt is claimed, subquery v20.0.0-alpha.1 (mapSettlement.ts, the ReceiptClaimed handler) creates an InstructionAffirmation with isMediator: false and offChainReceiptId set, but no accountId and no portfolioId. That row passes the isMediator: false filter, skips the accountId branch and reaches portfolioIdStringToPortfolio(undefined), which throws TypeError: Cannot read properties of undefined (reading 'split').
The return type already makes portfolio optional, so the smallest fix keeps the row and returns it with the Identity as the party (on develop these rows already come back with only an Identity as the party):
| } | |
| return { | |
| party: new Identity({ did: identity as string }, context), | |
| party: new Identity({ did: identityId }, context), | |
| // eslint-disable-next-line @typescript-eslint/no-non-null-assertion | |
| portfolio: middlewarePortfolioToPortfolio( | |
| portfolioIdStringToPortfolio(portfolioId!), | |
| context | |
| ), | |
| status: middlewareAffirmStatusToAffirmationStatus(status), | |
| }; | |
| } | |
| if (!portfolioId) { | |
| // a claimed off-chain receipt is affirmed by an Identity, with no Account or portfolio | |
| return { | |
| party: new Identity({ did: identityId }, context), | |
| status: middlewareAffirmStatusToAffirmationStatus(status), | |
| }; | |
| } | |
| return { | |
| party: new Identity({ did: identityId }, context), | |
| portfolio: middlewarePortfolioToPortfolio( | |
| portfolioIdStringToPortfolio(portfolioId), | |
| context | |
| ), | |
| status: middlewareAffirmStatusToAffirmationStatus(status), | |
| }; |
If receipts should be left out instead (they are already reported by getOffChainAffirmations), filter them in the query with offChainReceiptExists: false. createArgsAndFilters can't express that bare boolean, so it would have to be a fixed clause in instructionAffirmationsQuery, and totalCount then stays consistent. Either way, please add a test with a receipt row: the mocks only have rows with an account or a portfolio.
| @@ -59,13 +56,13 @@ export function assetHoldersQuery( | |||
| start?: BigNumber, | |||
| orderBy?: AssetHoldersOrderBy | AssetHoldersOrderBy[] | |||
| ): QueryOptions<PaginatedQueryArgs<QueryArgs<AssetHolder, 'identityId'> & { amount?: string }>> { | |||
There was a problem hiding this comment.
Bug: NFT collections come back as fungible Assets.
In the redesigned schema AssetHolder is one row per (asset, identity) for every asset. Subquery's getAssetHolder creates it with amount: 0, nftCount: 0, and the NFT path updates nftCount on the same row. The old assetHolders table was fungible only (NFTs lived in nftHolders), so this query never needed a filter. Now it does:
Identity.getHeldAssets()wraps each NFT collection asnew FungibleAsset(...).getFungibleAssetHoldings()(withoutheldNow) lists each collection with an amount of 0.- Both
totalCountvalues, and the paging, include these rows.
nftHoldersQuery already filters asset: { isNftCollection: { equalTo: true } }. This query needs the mirror image. Because createArgsAndFilters only emits field: { op: $var }, the simplest approach is to write the filter out the way nftHoldersQuery does:
const hasAmount = filters.amount !== undefined;
const amountArg = hasAmount ? ', $amount: BigFloat!' : '';
const amountFilter = hasAmount ? ', amount: { greaterThan: $amount }' : '';
const query = gql`
query AssetHoldersQuery($start: Int, $size: Int, $identityId: String!${amountArg}) {
assetHolders(
filter: {
identityId: { equalTo: $identityId }
asset: { isNftCollection: { equalTo: false } }${amountFilter}
}
...Please add a test asserting the filter, since the current mocks can't catch this.
| nfts: { nodes, totalCount: total }, | ||
| }, | ||
| } = await context.queryMiddleware<Ensured<Query, 'nfts'>>( | ||
| buildQuery(size, new BigNumber(nfts.length)) |
There was a problem hiding this comment.
Offset paging can skip or duplicate NFTs.
The next offset is nfts.length, and nothing pins the set between pages. If an NFT earlier in the order is burned or moved away mid-read, every later row shifts down one, so the next page skips an NFT. If one is minted or moved in, an NFT is returned twice. totalCount is re-read on every page, so the loop doesn't notice.
Keyset paging on Nft.id (unique, and NftsOrderBy.IdAsc exists) avoids both, and lets the loop end on a short page instead of trusting totalCount:
export async function getAllMiddlewareNfts(
context: Context,
buildQuery: (size: BigNumber, afterId: string | undefined) => QueryOptions
): Promise<MiddlewareNft[]> {
const size = MAX_GQL_PAGE_SIZE;
const nfts: MiddlewareNft[] = [];
let afterId: string | undefined;
let page: MiddlewareNft[];
do {
({
data: {
nfts: { nodes: page },
},
} = await context.queryMiddleware<Ensured<Query, 'nfts'>>(
buildQuery(new BigNumber(size), afterId)
));
nfts.push(...page);
afterId = page[page.length - 1]?.id;
} while (page.length === size);
return nfts;
}The two builders (heldNftsQuery and nftCollectionHoldersNftsQuery) would then take $afterId: String and use orderBy: [ID_ASC] with id: { greaterThan: $afterId }, adding that clause only when it is set. Grouping by assetId and holder is unaffected by the order.
| * | ||
| * @param buildQuery - the query for one page, given its size and offset | ||
| */ | ||
| export async function getAllMiddlewareNfts( |
There was a problem hiding this comment.
Unbounded sequential reads.
getHeldNfts and NftCollection's AssetHolders.get return one page of collections or holders, but then read every NFT of that page, 100 per request, one request after another. A page of 25 holders who together own 50k NFTs takes about 500 sequential requests in one SDK call, and the caller can't limit it.
Options, roughly in order of preference:
- Return only
nftCount(already selected) for each collection or holder, and let the caller page the NFTs with a separate paged method. This is the only option that keeps a "page" a bounded amount of work. It changes the return type, but this PR is already breaking. - Keep the full read but document the cost on both public methods (
@notereads every NFT the page holds, which can be many requests). - Parallelise from the first page's
totalCount, capped atMAX_CONCURRENT_REQUESTS. Note that this conflicts with the keyset fix on L1729: parallel offset pages bring the skip/duplicate problem back. If you go this way, fix: 🐛 Account NFT reads by collection, complete exemption reads, and modifying a Ballot #1688 addsallInOrder, which keeps error order deterministic.
| instructionAffirmationsQuery( | ||
| { | ||
| instructionId: id.toString(), | ||
| instructionId: bigNumberToMiddlewareSequenceId(id), |
There was a problem hiding this comment.
Nit: move the id padding into the query builders.
Seven call sites in this file (plus the Venue ones) have to remember bigNumberToMiddlewareSequenceId(id). The next caller that passes id.toString() silently matches nothing, because the middleware stores 0000000012, not 12, and no test would fail unless it asserted the variables.
Suggest that instructionsQuery, legsQuery, instructionAffirmationsQuery, instructionEventsQuery (and the venue queries) take instructionId: BigNumber / venueId: BigNumber and pad internally, the way subquery's padNumericId does at each lookup. The type change then makes a forgotten conversion a compile error.
| * Get a single event by any of its indexed arguments | ||
| * Get events of one type by what they involve and by their arguments | ||
| */ | ||
| export function eventsByArgs( |
There was a problem hiding this comment.
Nit: this query's selection (eventIdx block { blockId hash datetime }, L54–59) is written out by hand. Every other query this PR touches uses the new shared eventIdentifierFields, which exists so that middlewareEventToEventIdentifier always gets what it reads. Please use it here too:
nodes {
${eventIdentifierFields}
}| variables.to = identityId; | ||
| } else { | ||
| // a portfolio's id is `<did>/<number>`, the default portfolio being number 0 | ||
| const portfolio = `${identityId}/${portfolioId?.toString() ?? '0'}`; |
There was a problem hiding this comment.
Nit: the middleware portfolio id <did>/<number> (default portfolio as 0) is built inline here, in portfolios.ts:62 and in conversion.ts:5455, while the reverse direction already has portfolioIdStringToPortfolio. A portfolioToMiddlewarePortfolioId(identityId, portfolioId?) beside it in conversion.ts would keep the default-portfolio rule and the format in one place. If the indexer ever pads portfolio numbers, as it now does instruction ids, there is then only one function to change.
| const isLocalDateTime = value.includes('T') && !/(Z|[+-]\d{2}:?\d{2})$/.test(value); | ||
|
|
||
| return new Date(isLocalDateTime ? `${value}Z` : value); |
There was a problem hiding this comment.
Nit / hardening: only T-separated values count as zoneless date-times. A Postgres-style 2024-01-01 10:00:00 (which a proxy, or a change in how PostGraphile serialises dates, could produce) skips the Z and is read as local time, which is the bug this function exists to prevent. Also, new Date('2024-01-01 10:00:00Z') isn't guaranteed by the spec, so normalise the separator too:
| const isLocalDateTime = value.includes('T') && !/(Z|[+-]\d{2}:?\d{2})$/.test(value); | |
| return new Date(isLocalDateTime ? `${value}Z` : value); | |
| const isLocalDateTime = /\d[T ]\d{2}:\d{2}/.test(value) && !/(Z|[+-]\d{2}:?\d{2})$/.test(value); | |
| return new Date(isLocalDateTime ? `${value.replace(' ', 'T')}Z` : value); |




Summary
Moves every middleware (SubQuery) query to the redesigned SubQuery schema (
PolymeshAssociation/polymesh-subquery,docs/architecture,966b4a8c). After this, the SDK requires a middleware running that schema. This is a major-version (v32) change, delivered as abc-branch perNEXT_RELEASE.md. The migration guide is inchangelogs/v32.0.0-next.md.What changed
src/middleware/types.tsis regenerated.scripts/processMiddlewareTypes.tsno longer adds duplicateCreatedAt*members to an OrderBy enum that already has them.createdEvent/updatedEvent).middlewareEventToEventIdentifiertakes that event, andeventIdentifierFieldsis the shared selection for it.fromId,fromPortfolioId,toAccountId, …). Portfolio ids aredid/number.assetAgents,assetAgentHistoriesandassetAgentActions.Pending/Affirmed).assetTransactions: internal transfers that no instruction made.polyxEntries, one row per side of a movement.args, an object keyed by argument name.assetHolders, thennftsfor that page, 100 at a time. The middleware returns at most 100 rows per request without saying so, so a nested connection would truncate silently.refs(identity, account, asset, ticker or portfolio) and byargscontainment.Breaking changes
CREATED_BLOCK_ID_*andEVENT_IDX_*becomeCREATED_EVENT_ID_*.TickerExternalAgentActionsOrderBybecomesAssetAgentActionsOrderBy, andNftHoldersOrderBybecomesAssetHoldersOrderBy.PolyxTransactionsOrderBybecomesPolyxEntriesOrderBy.Instruction.getAffirmations()lists every portfolio and Account holder, includingPendingones, and gains an optionalportfolio. It no longer lists mediators.HistoricPolyxTransactionchanges shape:account/identityandcounterpartyAccount/counterpartyIdentity, pluskind,directionandpool. Itsamountis always positive.BalanceTypeEnumis no longer exported.ExtrinsicData.paramsisRecord<string, unknown> | nullinstead of an array.getEventByIndexedArgs/getEventsByIndexedArgstakerefsandargsinstead ofeventArg0..2.Fixes
assetPermissions.enabledAtreturns this Identity's agent event instead of that of the Asset's most recent agent.Testing
yarn build:ts,yarn lint: pass.966b4a8c: 48 of 48 valid on both chains, returning data.Open items
claimsGroupingtakes 4–10 s on both chains. Its query is unchanged and matches the fixture.19.6.0, soMINIMUM_SQ_VERSIONcan't tell it apart from the old one.getTransactionByHashfails on the public testnet because the chain's ownTransactionPaymentApi.queryInfopanics for that extrinsic.PolymeshAssociation/polymesh-subquery, branchfix/sdk-nft-collection-holders-fixture.