Repository navigation
feat!: 🎸 move middleware queries to the redesigned SubQuery schema - #1694
prashantasdeveloper wants to merge 27 commits into
Conversation
Types come from the indexer image 966b4a8c (PolymeshAssociation/polymesh-subquery, docs/architecture). The processing script no longer adds CreatedAt members to an OrderBy enum whose entity already has a createdAt column, which produced duplicate members for IndexerAnomaly and IndexOrigin. The SDK does not compile against these types yet; the following commits move each query and its mapping code over. BREAKING CHANGE: 🧨 the SDK requires a middleware running the redesigned SubQuery schema. Re-exported middleware enums and OrderBy enums follow the new schema (for example CREATED_BLOCK_ID_* becomes CREATED_EVENT_ID_*).
Entities now reference the event that created or last updated them (createdEvent, updatedEvent) instead of carrying a block and an event index. middlewareEventToEventIdentifier takes that event, and eventIdentifierFields is the selection every query uses for it. The middleware never indexes NoData or investor uniqueness claims, so the deprecated claim filter only has NoType left to drop.
…w schema Asset, claim, trusted issuer, custom claim type, distribution payment, multisig, portfolio, authorization and investment queries select createdEvent and updatedEvent, order by CREATED_EVENT_ID_*, and filter on the *Id columns that replaced string relations. External agent queries read assetAgents, assetAgentHistories and assetAgentActions. Authorizations sent to an Account are filtered and mapped by its SS58 address (toAccountId) instead of its public key. An investment is filtered by its offering's id, <offering asset id>/<offering number>. AssetPermissions.enabledAt now looks up this Identity's agent row. It used to return when the Asset's most recent agent was added, whichever Identity that was. BREAKING CHANGE: 🧨 AssetPermissions.getOperationHistory takes AssetAgentActionsOrderBy instead of TickerExternalAgentActionsOrderBy, and OrderBy values follow the new schema (CREATED_BLOCK_ID_* is now CREATED_EVENT_ID_*).
Legs select their holders as ids (fromId, fromPortfolioId, fromAccountId and the to* columns), and a portfolio holder is read from its <did>/<number> id. Settlement filters match portfolio ids and identity ids, historical instructions filter parties and legs by identity id, and a leg's addresses are matched with contains. Instruction events, instructions and affirmations select createdEvent and updatedEvent. Portfolio movements are read from assetTransactions: internal transfers that no instruction made. NFT movements are listed as NFT legs. Affirmation rows exist for every holder from the instruction's creation, as Pending or Affirmed. getAffirmations lists the holders, without mediators, and says which portfolio each Identity affirms for, as the chain path does. Adds changelogs/v32.0.0-next.md. BREAKING CHANGE: 🧨 Instruction.getAffirmations lists every portfolio and Account holder, including those still Pending, once per portfolio, and never the mediators; a rejection is no longer reported as an affirmation status.
polyxTransactions is gone. polyxEntries has one row per side of a movement, so an Account's or an Identity's history filters on accountId or identityId alone, with no from/to OR. Each HistoricPolyxTransaction is the change one movement made to one Account: account and identity, the counterparty if there is one, a positive amount with its direction (Credit or Debit), the movement kind and the balance pool. src/types re-exports PolyxPool, MovementKind, EntryDirection, PolyxEntriesOrderBy and NftsOrderBy, and no longer BalanceTypeEnum or NftHoldersOrderBy. BREAKING CHANGE: 🧨 HistoricPolyxTransaction replaces fromAccount/fromIdentity and toAccount/toIdentity with account/identity and counterpartyAccount/counterpartyIdentity, and type: BalanceTypeEnum with pool: PolyxPool, kind: MovementKind and direction: EntryDirection. getPolyxTransactions takes PolyxEntriesOrderBy. BalanceTypeEnum and NftHoldersOrderBy are no longer exported.
Extrinsics no longer have paramsTxt. args holds the call's arguments as an object keyed by argument name, with integers as decimal strings and Accounts as SS58 addresses, or null when the indexer could not decode them. BREAKING CHANGE: 🧨 ExtrinsicData.params is the call's arguments keyed by name (Record<string, unknown> | null) instead of an array.
nftHolders is gone. getHeldNfts reads a page of the Identity's assetHolders rows for NFT collections, then every NFT it holds in them from nfts. AssetHolders.get reads a page of the collection's holders, then every NFT they hold. Both page through nfts a hundred at a time, since the middleware returns no more than 100 rows to a request without saying so; a nested holdings connection would stop there. BREAKING CHANGE: 🧨 Identity.getHeldNfts takes AssetHoldersOrderBy instead of NftHoldersOrderBy for orderBy and the deprecated order.
The middleware no longer has eventArg0..eventArg3. getEventByIndexedArgs and getEventsByIndexedArgs take refs, what the event involves in any argument (identity:, account:, asset:, ticker: or portfolio: followed by its id), and args, argument values keyed by field name, both matched with contains. The event type is still required, which also narrows the unindexed args filter. BREAKING CHANGE: 🧨 getEventByIndexedArgs and getEventsByIndexedArgs take refs and args instead of eventArg0, eventArg1 and eventArg2.
… form In the 7.0 form an extrinsic permission maps each pallet name to its extrinsics as these, except or whole. Only pallets given as these were read, so a pallet given whole, or with exceptions, was dropped. The redesigned middleware writes every row in this form, including permissions set before 7.0, which used to be read through the legacy form.
The middleware stores dates in UTC and serialises them with no zone marker, so new Date() read them as local time and shifted them by the reader's offset. middlewareDateToDate reads them as UTC. It is used for event and block dates, instruction trade and value dates, authorization expiries, mediator affirmation expiries and distribution payment dates. Extrinsic block dates already appended the marker themselves.
Mocks select createdEvent and updatedEvent instead of a block and an event index, legs and affirmations carry id columns, portfolio movements come from assetTransactions, POLYX history from polyxEntries, and dates are strings as the middleware sends them. New tests cover per-holder affirmations, NFT holders paged past the middleware's 100 row limit, event refs and args, and the POLYX entry mapping.
The redesigned middleware stores an instruction's and a venue's chain sequence number zero padded to 10 digits, so ordering by id is numeric, and a filter has to use the padded form. Instruction and Venue filtered by the bare number and matched nothing: Instruction.details reported the instruction as not yet processed, and its affirmations, legs, events and a venue's instructions came back empty. bigNumberToMiddlewareSequenceId pads them.
Covers an Account holder on a leg, an NFT movement between portfolios and one between Accounts, a POLYX entry with no module, an empty page of NFT holders, a holder with no NFTs returned, and the NFT paging helper reading past one page and stopping at an empty one.
Types come from the final schema (polymesh-subquery docs/architecture 9d182a69, release/indexer-redesign 4a2b79d3). It drops StakingPosition, Validator, AssetPreApproval, EvmAccountMapping and CorporateActionDefaultConfig, some CheckpointSchedule and Proposal fields, and the one-to-one reverse fields on Account and CorporateAction. No SDK query reads any of them.
params.proposals is nullable in the middleware: bridge proposals have none, and from polymesh-subquery e9662b7c neither does a proposal whose extrinsic has no single proposing call for that MultiSig. getHistoricalProposals read it unconditionally, so one such row threw and lost the whole page. txTag and args are now null for that row. Also documents that an EventIdentifier for a seeded row has eventIndex -1, which is how the middleware now marks its synthetic seed event. BREAKING CHANGE: 🧨 HistoricalMultiSigProposal.txTag and args can be null.
The middleware keeps one AssetHolder row per Asset and Identity for fungible Assets and NFT collections alike; the old assetHolders table held fungible Assets only. getHeldAssets and getAssetHoldings therefore listed each NFT collection as a FungibleAsset, with an amount of 0, and counted it in totalCount and the paging. assetHoldersQuery now filters on asset.isNftCollection = false.
The middleware writes a claimed off-chain receipt as an affirmation row with the claiming Identity and the receipt, but no Account or portfolio. It passes the isMediator filter, so getAffirmations read its missing portfolio id and threw. Such a row now comes back with its Identity as the party and no portfolio, as it does on develop.
… at a time getHeldNfts and NftCollection.assetHolders.get read a page of collections or holders, then every NFT they held, a hundred at a time and one request after another. A page of holders owning tens of thousands of NFTs took hundreds of sequential requests the caller could not limit, and offset paging could skip or repeat an NFT moved while it was read. Each collection or holder now comes with its nftCount, which the holdings row already has, so a page is one request. Identity.getHeldNftsOfCollection lists the NFTs an Identity holds in one collection, a page at a time. BREAKING CHANGE: 🧨 HeldNfts and IdentityHeldNfts carry nftCount: BigNumber instead of nfts: Nft[]. Use identity.getHeldNftsOfCollection to list them.
…ic made A transaction made outside any extrinsic, such as a settlement the chain executes at the start of a block, has no extrinsic index, and getTransactionHistory turned it into NaN. It is null now. Testnet has 17,046 such transactions and mainnet 1,142. The transaction mocks are moved to the middleware's shape, with the indexes on createdEvent, and the indexes are now asserted: a 0, a null and a set value. BREAKING CHANGE: 🧨 HistoricAssetTransaction.extrinsicIndex and HistoricNftTransaction.extrinsicIndex are BigNumber | null.
getTrustingAssets wrapped every Asset an Identity is a trusted claim issuer for as a FungibleAsset, NFT collections included. It now selects the Asset's isNftCollection and returns an NftCollection for one. Testnet has 6 trusted claim issuers on NFT collections. BREAKING CHANGE: 🧨 Identity.getTrustingAssets returns (FungibleAsset | NftCollection)[] instead of FungibleAsset[].
instructionsQuery, instructionEventsQuery, instructionAffirmationsQuery, offChainAffirmationsQuery and legsQuery take Instruction and Venue ids as BigNumber and pad them to the form the middleware stores, so a caller passing the bare number is a compile error instead of a read that matches nothing. portfolioToMiddlewarePortfolioId builds a portfolio's <did>/<number> id, beside portfolioIdStringToPortfolio, for the settlement and movement filters and the settlement direction. eventsByArgs selects eventIdentifierFields like every other query.
middlewareDateToDate only treated a T separated value as a zoneless date-time. A Postgres style 2024-01-01 10:00:00, which a proxy or a change in how PostGraphile serialises dates could send, was read as local time, and new Date() is not sure to parse it with a Z appended. The separator is now normalised to T before the marker is added.
A middleware that starts its index after genesis seeds votes from chain storage, which from 7.0 keeps that a signer voted but not which way, so the seeded vote's action is null. votes() asserted it was set and returned undefined for it. It is null now, and typed so. BREAKING CHANGE: 🧨 MultiSigProposalVote.action is MultiSigProposalVoteActionEnum | null.
…are release getAffirmations lists pending holders only when it reads the middleware; the chain path still lists the holders that have affirmed, so the changelog no longer says the two return the same. It also says a claimed off-chain receipt is listed with its Identity as the party, and names polymesh-subquery v20.0.0-alpha.1 as the middleware this release needs.
isNil covers both a null and an absent extrinsic index in one condition, so the tests reach every branch of it.
273ed17 to
0f3d71b
Compare
The SDK (PolymeshAssociation/polymesh-sdk#1694) changed four of its queries after review, so the fixture now holds their current text: - sdk.assetHolders(identity) leaves NFT collections out, since AssetHolder rows cover fungible Assets and NFT collections alike; - sdk.nftHolders(identity) reads NFT collections, held now; - sdk.nftHolders(identity).nfts reads one collection a page at a time; - sdk.trustingAssets selects the Asset's isNftCollection. sdk.nftCollectionHolders.nfts is removed: the SDK returns each holder's nftCount and no longer reads the holders' NFTs.
|
| result = SettlementDirectionEnum.Incoming; | ||
| } | ||
| } else if (to === identityId) { | ||
| if (!portfolio || toPortfolio === portfolio) { | ||
| } else if (toId === identityId) { | ||
| if (!portfolio || toPortfolioId === portfolioId) { | ||
| result = SettlementDirectionEnum.Outgoing; |
There was a problem hiding this comment.
The direction is inverted: a leg from this Identity/portfolio is reported as Incoming, and one to it as Outgoing. So portfolio.getTransactionHistory() shows the sender's own transfer as incoming.
This predates the PR (it came in with settlements V2, and the pre-V2 version was already inconsistent: the portfolio-filter path returned Outgoing for from, the DID-only path Incoming), but since this function is being rewritten here and the PR is breaking anyway, it seems the right place to fix it. The existing tests only assert None, so a case for each direction would be worth adding.
| result = SettlementDirectionEnum.Incoming; | |
| } | |
| } else if (to === identityId) { | |
| if (!portfolio || toPortfolio === portfolio) { | |
| } else if (toId === identityId) { | |
| if (!portfolio || toPortfolioId === portfolioId) { | |
| result = SettlementDirectionEnum.Outgoing; | |
| result = SettlementDirectionEnum.Outgoing; | |
| } | |
| } else if (toId === identityId) { | |
| if (!portfolio || toPortfolioId === portfolioId) { | |
| result = SettlementDirectionEnum.Incoming; |
| // a date with no time is already read as UTC. A date-time may come `T` or space separated, and | ||
| // `new Date` is only sure to read the `T` form with a zone marker | ||
| const isLocalDateTime = /\d[T ]\d{2}:\d{2}/.test(value) && !/(Z|[+-]\d{2}:?\d{2})$/.test(value); | ||
|
|
||
| return new Date(isLocalDateTime ? `${value.replace(' ', 'T')}Z` : value); |
There was a problem hiding this comment.
nit (optional hardening): the zone check only recognises a four-digit offset (±HH:MM / ±HHMM). A two-digit one such as 2024-05-01 10:00:00+00 (Postgres' text form for a timestamptz) or 2024-05-01T10:00:00-05 is treated as zoneless and gets Z appended, which gives an Invalid Date.
The middleware sends zoneless values today, so this doesn't happen in practice, but it is the same kind of serialisation change the space-separated handling already guards against. Relaxing the regex alone isn't enough, since V8 doesn't parse T10:00:00-05 either, so the offset needs normalising to ±HH:00:
| // a date with no time is already read as UTC. A date-time may come `T` or space separated, and | |
| // `new Date` is only sure to read the `T` form with a zone marker | |
| const isLocalDateTime = /\d[T ]\d{2}:\d{2}/.test(value) && !/(Z|[+-]\d{2}:?\d{2})$/.test(value); | |
| return new Date(isLocalDateTime ? `${value.replace(' ', 'T')}Z` : value); | |
| // a date with no time is already read as UTC. A date-time may come `T` or space separated, with a | |
| // `±HH` zone marker (Postgres' form for a whole hour offset), and `new Date` is only sure to read | |
| // the `T` form, with an `±HH:MM` marker if any | |
| const dateTime = value.replace(' ', 'T').replace(/(T[\d:.]+[+-]\d{2})$/, '$1:00'); | |
| const isLocalDateTime = | |
| /\dT\d{2}:\d{2}/.test(dateTime) && !/(Z|[+-]\d{2}:?\d{2})$/.test(dateTime); | |
| return new Date(isLocalDateTime ? `${dateTime}Z` : dateTime); |
Checked against 2024-05-01, …T10:00:00, … 10:00:00, …T10:00:00.123456, …+00:00, …+0530, … 10:00:00+00, …T10:00:00-05 and …Z; all parse to the expected UTC instant.
| if (filters.length === 1) { | ||
| filter = `filter: ${filters[0]}`; | ||
| } else if (filters.length) { | ||
| filter = `filter: { or: [ ${filters.join(', ')} ] }`; |
There was a problem hiding this comment.
When both identityId and addresses are passed, this now returns entries matching either condition: every entry of the Identity plus every entry of the given Accounts. The previous query intersected them (identityId AND address in, on each side of the movement), so passing both narrowed the result to those Accounts of that Identity.
Since each PolyxEntry is one side of a movement and carries both its identityId and accountId, and gives back the old meaning. No caller passes both at the moment (Account.getPolyxTransactions only passes accounts), so this is latent, but Context.getPolyxTransactions accepts both and would silently widen the result. A test for the combined case would pin it down.
| filter = `filter: { or: [ ${filters.join(', ')} ] }`; | |
| filter = `filter: { and: [ ${filters.join(', ')} ] }`; |
| /** | ||
| * Represents the holdings of an NFT holder | ||
| */ |
There was a problem hiding this comment.
nit: the old doc comment was left above the new one, so IdentityHeldNfts carries two JSDoc blocks and typedoc/IDEs may show the stale "Represents the holdings of an NFT holder" description.
| /** | |
| * Represents the holdings of an NFT holder | |
| */ |
| // the middleware signs the amount by direction | ||
| amount: new BigNumber(amount).abs().shiftedBy(-6), |
There was a problem hiding this comment.
nit: the schema already exposes the unsigned value, amountAbs ("unsigned. SUM = gross volume", beside the signed amount). Selecting amountAbs in polyxTransactionsQuery instead of amount, and destructuring it here, would drop the client-side .abs() and this comment:
amount: new BigNumber(amountAbs).shiftedBy(-6),The query and mocks would need amount → amountAbs too.
| if (!portfolioId) { | ||
| // a claimed off-chain receipt is affirmed by an Identity, with no Account or portfolio | ||
| return { | ||
| party: new Identity({ did: identityId }, context), | ||
| status: middlewareAffirmStatusToAffirmationStatus(status), | ||
| }; | ||
| } | ||
| return { | ||
| party: new Identity({ did: identity as string }, context), | ||
| party: new Identity({ did: identityId }, context), | ||
| portfolio: middlewarePortfolioToPortfolio( | ||
| portfolioIdStringToPortfolio(portfolioId), | ||
| context | ||
| ), | ||
| status: middlewareAffirmStatusToAffirmationStatus(status), | ||
| }; |
There was a problem hiding this comment.
nit: the identity-only branch and the portfolio branch differ only by the portfolio key, so they can be one return. A conditional spread keeps the key absent (not undefined) for a claimed off-chain receipt, so existing toEqual expectations still hold:
| if (!portfolioId) { | |
| // a claimed off-chain receipt is affirmed by an Identity, with no Account or portfolio | |
| return { | |
| party: new Identity({ did: identityId }, context), | |
| status: middlewareAffirmStatusToAffirmationStatus(status), | |
| }; | |
| } | |
| return { | |
| party: new Identity({ did: identity as string }, context), | |
| party: new Identity({ did: identityId }, context), | |
| portfolio: middlewarePortfolioToPortfolio( | |
| portfolioIdStringToPortfolio(portfolioId), | |
| context | |
| ), | |
| status: middlewareAffirmStatusToAffirmationStatus(status), | |
| }; | |
| // a claimed off-chain receipt is affirmed by an Identity, with no Account or portfolio | |
| return { | |
| party: new Identity({ did: identityId }, context), | |
| ...(portfolioId | |
| ? { | |
| portfolio: middlewarePortfolioToPortfolio( | |
| portfolioIdStringToPortfolio(portfolioId), | |
| context | |
| ), | |
| } | |
| : {}), | |
| status: middlewareAffirmStatusToAffirmationStatus(status), | |
| }; |
| assetId | ||
| callerId | ||
| palletName | ||
| eventId |
There was a problem hiding this comment.
nit: AssetPermissions.getOperationHistory (the only caller) maps each node to middlewareEventToEventIdentifier(createdEvent!) and reads nothing else, so these four fields are unused. palletName and eventId were already selected before this PR; assetId and callerId are new. All four are still needed as filter variables, just not in the selection.
| assetId | |
| callerId | |
| palletName | |
| eventId |
There was a problem hiding this comment.
Callers of assetAgentHistoryQuery also doesn't use assetId, type or permissions
| function withMiddlewareIds(filters: Record<string, unknown>): Record<string, unknown> { | ||
| return Object.fromEntries( | ||
| Object.entries(filters).map(([key, value]) => [ | ||
| key, | ||
| BigNumber.isBigNumber(value) ? bigNumberToMiddlewareSequenceId(value) : value, | ||
| ]) | ||
| ); | ||
| } |
There was a problem hiding this comment.
nit (hardening): this pads every BigNumber value in the filters, not just the id keys. Today that's correct, since every BigNumber reaching it is an Instruction or Venue id (legIndex is a plain number), but a later filter with a BigNumber amount or index would be silently turned into a 10-digit string and match nothing. Padding by named key (id, venueId, instructionId) would make the intent explicit and keep it from catching other fields.



Summary
Moves every middleware (SubQuery) query to the redesigned SubQuery schema (
PolymeshAssociation/polymesh-subquery,docs/architecture,966b4a8c). After this, the SDK requires a middleware running that schema. This is a major-version (v32) change, delivered as abc-branch perNEXT_RELEASE.md. The migration guide is inchangelogs/v32.0.0-next.md.What changed
src/middleware/types.tsis regenerated.scripts/processMiddlewareTypes.tsno longer adds duplicateCreatedAt*members to an OrderBy enum that already has them.createdEvent/updatedEvent).middlewareEventToEventIdentifiertakes that event, andeventIdentifierFieldsis the shared selection for it.fromId,fromPortfolioId,toAccountId, …). Portfolio ids aredid/number.assetAgents,assetAgentHistoriesandassetAgentActions.Pending/Affirmed).assetTransactions: internal transfers that no instruction made.polyxEntries, one row per side of a movement.args, an object keyed by argument name.assetHolders, thennftsfor that page, 100 at a time. The middleware returns at most 100 rows per request without saying so, so a nested connection would truncate silently.refs(identity, account, asset, ticker or portfolio) and byargscontainment.Breaking changes
CREATED_BLOCK_ID_*andEVENT_IDX_*becomeCREATED_EVENT_ID_*.TickerExternalAgentActionsOrderBybecomesAssetAgentActionsOrderBy, andNftHoldersOrderBybecomesAssetHoldersOrderBy.PolyxTransactionsOrderBybecomesPolyxEntriesOrderBy.Instruction.getAffirmations()lists every portfolio and Account holder, includingPendingones, and gains an optionalportfolio. It no longer lists mediators.HistoricPolyxTransactionchanges shape:account/identityandcounterpartyAccount/counterpartyIdentity, pluskind,directionandpool. Itsamountis always positive.BalanceTypeEnumis no longer exported.ExtrinsicData.paramsisRecord<string, unknown> | nullinstead of an array.getEventByIndexedArgs/getEventsByIndexedArgstakerefsandargsinstead ofeventArg0..2.Fixes
assetPermissions.enabledAtreturns this Identity's agent event instead of that of the Asset's most recent agent.Testing
yarn build:ts,yarn lint: pass.966b4a8c: 48 of 48 valid on both chains, returning data.Open items
claimsGroupingtakes 4–10 s on both chains. Its query is unchanged and matches the fixture.19.6.0, soMINIMUM_SQ_VERSIONcan't tell it apart from the old one.getTransactionByHashfails on the public testnet because the chain's ownTransactionPaymentApi.queryInfopanics for that extrinsic.PolymeshAssociation/polymesh-subquery, branchfix/sdk-nft-collection-holders-fixture.