Skip to content

feat!: 🎸 move middleware queries to the redesigned SubQuery schema - #1694

Draft
prashantasdeveloper wants to merge 17 commits into
developfrom
bc-1-middleware-redesigned-schema
Draft

prashantasdeveloper wants to merge 17 commits into
developfrom
bc-1-middleware-redesigned-schema

Conversation

@prashantasdeveloper

Copy link
Copy Markdown
Contributor

Summary

Moves every middleware (SubQuery) query to the redesigned SubQuery schema (PolymeshAssociation/polymesh-subquery, docs/architecture, 966b4a8c). After this, the SDK requires a middleware running that schema. This is a major-version (v32) change, delivered as a bc- branch per NEXT_RELEASE.md. The migration guide is in changelogs/v32.0.0-next.md.

What changed

  • src/middleware/types.ts is regenerated. scripts/processMiddlewareTypes.ts no longer adds duplicate CreatedAt* members to an OrderBy enum that already has them.
  • Rows reference the event that created and last updated them (createdEvent / updatedEvent). middlewareEventToEventIdentifier takes that event, and eventIdentifierFields is the shared selection for it.
  • Relations are read as id columns (fromId, fromPortfolioId, toAccountId, …). Portfolio ids are did/number.
  • External agents read assetAgents, assetAgentHistories and assetAgentActions.
  • Settlement: legs and affirmations are mapped from id columns. Affirmations are per holder (Pending / Affirmed).
  • Portfolio movements are read from assetTransactions: internal transfers that no instruction made.
  • POLYX history is read from polyxEntries, one row per side of a movement.
  • Extrinsic parameters are read from args, an object keyed by argument name.
  • NFT holders take two queries: a page of assetHolders, then nfts for that page, 100 at a time. The middleware returns at most 100 rows per request without saying so, so a nested connection would truncate silently.
  • Events are found by refs (identity, account, asset, ticker or portfolio) and by args containment.
  • Instruction and Venue lookups use the middleware's zero-padded ids.

Breaking changes

  • The SDK requires the redesigned middleware.
  • OrderBy enums: CREATED_BLOCK_ID_* and EVENT_IDX_* become CREATED_EVENT_ID_*. TickerExternalAgentActionsOrderBy becomes AssetAgentActionsOrderBy, and NftHoldersOrderBy becomes AssetHoldersOrderBy. PolyxTransactionsOrderBy becomes PolyxEntriesOrderBy.
  • Instruction.getAffirmations() lists every portfolio and Account holder, including Pending ones, and gains an optional portfolio. It no longer lists mediators.
  • HistoricPolyxTransaction changes shape: account / identity and counterpartyAccount / counterpartyIdentity, plus kind, direction and pool. Its amount is always positive. BalanceTypeEnum is no longer exported.
  • ExtrinsicData.params is Record<string, unknown> | null instead of an array.
  • getEventByIndexedArgs / getEventsByIndexedArgs take refs and args instead of eventArg0..2.

Fixes

  • Middleware dates are read as UTC. They have no zone marker, so they used to be read as local time.
  • Extrinsic permissions in the per-pallet form keep pallets given in full or with exceptions. These used to be dropped, and the new middleware writes every row in this form.
  • assetPermissions.enabledAt returns this Identity's agent event instead of that of the Asset's most recent agent.

Testing

  • yarn build:ts, yarn lint: pass.
  • Unit tests: 213 suites, 2,883 tests passed.
  • Every query builder was built with the branch's own code and run against the dev testnet and mainnet indexers at 966b4a8c: 48 of 48 valid on both chains, returning data.
  • Read-only end to end: the built SDK ran against the public testnet RPC with the testnet indexer, which shares its genesis. Events, POLYX history, extrinsic history, held NFTs (one holder has 2,724), collection holders, asset and agent history, instruction details, affirmations, legs and mediators, portfolio history, authorizations, MultiSig proposals, investments, distribution payments and trusted issuers all returned correct data.

Open items

  • claimsGrouping takes 4–10 s on both chains. Its query is unchanged and matches the fixture.
  • Reads the SDK never paged (settlement history, portfolio movements, trusting assets, agent history) stop at the middleware's 100-row cap. This predates this PR.
  • The redesigned indexer still reports version 19.6.0, so MINIMUM_SQ_VERSION can't tell it apart from the old one.
  • getTransactionByHash fails on the public testnet because the chain's own TransactionPaymentApi.queryInfo panics for that extrinsic.
  • The matching fixture change is in PolymeshAssociation/polymesh-subquery, branch fix/sdk-nft-collection-holders-fixture.

Types come from the indexer image 966b4a8c (PolymeshAssociation/polymesh-subquery,
docs/architecture). The processing script no longer adds CreatedAt members to an
OrderBy enum whose entity already has a createdAt column, which produced
duplicate members for IndexerAnomaly and IndexOrigin.

The SDK does not compile against these types yet; the following commits move
each query and its mapping code over.

BREAKING CHANGE: 🧨 the SDK requires a middleware running the redesigned
SubQuery schema. Re-exported middleware enums and OrderBy enums follow the new
schema (for example CREATED_BLOCK_ID_* becomes CREATED_EVENT_ID_*).
Entities now reference the event that created or last updated them
(createdEvent, updatedEvent) instead of carrying a block and an event index.
middlewareEventToEventIdentifier takes that event, and eventIdentifierFields is
the selection every query uses for it.

The middleware never indexes NoData or investor uniqueness claims, so the
deprecated claim filter only has NoType left to drop.
…w schema

Asset, claim, trusted issuer, custom claim type, distribution payment,
multisig, portfolio, authorization and investment queries select createdEvent
and updatedEvent, order by CREATED_EVENT_ID_*, and filter on the *Id columns
that replaced string relations. External agent queries read assetAgents,
assetAgentHistories and assetAgentActions.

Authorizations sent to an Account are filtered and mapped by its SS58 address
(toAccountId) instead of its public key. An investment is filtered by its
offering's id, <offering asset id>/<offering number>.

AssetPermissions.enabledAt now looks up this Identity's agent row. It used to
return when the Asset's most recent agent was added, whichever Identity that
was.

BREAKING CHANGE: 🧨 AssetPermissions.getOperationHistory takes
AssetAgentActionsOrderBy instead of TickerExternalAgentActionsOrderBy, and
OrderBy values follow the new schema (CREATED_BLOCK_ID_* is now
CREATED_EVENT_ID_*).
Legs select their holders as ids (fromId, fromPortfolioId, fromAccountId and
the to* columns), and a portfolio holder is read from its <did>/<number> id.
Settlement filters match portfolio ids and identity ids, historical
instructions filter parties and legs by identity id, and a leg's addresses are
matched with contains. Instruction events, instructions and affirmations select
createdEvent and updatedEvent.

Portfolio movements are read from assetTransactions: internal transfers that no
instruction made. NFT movements are listed as NFT legs.

Affirmation rows exist for every holder from the instruction's creation, as
Pending or Affirmed. getAffirmations lists the holders, without mediators, and
says which portfolio each Identity affirms for, as the chain path does.

Adds changelogs/v32.0.0-next.md.

BREAKING CHANGE: 🧨 Instruction.getAffirmations lists every portfolio and
Account holder, including those still Pending, once per portfolio, and never
the mediators; a rejection is no longer reported as an affirmation status.
polyxTransactions is gone. polyxEntries has one row per side of a movement,
so an Account's or an Identity's history filters on accountId or identityId
alone, with no from/to OR. Each HistoricPolyxTransaction is the change one
movement made to one Account: account and identity, the counterparty if there
is one, a positive amount with its direction (Credit or Debit), the movement
kind and the balance pool.

src/types re-exports PolyxPool, MovementKind, EntryDirection,
PolyxEntriesOrderBy and NftsOrderBy, and no longer BalanceTypeEnum or
NftHoldersOrderBy.

BREAKING CHANGE: 🧨 HistoricPolyxTransaction replaces fromAccount/fromIdentity
and toAccount/toIdentity with account/identity and
counterpartyAccount/counterpartyIdentity, and type: BalanceTypeEnum with
pool: PolyxPool, kind: MovementKind and direction: EntryDirection.
getPolyxTransactions takes PolyxEntriesOrderBy. BalanceTypeEnum and
NftHoldersOrderBy are no longer exported.
Extrinsics no longer have paramsTxt. args holds the call's arguments as an
object keyed by argument name, with integers as decimal strings and Accounts
as SS58 addresses, or null when the indexer could not decode them.

BREAKING CHANGE: 🧨 ExtrinsicData.params is the call's arguments keyed by
name (Record<string, unknown> | null) instead of an array.
nftHolders is gone. getHeldNfts reads a page of the Identity's assetHolders
rows for NFT collections, then every NFT it holds in them from nfts.
AssetHolders.get reads a page of the collection's holders, then every NFT they
hold. Both page through nfts a hundred at a time, since the middleware returns
no more than 100 rows to a request without saying so; a nested holdings
connection would stop there.

BREAKING CHANGE: 🧨 Identity.getHeldNfts takes AssetHoldersOrderBy instead of
NftHoldersOrderBy for orderBy and the deprecated order.
The middleware no longer has eventArg0..eventArg3. getEventByIndexedArgs and
getEventsByIndexedArgs take refs, what the event involves in any argument
(identity:, account:, asset:, ticker: or portfolio: followed by its id), and
args, argument values keyed by field name, both matched with contains. The
event type is still required, which also narrows the unindexed args filter.

BREAKING CHANGE: 🧨 getEventByIndexedArgs and getEventsByIndexedArgs take
refs and args instead of eventArg0, eventArg1 and eventArg2.
… form

In the 7.0 form an extrinsic permission maps each pallet name to its
extrinsics as these, except or whole. Only pallets given as these were read,
so a pallet given whole, or with exceptions, was dropped. The redesigned
middleware writes every row in this form, including permissions set before
7.0, which used to be read through the legacy form.
The middleware stores dates in UTC and serialises them with no zone marker,
so new Date() read them as local time and shifted them by the reader's
offset. middlewareDateToDate reads them as UTC. It is used for event and
block dates, instruction trade and value dates, authorization expiries,
mediator affirmation expiries and distribution payment dates. Extrinsic block
dates already appended the marker themselves.
Mocks select createdEvent and updatedEvent instead of a block and an event
index, legs and affirmations carry id columns, portfolio movements come from
assetTransactions, POLYX history from polyxEntries, and dates are strings as
the middleware sends them. New tests cover per-holder affirmations, NFT
holders paged past the middleware's 100 row limit, event refs and args, and
the POLYX entry mapping.
The redesigned middleware stores an instruction's and a venue's chain
sequence number zero padded to 10 digits, so ordering by id is numeric, and a
filter has to use the padded form. Instruction and Venue filtered by the bare
number and matched nothing: Instruction.details reported the instruction as
not yet processed, and its affirmations, legs, events and a venue's
instructions came back empty. bigNumberToMiddlewareSequenceId pads them.
Covers an Account holder on a leg, an NFT movement between portfolios and one
between Accounts, a POLYX entry with no module, an empty page of NFT holders,
a holder with no NFTs returned, and the NFT paging helper reading past one
page and stopping at an empty one.
Types come from the final schema (polymesh-subquery docs/architecture
9d182a69, release/indexer-redesign 4a2b79d3). It drops StakingPosition,
Validator, AssetPreApproval, EvmAccountMapping and
CorporateActionDefaultConfig, some CheckpointSchedule and Proposal fields,
and the one-to-one reverse fields on Account and CorporateAction. No SDK
query reads any of them.
params.proposals is nullable in the middleware: bridge proposals have none,
and from polymesh-subquery e9662b7c neither does a proposal whose extrinsic
has no single proposing call for that MultiSig. getHistoricalProposals read
it unconditionally, so one such row threw and lost the whole page. txTag and
args are now null for that row.

Also documents that an EventIdentifier for a seeded row has eventIndex -1,
which is how the middleware now marks its synthetic seed event.

BREAKING CHANGE: 🧨 HistoricalMultiSigProposal.txTag and args can be null.
@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Failed Quality Gate failed

Failed conditions
2 New Code Smells (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Comment on lines 673 to 682
}
return {
party: new Identity({ did: identity as string }, context),
party: new Identity({ did: identityId }, context),
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
portfolio: middlewarePortfolioToPortfolio(
portfolioIdStringToPortfolio(portfolioId!),
context
),
status: middlewareAffirmStatusToAffirmationStatus(status),
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: getAffirmations() throws for an instruction with a claimed off-chain receipt.

When a receipt is claimed, subquery v20.0.0-alpha.1 (mapSettlement.ts, the ReceiptClaimed handler) creates an InstructionAffirmation with isMediator: false and offChainReceiptId set, but no accountId and no portfolioId. That row passes the isMediator: false filter, skips the accountId branch and reaches portfolioIdStringToPortfolio(undefined), which throws TypeError: Cannot read properties of undefined (reading 'split').

The return type already makes portfolio optional, so the smallest fix keeps the row and returns it with the Identity as the party (on develop these rows already come back with only an Identity as the party):

Suggested change
}
return {
party: new Identity({ did: identity as string }, context),
party: new Identity({ did: identityId }, context),
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
portfolio: middlewarePortfolioToPortfolio(
portfolioIdStringToPortfolio(portfolioId!),
context
),
status: middlewareAffirmStatusToAffirmationStatus(status),
};
}
if (!portfolioId) {
// a claimed off-chain receipt is affirmed by an Identity, with no Account or portfolio
return {
party: new Identity({ did: identityId }, context),
status: middlewareAffirmStatusToAffirmationStatus(status),
};
}
return {
party: new Identity({ did: identityId }, context),
portfolio: middlewarePortfolioToPortfolio(
portfolioIdStringToPortfolio(portfolioId),
context
),
status: middlewareAffirmStatusToAffirmationStatus(status),
};

If receipts should be left out instead (they are already reported by getOffChainAffirmations), filter them in the query with offChainReceiptExists: false. createArgsAndFilters can't express that bare boolean, so it would have to be a fixed clause in instructionAffirmationsQuery, and totalCount then stays consistent. Either way, please add a test with a receipt row: the mocks only have rows with an account or a portfolio.

@@ -59,13 +56,13 @@ export function assetHoldersQuery(
start?: BigNumber,
orderBy?: AssetHoldersOrderBy | AssetHoldersOrderBy[]
): QueryOptions<PaginatedQueryArgs<QueryArgs<AssetHolder, 'identityId'> & { amount?: string }>> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: NFT collections come back as fungible Assets.

In the redesigned schema AssetHolder is one row per (asset, identity) for every asset. Subquery's getAssetHolder creates it with amount: 0, nftCount: 0, and the NFT path updates nftCount on the same row. The old assetHolders table was fungible only (NFTs lived in nftHolders), so this query never needed a filter. Now it does:

  • Identity.getHeldAssets() wraps each NFT collection as new FungibleAsset(...).
  • getFungibleAssetHoldings() (without heldNow) lists each collection with an amount of 0.
  • Both totalCount values, and the paging, include these rows.

nftHoldersQuery already filters asset: { isNftCollection: { equalTo: true } }. This query needs the mirror image. Because createArgsAndFilters only emits field: { op: $var }, the simplest approach is to write the filter out the way nftHoldersQuery does:

const hasAmount = filters.amount !== undefined;
const amountArg = hasAmount ? ', $amount: BigFloat!' : '';
const amountFilter = hasAmount ? ', amount: { greaterThan: $amount }' : '';

const query = gql`
  query AssetHoldersQuery($start: Int, $size: Int, $identityId: String!${amountArg}) {
    assetHolders(
      filter: {
        identityId: { equalTo: $identityId }
        asset: { isNftCollection: { equalTo: false } }${amountFilter}
      }
      ...

Please add a test asserting the filter, since the current mocks can't catch this.

Comment thread src/utils/internal.ts
nfts: { nodes, totalCount: total },
},
} = await context.queryMiddleware<Ensured<Query, 'nfts'>>(
buildQuery(size, new BigNumber(nfts.length))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Offset paging can skip or duplicate NFTs.

The next offset is nfts.length, and nothing pins the set between pages. If an NFT earlier in the order is burned or moved away mid-read, every later row shifts down one, so the next page skips an NFT. If one is minted or moved in, an NFT is returned twice. totalCount is re-read on every page, so the loop doesn't notice.

Keyset paging on Nft.id (unique, and NftsOrderBy.IdAsc exists) avoids both, and lets the loop end on a short page instead of trusting totalCount:

export async function getAllMiddlewareNfts(
  context: Context,
  buildQuery: (size: BigNumber, afterId: string | undefined) => QueryOptions
): Promise<MiddlewareNft[]> {
  const size = MAX_GQL_PAGE_SIZE;
  const nfts: MiddlewareNft[] = [];
  let afterId: string | undefined;
  let page: MiddlewareNft[];

  do {
    ({
      data: {
        nfts: { nodes: page },
      },
    } = await context.queryMiddleware<Ensured<Query, 'nfts'>>(
      buildQuery(new BigNumber(size), afterId)
    ));

    nfts.push(...page);
    afterId = page[page.length - 1]?.id;
  } while (page.length === size);

  return nfts;
}

The two builders (heldNftsQuery and nftCollectionHoldersNftsQuery) would then take $afterId: String and use orderBy: [ID_ASC] with id: { greaterThan: $afterId }, adding that clause only when it is set. Grouping by assetId and holder is unaffected by the order.

Comment thread src/utils/internal.ts
*
* @param buildQuery - the query for one page, given its size and offset
*/
export async function getAllMiddlewareNfts(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unbounded sequential reads.

getHeldNfts and NftCollection's AssetHolders.get return one page of collections or holders, but then read every NFT of that page, 100 per request, one request after another. A page of 25 holders who together own 50k NFTs takes about 500 sequential requests in one SDK call, and the caller can't limit it.

Options, roughly in order of preference:

  1. Return only nftCount (already selected) for each collection or holder, and let the caller page the NFTs with a separate paged method. This is the only option that keeps a "page" a bounded amount of work. It changes the return type, but this PR is already breaking.
  2. Keep the full read but document the cost on both public methods (@note reads every NFT the page holds, which can be many requests).
  3. Parallelise from the first page's totalCount, capped at MAX_CONCURRENT_REQUESTS. Note that this conflicts with the keyset fix on L1729: parallel offset pages bring the skip/duplicate problem back. If you go this way, fix: 🐛 Account NFT reads by collection, complete exemption reads, and modifying a Ballot #1688 adds allInOrder, which keeps error order deterministic.

instructionAffirmationsQuery(
{
instructionId: id.toString(),
instructionId: bigNumberToMiddlewareSequenceId(id),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: move the id padding into the query builders.

Seven call sites in this file (plus the Venue ones) have to remember bigNumberToMiddlewareSequenceId(id). The next caller that passes id.toString() silently matches nothing, because the middleware stores 0000000012, not 12, and no test would fail unless it asserted the variables.

Suggest that instructionsQuery, legsQuery, instructionAffirmationsQuery, instructionEventsQuery (and the venue queries) take instructionId: BigNumber / venueId: BigNumber and pad internally, the way subquery's padNumericId does at each lookup. The type change then makes a forgotten conversion a compile error.

* Get a single event by any of its indexed arguments
* Get events of one type by what they involve and by their arguments
*/
export function eventsByArgs(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this query's selection (eventIdx block { blockId hash datetime }, L54–59) is written out by hand. Every other query this PR touches uses the new shared eventIdentifierFields, which exists so that middlewareEventToEventIdentifier always gets what it reads. Please use it here too:

        nodes {
          ${eventIdentifierFields}
        }

variables.to = identityId;
} else {
// a portfolio's id is `<did>/<number>`, the default portfolio being number 0
const portfolio = `${identityId}/${portfolioId?.toString() ?? '0'}`;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the middleware portfolio id <did>/<number> (default portfolio as 0) is built inline here, in portfolios.ts:62 and in conversion.ts:5455, while the reverse direction already has portfolioIdStringToPortfolio. A portfolioToMiddlewarePortfolioId(identityId, portfolioId?) beside it in conversion.ts would keep the default-portfolio rule and the format in one place. If the indexer ever pads portfolio numbers, as it now does instruction ids, there is then only one function to change.

Comment thread src/utils/conversion.ts
Comment on lines +2459 to +2461
const isLocalDateTime = value.includes('T') && !/(Z|[+-]\d{2}:?\d{2})$/.test(value);

return new Date(isLocalDateTime ? `${value}Z` : value);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit / hardening: only T-separated values count as zoneless date-times. A Postgres-style 2024-01-01 10:00:00 (which a proxy, or a change in how PostGraphile serialises dates, could produce) skips the Z and is read as local time, which is the bug this function exists to prevent. Also, new Date('2024-01-01 10:00:00Z') isn't guaranteed by the spec, so normalise the separator too:

Suggested change
const isLocalDateTime = value.includes('T') && !/(Z|[+-]\d{2}:?\d{2})$/.test(value);
return new Date(isLocalDateTime ? `${value}Z` : value);
const isLocalDateTime = /\d[T ]\d{2}:\d{2}/.test(value) && !/(Z|[+-]\d{2}:?\d{2})$/.test(value);
return new Date(isLocalDateTime ? `${value.replace(' ', 'T')}Z` : value);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants