[skip-ci][ntuple] add SoA design document by jblomer · Pull Request #23202 · root-project/root

jblomer · 2026-08-31T13:16:14Z

Part of #19230

This should be merged once the first complete version of the code is ready. Submitted as a preview PR.

github-actions · 2026-09-01T06:26:24Z

Test Results

23 files 23 suites 3d 12h 29m 3s ⏱️
3 860 tests 3 855 ✅ 0 💤 5 ❌
78 655 runs 78 649 ✅ 1 💤 5 ❌

For more details on these failures, see this check.

Results for commit 8333a10.

pcanal · 2026-10-16T18:06:36Z

+Without dictionaries, RNTuple can still read data with an emulated schema derived from the RNTuple fields.
+Type emulation reads classes as untyped records, collections as `std::vector`s, and custom enums as integers.
+Type emulation works for all fields except streamer fields.
+Emulated types cannot be used for writing, but they can be used for merging.


I guess this technically means that Emulated types can not be used to create field (but can be used for writing when merging). (Related questions, since the slow merging can append entries to existing RNTuple, can a clever user find a way to leverage that code to also write into an existing RNTuple?)

The merger does not close the output ntuple during merging, so it's not really appending to an existing one...

pcanal · 2026-10-16T18:10:19Z

@@ -0,0 +1,111 @@
+# SoA I/O in RNTuple
+
+RNTuple has a mechanism to represent a collection of a class, i.e. an "array of struct" (AoS),


represent

Would read back or connect be more accurate?

Changed to read and write

pcanal · 2026-10-16T18:26:41Z

+    The data type of that data member in the SoA class must be either `RVec<T>` or 
+    a SoA type that has `T` as an underlying record type (nested SoA type).
+    The SoA type must have no additional persistent data members.
+  - The SoA type and its underlying record type must have the same class version number.


Is that a fundamental limitation? In particular this means that un-versioned class/struct are not eligible to be SOA. Should it be made more pro-eminent that the class need be versioned (either via a ClassDef (intrusive) or via a dictionary generation annotation (non intrusive to the underlying record type)?

It's not a fundamental limitation, it is a limitation that we just impose for sanity. With this limitation, yes, I would like to not make SoA types available as unversioned class.

pcanal · 2026-10-16T18:28:25Z

+    The SoA type must have no additional persistent data members.
+  - The SoA type and its underlying record type must have the same class version number.
+
+These conditions are checked at runtime when an `RSoAField` is created.


Related to the question about version. Does it also check that the class is versioned or just that the class version match (2 un-versioned classes might have matching version by happenstance ... in which case upon reading back they may not have the same assigned version number).

We may be susceptible to this. The goal is enforcing versioned classes: #23749

pcanal · 2026-10-16T18:30:50Z

+  - For every persistent member of type `T` in the underlying record type, 
+    there must be a member with the same name in the SoA type.
+    The data type of that data member in the SoA class must be either `RVec<T>` or 
+    a SoA type that has `T` as an underlying record type (nested SoA type).


Suggested change

a SoA type that has `T` as an underlying record type (nested SoA type).

a SoA type that has `T` as an underlying record type (nested SoA type).

- For all the (persisted) instances of the SoA all the persistent data members much have the same number of elements.

Yes, however that's not a restriction on the type. A few lines further down we have the line

Equal vector lengths are ensured by construction when reading from disk and checked when writing to disk.

It is an invariant of the class. The next section is different in a bit in nature. I.e. When I am reading it, I see this section as "this is what the class must have" while the later section is "and here is how we enforce those requirements"

Makes sense. I added a bullet point along these lines.

pcanal · 2026-10-16T18:34:52Z

+For added members, reading will set the corresponding vector(s) to the collection length 
+and default-initialize the vector elements.
+This is different to added members of normal classes, for which reading is a no-op.


This implies that the default-initialize is executed at each entry read. Is that correct? (i.e. at each entry we do the moral equivalent of vec.clear(); vec.resize(n);)

Yes, that is correct.

pcanal

LGTM.

jblomer requested review from enirolf, hahnjo, pcanal and vepadulano August 31, 2026 13:16

jblomer self-assigned this Aug 31, 2026

jblomer requested a review from silverweed as a code owner August 31, 2026 13:16

jblomer added the in:RNTuple label Aug 31, 2026

jblomer marked this pull request as draft August 31, 2026 13:16

jblomer force-pushed the ntuple-soa-doc branch from a9c8734 to 8333a10 Compare August 31, 2026 21:35

jblomer changed the title ~~[ski-ci][ntuple] add SoA design document~~ [skip-ci][ntuple] add SoA design document Sep 1, 2026

jblomer closed this Sep 1, 2026

jblomer reopened this Sep 1, 2026

hahnjo reviewed Sep 2, 2026

View reviewed changes

Comment thread tree/ntuple/doc/README.md

Comment thread tree/ntuple/doc/SoA.md Outdated

Comment thread tree/ntuple/doc/SoA.md Outdated

silverweed reviewed Sep 3, 2026

View reviewed changes

jblomer force-pushed the ntuple-soa-doc branch 2 times, most recently from 2fc5b05 to 95d209d Compare September 3, 2026 09:39

jblomer added 4 commits September 3, 2026 11:39

[NFC][ntuple] match naming style for Tuning.md

67ec025

[NFC][ntuple] minor updates to README.md

fff624a

[NFC][ntuple] add RNTupleExporter to Architecture.md

cab56a6

[NFC][ntuple] mention type emulation in Architecture.md

444cb7e

vepadulano reviewed Sep 3, 2026

View reviewed changes

Comment thread tree/ntuple/doc/SoA.md Outdated

jblomer marked this pull request as ready for review October 10, 2026 22:56

jblomer requested review from hahnjo, silverweed and vepadulano October 10, 2026 22:56

pcanal reviewed Oct 16, 2026

View reviewed changes

Comment thread tree/ntuple/doc/SoA.md Outdated

pcanal reviewed Oct 16, 2026

View reviewed changes

Comment thread tree/ntuple/doc/SoA.md Outdated

pcanal approved these changes Oct 16, 2026

View reviewed changes

jblomer force-pushed the ntuple-soa-doc branch from 95d209d to 93c3af9 Compare October 17, 2026 21:50

[NFC][ntuple] add SoA design document

1cf6381

jblomer force-pushed the ntuple-soa-doc branch from 93c3af9 to 1cf6381 Compare October 19, 2026 10:57

		@@ -0,0 +1,111 @@
		# SoA I/O in RNTuple

		RNTuple has a mechanism to represent a collection of a class, i.e. an "array of struct" (AoS),

	a SoA type that has `T` as an underlying record type (nested SoA type).
	a SoA type that has `T` as an underlying record type (nested SoA type).
	- For all the (persisted) instances of the SoA all the persistent data members much have the same number of elements.

Conversation

jblomer commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 1, 2026

Test Results

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pcanal Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jblomer Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pcanal left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jblomer commented Aug 31, 2026 •

edited

Loading

pcanal Oct 16, 2026 •

edited

Loading

jblomer Oct 17, 2026 •

edited

Loading