Skip to content

HDDS-16377. Datanode ReadBlock streaming path allocates and copies a fresh buffer for every request - #11873

Open
ss77892 wants to merge 4 commits into
apache:masterfrom
ss77892:HDDS-16377
Open

ss77892 wants to merge 4 commits into
apache:masterfrom
ss77892:HDDS-16377

Conversation

@ss77892

@ss77892 ss77892 commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

On the datanode, a streaming ReadBlock is served over one gRPC stream per block, through one RandomAccessFileChannel that GrpcXceiverService keeps open for the stream. Every ReadBlock request on that stream used to:

  • look up the block's BlockData in the container DB via BlockManager.getBlock,
  • allocate a new heap ByteBuffer sized by the client-chosen response size, and
  • copy that buffer again into the response with ByteString.copyFrom.

For a sequential read of a large block this is one DB lookup, one allocation and two copies per response. This PR removes the per-request allocation and the extra copy, and caches the block metadata for the life of the stream:

  1. Reuse the read buffer. RandomAccessFileChannel.getReadBuffer(capacity) keeps one buffer per stream, reallocates it only when a larger one is needed, and drops it on close(). The buffer is now sized by the range actually being read rather than by the response size the client asked for.
  2. Avoid the copy into the response. ContainerCommandResponseBuilders.getReadBlockResponse takes the handler's byteBufferToByteString conversion, as getReadChunkResponse already does. With ozone.UnsafeByteOperations.enabled (default true) the response wraps the buffer instead of copying it. Reusing the buffer is safe because gRPC serializes the response inside StreamObserver.onNext, before the next range refills the buffer; this contract is documented on getReadBlockResponse.
  3. Cache BlockData per stream. RandomAccessFileChannel caches the BlockData of the block it serves and clears it on close().
    • A request hits the cache only for the same block (container and local ID), and only when the cached BCSID is not lower than the requested one. Otherwise the block is re-resolved via BlockManager.getBlock, which keeps its BCSID_MISMATCH check.
    • The cache is bypassed and refreshed whenever the requested range reaches the cached block size, so a block that grew under an open stream is re-resolved.
  4. Reject negative offset or length with INVALID_ARGUMENT before the cache check, closing the block file like the existing OUT_OF_RANGE rejection.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16377

How was this patch tested?

New unit tests:

  • TestRandomAccessFileChannel
    • blockDataIsCachedPerBlockAndClearedOnClose: cache hit per BlockID, miss for another block or BCSID, cleared on close.
    • readBufferIsReusedGrownAndDroppedOnClose: buffer reused and cleared, reallocated only to grow, dropped on close.
  • TestKeyValueHandler
    • testReadBlockReadBufferIsSizedByTheRequest: a large client response size does not size the buffer.
    • testReadBlockLengthBeyondTheBlockIsClamped: Long.MAX_VALUE length, aligned and misaligned offset.
    • testReadBlockCachedBlockDataIsReusedWithinTheBlock: one getBlock for two requests within the block.
    • testReadBlockCachedBlockDataIsRefreshedWhenTheBlockGrows: a request past the cached size sees the grown block.
    • testReadBlockCachedBlockDataBoundsTheReadWhenTheBlockGrows: a request inside the cached size does not read past it.
    • testReadBlockErrorOnTheStreamClosesTheBlockFile: extended with negative offset and negative length.

Benchmark report vs master (3 node cluster using LXD containers)
build_comparison_20261030_204023.pdf

…fresh buffer for every request

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ss77892

ss77892 commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

@szetszwo @chungen0126 @peterxcli would you mind take a look?

@szetszwo szetszwo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ss77892 , thanks for working on this! Please see the comments inlined.

Comment on lines +44 to +45
private BlockID cachedBlockID;
private BlockData cachedBlockData;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • cachedBlockID is not needed since cachedBlockData has blockId
  • Use AtomicReference to avoid synchronized.
  private final AtomicReference<BlockData> cachedBlockData = new AtomicReference<>();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

if (blockData == null || readBlock.getLength() >= blockData.getSize() - readBlock.getOffset()) {
blockData = getBlockManager().getBlock(kvContainer, blockID);
blockFile.cacheBlockData(blockID, blockData);
} else if (!getBlockManager().blockExists(kvContainer, blockID)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blockExists check is not needed since the data was from getBlockManager().getBlock(..);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the blockExists check on a cache hit is removed

}

/** @return a cleared read buffer of at least the given capacity, reallocated only when it has to grow. */
public synchronized ByteBuffer getReadBuffer(int capacity) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

synchronized is not needed -- it must be synchronized externally; otherwise, the buffer is shared between two threads.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@ss77892

ss77892 commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

@szetszwo Thank you for the review! I've updated PR with the suggested changes. It should be even slighly faster now.

Comment on lines +2368 to +2369
// Reused across the stream's requests: gRPC serializes the response inside onNext, before the buffer is refilled.
final ByteBuffer buffer = blockFile.getReadBuffer(range.length());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could still manage the ownership of the read buffer in readblockImpl instead of controlling it in RandomAccessFileChannel. as it doesnt have any relationship with RandomAccessFileChannel

and #12042 should allow us to remove the code that is trying to reuse the read buffer here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @peterxcli for looking into it. I agree the buffer has no business in RandomAccessFileChannel. But readBlockImpl can't own it. It runs once per request on a KeyValueHandler that all streams share. So a buffer kept across requests has to live in something per stream. blockFile was the only per-stream object reaching the handler, which is why it ended up there.
On #12042: it does overlap. Its min(responseSize, end - offset) sizing is what this PR does with getReadBuffer(range.length()), so I'll rebase on it once it's merged and drop that part from here. #12042 still allocates buffer of at least one checksum interval (64 KiB) per request, though. For large sequential requests that's spread over many responses and doesn't matter. For small positioned reads (e.g. HBase random gets at tens of thousands per second) it's still GB/s of allocation per datanode. I've checked #12042 alone vs #12042 plus reuse and I see a better result with reuse.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @peterxcli for looking into it. I agree the buffer has no business in RandomAccessFileChannel. But readBlockImpl can't own it. It runs once per request on a KeyValueHandler that all streams share. So a buffer kept across requests has to live in something per stream. blockFile was the only per-stream object reaching the handler, which is why it ended up there.

https://github.com/apache/ozone/pull/11873/changes#r4578860227 could be one of the solution?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that should be one. Please check the latest changes.

Comment on lines +71 to +75
public BlockData getCachedBlockData(BlockID blockID) {
final BlockData cached = cachedBlockData.get();
return cached != null && cached.getBlockID().getContainerBlockID().equals(blockID.getContainerBlockID())
&& cached.getBlockCommitSequenceId() >= blockID.getBlockCommitSequenceId() ? cached : null;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should manage the block data cache in GrpcXceiverService#send

@Override
public StreamObserver<ContainerCommandRequestProto> send(
StreamObserver<ContainerCommandResponseProto> responseObserver) {
return new StreamObserver<ContainerCommandRequestProto>() {
private final AtomicBoolean isClosed = new AtomicBoolean(false);
private final RandomAccessFileChannel blockFile = new RandomAccessFileChannel();

because the block data is actually along with RandomAccessFileChannel blockFile

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, this state belongs with the stream, not the file wrapper. Didn't want to add a new parameters through HddsDispatcher/Handler. Let me think how to better handle it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a new class to enclapse block data and block file, replace the RandomAccessFileChannel with it in HddsDispatcher/Handler.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants