Repository navigation
HDDS-16377. Datanode ReadBlock streaming path allocates and copies a fresh buffer for every request - #11873
HDDS-16377. Datanode ReadBlock streaming path allocates and copies a fresh buffer for every request#11873ss77892 wants to merge 4 commits into
Conversation
…fresh buffer for every request Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
@szetszwo @chungen0126 @peterxcli would you mind take a look? |
| private BlockID cachedBlockID; | ||
| private BlockData cachedBlockData; |
There was a problem hiding this comment.
- cachedBlockID is not needed since cachedBlockData has blockId
- Use AtomicReference to avoid synchronized.
private final AtomicReference<BlockData> cachedBlockData = new AtomicReference<>();
| if (blockData == null || readBlock.getLength() >= blockData.getSize() - readBlock.getOffset()) { | ||
| blockData = getBlockManager().getBlock(kvContainer, blockID); | ||
| blockFile.cacheBlockData(blockID, blockData); | ||
| } else if (!getBlockManager().blockExists(kvContainer, blockID)) { |
There was a problem hiding this comment.
blockExists check is not needed since the data was from getBlockManager().getBlock(..);
There was a problem hiding this comment.
the blockExists check on a cache hit is removed
| } | ||
|
|
||
| /** @return a cleared read buffer of at least the given capacity, reallocated only when it has to grow. */ | ||
| public synchronized ByteBuffer getReadBuffer(int capacity) { |
There was a problem hiding this comment.
synchronized is not needed -- it must be synchronized externally; otherwise, the buffer is shared between two threads.
|
@szetszwo Thank you for the review! I've updated PR with the suggested changes. It should be even slighly faster now. |
| // Reused across the stream's requests: gRPC serializes the response inside onNext, before the buffer is refilled. | ||
| final ByteBuffer buffer = blockFile.getReadBuffer(range.length()); |
There was a problem hiding this comment.
I think we could still manage the ownership of the read buffer in readblockImpl instead of controlling it in RandomAccessFileChannel. as it doesnt have any relationship with RandomAccessFileChannel
and #12042 should allow us to remove the code that is trying to reuse the read buffer here
There was a problem hiding this comment.
Thanks @peterxcli for looking into it. I agree the buffer has no business in RandomAccessFileChannel. But readBlockImpl can't own it. It runs once per request on a KeyValueHandler that all streams share. So a buffer kept across requests has to live in something per stream. blockFile was the only per-stream object reaching the handler, which is why it ended up there.
On #12042: it does overlap. Its min(responseSize, end - offset) sizing is what this PR does with getReadBuffer(range.length()), so I'll rebase on it once it's merged and drop that part from here. #12042 still allocates buffer of at least one checksum interval (64 KiB) per request, though. For large sequential requests that's spread over many responses and doesn't matter. For small positioned reads (e.g. HBase random gets at tens of thousands per second) it's still GB/s of allocation per datanode. I've checked #12042 alone vs #12042 plus reuse and I see a better result with reuse.
There was a problem hiding this comment.
Thanks @peterxcli for looking into it. I agree the buffer has no business in RandomAccessFileChannel. But readBlockImpl can't own it. It runs once per request on a KeyValueHandler that all streams share. So a buffer kept across requests has to live in something per stream. blockFile was the only per-stream object reaching the handler, which is why it ended up there.
https://github.com/apache/ozone/pull/11873/changes#r4578860227 could be one of the solution?
There was a problem hiding this comment.
Yes, that should be one. Please check the latest changes.
| public BlockData getCachedBlockData(BlockID blockID) { | ||
| final BlockData cached = cachedBlockData.get(); | ||
| return cached != null && cached.getBlockID().getContainerBlockID().equals(blockID.getContainerBlockID()) | ||
| && cached.getBlockCommitSequenceId() >= blockID.getBlockCommitSequenceId() ? cached : null; | ||
| } |
There was a problem hiding this comment.
I think we should manage the block data cache in GrpcXceiverService#send
because the block data is actually along with RandomAccessFileChannel blockFile
There was a problem hiding this comment.
Yep, this state belongs with the stream, not the file wrapper. Didn't want to add a new parameters through HddsDispatcher/Handler. Let me think how to better handle it.
There was a problem hiding this comment.
add a new class to enclapse block data and block file, replace the RandomAccessFileChannel with it in HddsDispatcher/Handler.
What changes were proposed in this pull request?
On the datanode, a streaming ReadBlock is served over one gRPC stream per block, through one
RandomAccessFileChannelthatGrpcXceiverServicekeeps open for the stream. Every ReadBlock request on that stream used to:BlockDatain the container DB viaBlockManager.getBlock,ByteBuffersized by the client-chosen response size, andByteString.copyFrom.For a sequential read of a large block this is one DB lookup, one allocation and two copies per response. This PR removes the per-request allocation and the extra copy, and caches the block metadata for the life of the stream:
RandomAccessFileChannel.getReadBuffer(capacity)keeps one buffer per stream, reallocates it only when a larger one is needed, and drops it onclose(). The buffer is now sized by the range actually being read rather than by the response size the client asked for.ContainerCommandResponseBuilders.getReadBlockResponsetakes the handler'sbyteBufferToByteStringconversion, asgetReadChunkResponsealready does. Withozone.UnsafeByteOperations.enabled(defaulttrue) the response wraps the buffer instead of copying it. Reusing the buffer is safe because gRPC serializes the response insideStreamObserver.onNext, before the next range refills the buffer; this contract is documented ongetReadBlockResponse.BlockDataper stream.RandomAccessFileChannelcaches theBlockDataof the block it serves and clears it onclose().BlockManager.getBlock, which keeps itsBCSID_MISMATCHcheck.INVALID_ARGUMENTbefore the cache check, closing the block file like the existingOUT_OF_RANGErejection.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16377
How was this patch tested?
New unit tests:
TestRandomAccessFileChannelblockDataIsCachedPerBlockAndClearedOnClose: cache hit perBlockID, miss for another block or BCSID, cleared on close.readBufferIsReusedGrownAndDroppedOnClose: buffer reused and cleared, reallocated only to grow, dropped on close.TestKeyValueHandlertestReadBlockReadBufferIsSizedByTheRequest: a large client response size does not size the buffer.testReadBlockLengthBeyondTheBlockIsClamped:Long.MAX_VALUElength, aligned and misaligned offset.testReadBlockCachedBlockDataIsReusedWithinTheBlock: onegetBlockfor two requests within the block.testReadBlockCachedBlockDataIsRefreshedWhenTheBlockGrows: a request past the cached size sees the grown block.testReadBlockCachedBlockDataBoundsTheReadWhenTheBlockGrows: a request inside the cached size does not read past it.testReadBlockErrorOnTheStreamClosesTheBlockFile: extended with negative offset and negative length.Benchmark report vs master (3 node cluster using LXD containers)
build_comparison_20261030_204023.pdf