Skip to content
This repository has been archived by the owner on Apr 15, 2024. It is now read-only.

ISSUE-2064: DirectMemoryCRC32Digest.update(ByteBuf) allocates lots of byte[] #164

Open
sijie opened this issue Jan 15, 2020 · 0 comments
Open

Comments

@sijie
Copy link
Member

sijie commented Jan 15, 2020

Original Issue: apache#2064


FEATURE REQUEST

  1. Please describe the feature you are requesting.
    Profiling shows the code block in bold is hit every time an entry is read from bookie.
    @Override
    public void update(ByteBuf buf) {
        int index = buf.readerIndex();
        int length = buf.readableBytes();

        try {
            if (buf.hasMemoryAddress()) {
                // Calculate CRC directly from the direct memory pointer
                crcValue = (int) updateByteBuffer.invoke(null, crcValue, buf.memoryAddress(), index, length);
            } else if (buf.hasArray()) {
                // Use the internal method to update from array based
                crcValue = (int) updateBytes.invoke(null, crcValue, buf.array(), buf.arrayOffset() + index, length);
            } else {
                **// Fallback to data copy if buffer is not contiguous
                byte[] b = new byte[length];
                buf.getBytes(index, b, 0, length);
                crcValue = (int) updateBytes.invoke(null, crcValue, b, 0, b.length);**
            }
        } catch (IllegalAccessException | InvocationTargetException e) {
            throw new RuntimeException(e);
        }
    }

This is due to ByteBuf being marked as read only (as a result buf.hasArray() == false) from

ReadCompletion
public void handleV3Response(BookkeeperProtocol.Response response) {
            readEntryOutstanding.dec();
            ReadResponse readResponse = response.getReadResponse();
            StatusCode status = response.getStatus() == StatusCode.EOK
                ? readResponse.getStatus() : response.getStatus();
            ByteBuf buffer = Unpooled.EMPTY_BUFFER;
            if (readResponse.hasBody()) {
                **buffer = Unpooled.wrappedBuffer(readResponse.getBody().asReadOnlyByteBuffer());**
            }
  1. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have).
    Are you currently using any workarounds to address this issue?

This generates lots of garbage. See attached image. Would be nice to get your thoughts on this issue, maybe using a buffer pool.
bk_digest_memory

  1. Provide any additional detail on your proposed use case for this feature.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant