Add SingleIOBuf for efficient flatbuffers serialization/deserialization by Q1ngbo · Pull Request #3062 · apache/brpc

Q1ngbo · 2025-08-11T11:18:31Z

What problem does this PR solve?

Issue Number:
#2354 #978

Problem Summary:
Hi, We plan to migrate the flatbuffers protocol used internally by Baidu to the community, which is expected to be divided into the following three steps:

Submit the data structure SingleIOBuf, which is used for Flatbuffers serialization and deserialization, to the apache/brpc repository;
Submit message construction and serialization/deserialization methods to the google/flatbuffers repository;
Add Flatbuffers-based RPC handling policy to apache/brpc.
These steps have a sequential dependency, and this commit completes the first step.
Flatbuffers requires that data to be serialized/deserialized be stored in a contiguous memory buffer. To meet this requirement, we designed SingleIOBuf. Unlike brpc’s existing IOBuf, SingleIOBuf uses a single Block to hold contiguous data, which better aligns with Flatbuffers' memory layout requirements. We also defined corresponding memory allocation and release interfaces to support efficient management of SingleIOBuf.

What is changed and the side effects?

Changed:

Side effects:

Performance effects:
Breaking backward compatibility:

Check List:

Please make sure your changes are compilable.
When providing us with a new feature, it is best to add related tests.
Please follow Contributor Covenant Code of Conduct.

Copilot

Pull Request Overview

This PR introduces a new SingleIOBuf class designed to provide efficient flatbuffers serialization/deserialization by maintaining data in a contiguous single memory block, unlike the existing IOBuf which can use multiple blocks.

Adds SingleIOBuf class with contiguous memory management for flatbuffers integration
Implements memory allocation, deallocation, and reallocation methods specific to single-block operations
Provides conversion methods between SingleIOBuf and IOBuf for interoperability

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
src/butil/iobuf.h	Declares the new `SingleIOBuf` class with public interface and friend declarations
src/butil/iobuf.cpp	Implements all `SingleIOBuf` methods including memory management and data operations
test/iobuf_unittest.cpp	Adds comprehensive unit tests covering `SingleIOBuf` functionality

src/butil/iobuf.h

Copilot · 2025-08-12T04:39:43Z

src/butil/iobuf.cpp

+                                        uint32_t in_use_front) {
+    IOBuf::BlockRef& ref = _cur_ref;
+    if (BAIDU_UNLIKELY(new_size <= ref.length)) {
+        LOG(ERROR) << "invaild new size:" << new_size;


Typo in error message: 'invaild' should be 'invalid'.

Suggested change

LOG(ERROR) << "invaild new size:" << new_size;

LOG(ERROR) << "invalid new size:" << new_size;

Copilot · 2025-08-12T04:39:43Z

src/butil/iobuf.cpp

+        _cur_ref.block->inc_ref();
+        return true;
+    }
+    return false;


This return statement is unreachable code. The function will always return from either the true or false branch of the preceding if-else statement.

Suggested change

return false;

Copilot · 2025-08-12T04:39:43Z

test/iobuf_unittest.cpp


+TEST_F(IOBufTest, single_iobuf) {
+    butil::IOBuf buf1;
+    char *usr_str = (char *)malloc(16);


Memory allocated with malloc() is not freed in this test, causing a memory leak. Consider using RAII or explicitly calling free().

Copilot · 2025-08-12T04:39:43Z

src/butil/iobuf.cpp

+
+void SingleIOBuf::deallocate(void* p) {
+    if (_cur_ref.block) {
+        if (_cur_ref.block->data + _cur_ref.offset == (char*)p) { 


[nitpick] The deallocate method only resets if the pointer matches exactly the beginning of the allocated region. This seems overly restrictive and may not handle all valid deallocation scenarios correctly.

Suggested change

if (_cur_ref.block->data + _cur_ref.offset == (char*)p) {

char* start = _cur_ref.block->data + _cur_ref.offset;

char* end = start + _cur_ref.length;

char* ptr = static_cast<char*>(p);

if (ptr >= start && ptr < end) {

src/butil/iobuf.cpp

src/butil/iobuf.h

zhangqiongyu · 2025-08-12T12:02:46Z

Very useful feature for me!

src/butil/iobuf.h

yanglimingcn · 2025-08-13T08:08:22Z

src/butil/iobuf.cpp

+    return false;
+}
+
+void SingleIOBuf::append_to(IOBuf* buf) const {


In RPC scenarios, data length is dynamic, while SingleIOBuf typically manages a fixed-size memory block. would you please explain how to use it in PRC parse message and pack message？
for example, when receive a message we don't know the length of this message, how to allocate memory, if memory is short, we need to reallocate and need memcopy?

I'm also curious about the implementation here.

在flatbuffers中用户使用fbs文件定义message格式（类似于proto）。所以在用户应用程序中创建请求时可以按消息大小分配内存。构造出的消息内容保存在SingleIOBuf中，最终它会通过SingleIOBuf::append_to添加到IOBuf中发送出去。brpc接收到的消息同样保存在IOBuf中。因此在解析请求时，将调用SingleIOBuf::assign将IOBuf assign给SingleIOBuf, 如果IOBuf的单个Block能容纳整个message，则直接将该Block赋值给SingleIOBuf，否则会根据消息大小重新申请足够大的Block，再从传入的IOBuf中拷贝数据。

RPC message的Serialize与Deserialize过程将在flatbuffers中实现。

所以接收消息这块可能会有拷贝，这块最佳实践是评估请求大小，最好是一个Block能容纳得下。

所以接收消息这块可能会有拷贝，这块最佳实践是评估请求大小，最好是一个Block能容纳得下。

是的，当前单个Block的默认大小是8KB（由DEFAULT_BLOCK_SIZE指定）。

Co-authored-by: lishuo02<lishuo02@baidu.com> Co-authored-by: xulei25 <xulei25@baidu.com>

src/butil/single_iobuf.cpp

test/iobuf_unittest.cpp

src/butil/single_iobuf.cpp

yanglimingcn · 2025-08-15T02:08:13Z

src/butil/single_iobuf.cpp

+SingleIOBuf& SingleIOBuf::operator=(const SingleIOBuf& rhs) {
+    reset();
+    _block_size = 0;
+    if (rhs._cur_ref.block != NULL) {


Is it reasonable to set _block_size=0 after assignment? use swap function?

Yes, _block_size is used to determine whether the block should be released.

yanglimingcn · 2025-08-15T02:09:42Z

src/butil/single_iobuf.h

+    IOBuf::Block* alloc_block_by_size(uint32_t data_size);
+
+private:
+    // Current block the SingleIOBuf used.


is this only one block in SingleIOBuf？

Yes, That's why it is called "Single"

src/butil/single_iobuf.cpp

ivanallen · 2025-08-15T03:26:15Z

src/butil/single_iobuf.cpp

+
+int SingleIOBuf::assign_user_data(void* data, size_t size, void (*deleter)(void*)) {
+    reset();
+    if (size > 0xFFFFFFFFULL - 100) {


What is the exact meaning of 100? Does it refer to the reserved area of in_use_back?

The threshold 0xFFFFFFFFULL - 100 is chosen to stay consistent with the implementation of IOBuf::append_user_data_with_meta().

src/butil/single_iobuf.cpp

src/butil/iobuf_inl.cpp

wwbmmm · 2025-08-15T04:05:55Z

src/butil/iobuf.cpp

 }

-inline IOBuf::Block* create_block(const size_t block_size) {
+IOBuf::Block* create_block(const size_t block_size) {


remove the inline keyword may affect performance, you can move the function body to iobuf_inl.h, since that file is for inline functions.

src/butil/single_iobuf.cpp

wwbmmm · 2025-08-19T12:39:58Z

LGTM

yanglimingcn · 2025-08-20T09:06:21Z

Add Flatbuffers-based RPC handling policy to apache/brpc.
These steps have a sequential dependency, and this commit completes the first step.
Flatbuffers requires that data to be serialized/deserialized be stored in a contiguous memory buffer. To meet this requirement, we designed SingleIOBuf. Unlike brpc’s existing IOBuf, SingleIOBuf uses a single Block to hold contiguous data, which better aligns with Flatbuffers' memory layout requirements. We also defined corresponding memory allocation and release interfaces to support efficient management of SingleIOBuf.
@Q1ngbo 想问一下，这个支持完了之后，bRPC是要实现一个新的协议吗？还是可以直接用baidu_std协议？

Q1ngbo · 2025-08-20T10:10:24Z

Add Flatbuffers-based RPC handling policy to apache/brpc. These steps have a sequential dependency, and this commit completes the first step. Flatbuffers requires that data to be serialized/deserialized be stored in a contiguous memory buffer. To meet this requirement, we designed SingleIOBuf. Unlike brpc’s existing IOBuf, SingleIOBuf uses a single Block to hold contiguous data, which better aligns with Flatbuffers' memory layout requirements. We also defined corresponding memory allocation and release interfaces to support efficient management of SingleIOBuf. @Q1ngbo 想问一下，这个支持完了之后，bRPC是要实现一个新的协议吗？还是可以直接用baidu_std协议？

是的，会实现一个新的协议。

chenBright · 2025-08-24T07:57:47Z

src/butil/single_iobuf.cpp

+    } else {
+        IOBuf::Block* b = alloc_block_by_size(msg_size);
+        if (!b) {
+            return false;


Is it better to ensure that the block is allocated successfully before reset?

Yes, it's meaningful. The latest commit has included this.

chenBright · 2025-08-24T08:04:18Z

src/butil/single_iobuf.cpp

+    }
+}
+
+int SingleIOBuf::assign_user_data(void* data, size_t size, void (*deleter)(void*)) {


Like IOBuf, the deleter type should be std::function, not a function pointer.

brpc/src/butil/iobuf.h

Line 250 in f949962

int append_user_data(void* data, size_t size, std::function<void(void*)> deleter);

chenBright · 2025-08-24T08:05:34Z

src/butil/single_iobuf.h

+    // Reallocates the current buffer to a larger size.
+    void* reallocate_downward(uint32_t new_size,
+                            uint32_t in_use_back,
+                            uint32_t in_use_front);


Add a comment to explain in_use_back and in_use_front.

Co-authored-by: lishuo02<lishuo02@baidu.com> Co-authored-by: xulei25 <xulei25@baidu.com> Co-authored-by: qiqingbo<qiqingbo@baidu.com>

chenBright

LGTM

yanglimingcn · 2025-08-28T02:36:21Z

LGTM

zchuango · 2025-10-27T08:21:12Z

@Q1ngbo 挺感兴趣brpc引入flatbuffers特性的，这块后续的计划是什么？目前还没看到step2和step3的pr

AnDiXL · 2026-01-15T12:30:48Z

@Q1ngbo 挺感兴趣brpc引入flatbuffers特性的，这块后续的计划是什么？目前还没看到step2和step3的pr

现在有啦

wwbmmm requested review from Copilot and yanglimingcn and removed request for yanglimingcn August 12, 2025 04:38

Copilot AI reviewed Aug 12, 2025

View reviewed changes

wwbmmm reviewed Aug 12, 2025

View reviewed changes

src/butil/iobuf.cpp Outdated Show resolved Hide resolved

src/butil/iobuf.h Outdated Show resolved Hide resolved

ivanallen reviewed Aug 13, 2025

View reviewed changes

src/butil/iobuf.h Outdated Show resolved Hide resolved

yanglimingcn reviewed Aug 13, 2025

View reviewed changes

Add SingleIOBuf for efficient flatbuffers serialization/deserialization

a271976

Co-authored-by: lishuo02<lishuo02@baidu.com> Co-authored-by: xulei25 <xulei25@baidu.com>

Q1ngbo force-pushed the feature-singleiobuf branch from 1610d89 to a271976 Compare August 14, 2025 10:49

ivanallen reviewed Aug 15, 2025

View reviewed changes

src/butil/single_iobuf.cpp Show resolved Hide resolved

ivanallen reviewed Aug 15, 2025

View reviewed changes

test/iobuf_unittest.cpp Show resolved Hide resolved

ivanallen reviewed Aug 15, 2025

View reviewed changes

src/butil/single_iobuf.cpp Show resolved Hide resolved

yanglimingcn reviewed Aug 15, 2025

View reviewed changes

src/butil/single_iobuf.cpp Show resolved Hide resolved

ivanallen reviewed Aug 15, 2025

View reviewed changes

src/butil/single_iobuf.cpp Outdated Show resolved Hide resolved

wwbmmm reviewed Aug 15, 2025

View reviewed changes

yanglimingcn reviewed Aug 15, 2025

View reviewed changes

src/butil/single_iobuf.cpp Outdated Show resolved Hide resolved

Merge branch 'apache:master' into feature-singleiobuf

1aaba4a

Q1ngbo force-pushed the feature-singleiobuf branch 2 times, most recently from 0ef1c7d to 4c148bf Compare August 19, 2025 04:54

Q1ngbo requested review from chenBright, ivanallen and yanglimingcn August 21, 2025 10:50

ivanallen approved these changes Aug 21, 2025

View reviewed changes

chenBright reviewed Aug 24, 2025

View reviewed changes

Add SingleIOBuf for efficient flatbuffers serialization/deserialization

3a7eb33

Co-authored-by: lishuo02<lishuo02@baidu.com> Co-authored-by: xulei25 <xulei25@baidu.com> Co-authored-by: qiqingbo<qiqingbo@baidu.com>

Q1ngbo force-pushed the feature-singleiobuf branch from 4c148bf to 3a7eb33 Compare August 25, 2025 08:04

chenBright approved these changes Aug 25, 2025

View reviewed changes

yanglimingcn approved these changes Aug 28, 2025

View reviewed changes

chenBright merged commit ffeb407 into apache:master Aug 28, 2025
15 checks passed

This was referenced Jan 15, 2026

flatbuffers support(2/3): Add flatbuffers message construction logic #3196

Open

flatbuffers support(3/3): Add flatbuffers protocol support for brpc #3197

Open

	LOG(ERROR) << "invaild new size:" << new_size;
	LOG(ERROR) << "invalid new size:" << new_size;

-        if (_cur_ref.block->data + _cur_ref.offset == (char*)p) {
+        char* start = _cur_ref.block->data + _cur_ref.offset;
+        char* end = start + _cur_ref.length;
+        char* ptr = static_cast<char*>(p);
+        if (ptr >= start && ptr < end) {

Conversation

Q1ngbo commented Aug 11, 2025

What problem does this PR solve?

What is changed and the side effects?

Check List:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zhangqiongyu commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

yanglimingcn Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Q1ngbo Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yanglimingcn Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wwbmmm commented Aug 19, 2025

Uh oh!

yanglimingcn commented Aug 20, 2025

Uh oh!

Q1ngbo commented Aug 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

zhangqiongyu commented Aug 12, 2025 •

edited

Loading

yanglimingcn Aug 13, 2025 •

edited

Loading

Q1ngbo Aug 13, 2025 •

edited

Loading

yanglimingcn Aug 15, 2025 •

edited

Loading