Hopefully useful for you!
Originally, if you're using std::vector as a serialize field, you must get memory copying, it's annoying.
And for some cases (game engine assets etc.) you definitely don't want any copying occured.
and if you would like to free the copied vector memory, it's typically hard in C++, because standard not say anything about how compliers should free the memory.
in my cases, a 70mb asset file, deserialize it directly eat my over 120mb memory, it's unacceptable, it can direcly use original file buffer.
So I dive into BufferAdapter and Extension, I found it actually very easy to eliminate it copying behavior.
The only things you need, is a custom adapter and an extension.
with caution : There is only be single alignment value available for all the noncopyable buffers(otherwise alignment value is multiple of others buffer like 32, 16, 8, 4), this might need some safety improvements(currently it's allow each buffer to say a wanted alignment)
The deserialization buffer(your passed into deserialize), MUST BE ALIGNED AS SAME AS noncopyable buffer, otherwise nothing will actually aligned.
Usage :
template <typename S>
void serialize(S& s)
{
s.ext(ProcessedAssetData, bitsery::ext::BufferDataNoCopying<uint32_t>(ProcessedAssetDataSize,32));
s.ext(SerializationData, bitsery::ext::BufferDataNoCopying<uint32_t>(SerializationDataSize,32));
s.text1b(FactoryName,40);
s.text1b(AssetName,1023);
}
namespace bitsery {
namespace ext {
template <typename TSize>
class BufferDataNoCopying
{
public:
explicit BufferDataNoCopying(TSize& size, uint16_t alignment):_size{size},_alignment(alignment) {}
template<typename Ser, typename T, typename Fnc>
void serialize(Ser &ser, const T &obj, Fnc &&fnc) const {
auto& writer = ser.adapter();
writer.template writeBytes<4>(static_cast<uint32_t>(_size));
auto dummyNeed = writer.currentWritePos() % _alignment;
if (dummyNeed != 0)
{
dummyNeed = _alignment - dummyNeed;
// Because we need to copy these meaningless data, so just point to an random address and copy these data.
// Avoid any extra allocation.
writer.template writeBuffer<1>(((const uint8_t*)&writer), dummyNeed);
}
writer.template writeBuffer<1>(static_cast<const uint8_t*>(obj), _size);
}
template<typename Des, typename T, typename Fnc>
void deserialize(Des &des, T &obj, Fnc &&fnc) const {
auto& reader = des.adapter();
reader.template readBytes<4>(_size);
auto dummyNeed = reader.currentReadPos() % _alignment;
if (dummyNeed != 0)
{
dummyNeed = _alignment - dummyNeed;
//Skip the meaningless.
reader.currentReadPos(reader.currentReadPos() + dummyNeed);
}
reader.readBufferNoCopying((obj), _size);
}
private:
TSize& _size;
uint16_t _alignment;
};
}
namespace traits {
template<typename TSize, typename T>
struct ExtensionTraits<ext::BufferDataNoCopying<TSize>, T>
{
using TValue = void;
static constexpr bool SupportValueOverload = false;
static constexpr bool SupportObjectOverload = true;
static constexpr bool SupportLambdaOverload = false;
};
}
}
Hopefully useful for you!
Originally, if you're using std::vector as a serialize field, you must get memory copying, it's annoying.
And for some cases (game engine assets etc.) you definitely don't want any copying occured.
and if you would like to free the copied vector memory, it's typically hard in C++, because standard not say anything about how compliers should free the memory.
in my cases, a 70mb asset file, deserialize it directly eat my over 120mb memory, it's unacceptable, it can direcly use original file buffer.
So I dive into BufferAdapter and Extension, I found it actually very easy to eliminate it copying behavior.
The only things you need, is a custom adapter and an extension.
with caution : There is only be single alignment value available for all the noncopyable buffers(otherwise alignment value is multiple of others buffer like 32, 16, 8, 4), this might need some safety improvements(currently it's allow each buffer to say a wanted alignment)
The deserialization buffer(your passed into deserialize), MUST BE ALIGNED AS SAME AS noncopyable buffer, otherwise nothing will actually aligned.
Usage :