how to decode_and_store_arg a variable size argument

[p-selini]

We have an object which has a dynamic appendage like msgpack after the fix portion of the class (think of it as Class A + msgpack, where Class A has a field indicating length of the msgpack), I am trying to write Codec. compute_encoded_size is relative easy since we know the size (sizeof A + length of msgpack). for decode_arg I can't return the class A as value since my object is bigger so I just return the buffer cast to A, in decode_and_store_arg, I can't seem to find a way to push an arg bigger than A so that ostreamformatter can look pass the the size. Any other way to encode a dynamic size class?

[odygrd]

Hey, could you share a minimal example, including the type definition of the class you're trying to log and the exact log statement?

Just want to make sure I create a precise example rather than guessing across multiple cases.

For example, is it something like this?

Also the payload i assume it consists only of printable ascii characters?

struct Msg
{
    Header header;
    uint8_t* payload;
};

LOG_INFO(l, “{}”, msg);

[p-selini]

Sure, I wrote a very watered down version to illustrate:
godbolt link here to show idea

[odygrd]

try something like this, let me know if it helps ..

https://godbolt.org/z/cP6GWzavf

You might want to add a std::launder to reinterpret cast in decode but a more portable way would be to memcpy each payload to a std::vector and return the std::vector instead of the span, since memcpy deals better with misaligned reads. Alternatively you could leave some gap in the buffer after the size in encode and start the payload write from alignof payload and then do the same during decoding

decode runs in backend thread so latency there not too important

[p-selini]

Two issues remaining though. First is you treat dynamic portion as a vector but missing the fixed part, the idea is first print out the fixed portion then followed by the dynamic part. My original code has two << overload, first one is print fix part then loop to print dynamic part.

Second issue is more of a question: what if dynamic also has nested dynamic part (less important but a composable design is nice)?

[odygrd]

Regarding the second question, supporting nested types is possible but might be time-consuming to implement if not strictly needed. To implement it, you’d define Codec<NestedType>, then Codec<Payload> using Codec<NestedType>, and finally Codec<Foo> using Codec<Payload>.

This approach is already supported for STL types in the library, for example:
StdVectorLoggingTest.cpp#L179

And example of potential nested codec calls can be seen here:
Vector.h#L70

I assumed you were only interested in printing the dynamic part since Foo was doing so in the example.

I've updated the example to print both parts as intended. Please have a look here

https://godbolt.org/z/45cvdnev5

#include "quill/Backend.h"
#include "quill/Frontend.h"
#include "quill/LogMacros.h"
#include "quill/Logger.h"
#include "quill/sinks/ConsoleSink.h"

#include <cstddef>
#include <string>
#include <utility>
#include <span>

#include "quill/core/Codec.h"
#include "quill/core/DynamicFormatArgStore.h"

#include "quill/bundled/fmt/format.h"
#include "quill/bundled/fmt/ostream.h"

// to format span
#include "quill/bundled/fmt/ranges.h"

struct Foo
{
  bool b;
  char foo[5] = "test";
  size_t num{0};  // total number of Payload
  int dynamic[0]; // dynamic part from here...
};

struct Payload
{
  // the dynamic portion appending after the fixed part
  double d;
};

template <>
struct fmtquill::formatter<Foo> : fmtquill::ostream_formatter {};

template <>
struct fmtquill::formatter<Payload> : fmtquill::ostream_formatter
{
};

std::ostream& operator<<(std::ostream& os, Payload const& p)
{
  os << p.d;
  return os; // do output here...
}

std::ostream& operator<<(std::ostream& os, const Foo & c) {
  os << c.b << " ";
  os << c.foo;
  return os;
}

std::byte* align_pointer(void* pointer, size_t alignment) noexcept
{
  return reinterpret_cast<std::byte*>((reinterpret_cast<uintptr_t>(pointer) + (alignment - 1ul)) &
                                      ~(alignment - 1ul));
}

template <>
struct quill::Codec<Foo>
{
  static size_t compute_encoded_size(detail::SizeCacheVector&, Foo const& f) noexcept
  {
    // compute the dynamic size plus storage for num
    return sizeof(Foo) + alignof(Payload) + f.num * sizeof(Payload);
  }

  static void encode(std::byte*& buffer, detail::SizeCacheVector const&,
                     uint32_t&, Foo const& f) noexcept
  {
    std::memcpy(buffer, &f, sizeof(f));

    // now copy the dynamic part
    std::byte* buffer_payload = align_pointer(buffer + sizeof(f), alignof(Payload));
    size_t const dyn_bytes = f.num * sizeof(Payload);
    std::memcpy(buffer_payload, &f.dynamic[0], dyn_bytes);

    buffer += sizeof(Foo) + alignof(Payload) + dyn_bytes;
  }

  static auto decode_arg(std::byte*& buffer)
  {
    Foo foo;
    std::memcpy(&foo, buffer, sizeof(foo));

    std::byte* buffer_payload = align_pointer(buffer + sizeof(foo), alignof(Payload));
    std::span<Payload> payload_view {std::launder(reinterpret_cast<Payload*>(buffer_payload)), foo.num};

    buffer += sizeof(Foo) + alignof(Payload) + foo.num * sizeof(Payload);

    return std::make_pair(foo, payload_view);
  }

  static void decode_and_store_arg(std::byte*& buffer, DynamicFormatArgStore* args_store)
  {
    args_store->push_back(decode_arg(buffer));
  }
};

// test program
void test_log()
{
  quill::Backend::start(quill::BackendOptions{});
  auto console_sink = quill::Frontend::create_or_get_sink<quill::ConsoleSink>("sink_id_1");
  quill::Logger* logger = quill::Frontend::create_or_get_logger("root", std::move(console_sink));

  char buffer[256];
  auto f = new (buffer)(Foo);
  f->b = true;
  f->num = 4;

  char* ptr = buffer + sizeof(Foo);
  auto b = new (ptr) Payload;
  b->d = 1.0;

  char* ptr2 = buffer + sizeof(Foo) + sizeof(Payload);
  auto b2 = new (ptr2) Payload;
  b2->d = 20.0;

  char* ptr3 = buffer + sizeof(Foo) + sizeof(Payload) + sizeof(Payload);
  auto b3 = new (ptr3) Payload;
  b3->d = 120.0;

  char* ptr4 = buffer + sizeof(Foo) + sizeof(Payload) + sizeof(Payload) + sizeof(Payload);
  auto b4 = new (ptr4) Payload;
  b4->d = 400.0;

  LOG_INFO(logger, "{}", *f);
  LOG_INFO(logger, "{}", *f);
  LOG_INFO(logger, "{}", *f);
}

int main()
{
  test_log();
  return 0;
}

[p-selini]

Brilliant, I got it. Looks like just wrap around tuple like structure and view can go long way for composite structures.

[odygrd]

You are welcome! For composite structures yes or alternatively you could create your own type and format them in a different way if you don’t like the way libfmt is formatting pair and tuple.

in the example i added also the alignment for demonstration but up to you if you want to keep it usually x86 is fine with misaligned reads

[p-selini]

I know I accept the answer which does solve my question. But I come with som more wrinkles in reality. In our existing code base we do loop payload (I showed in my first example) this way to output:

std::ostream& operator<<(std::ostream& os, const Foo & c) {
    os << c.num_elements  << "=[";
    for (auto i = 0uz; i < c.num; ++i ) {//going through each dynamic part output
        os << c.dynamic[i];
    }
  os << "]"
    return os;
}

This means 1) remove such code everywhere from the code base, 2) if everywhere people do std::cout << c <"\" they will have to change their code to do explicit loop since the << overload is unfortunately global.

My feeling the solution probably is around arg_store->push_back where you can specify an object size (so we will work on a heap memory instead stack)?

 arg_store->push_back_with_size( foo, foo.length);

[odygrd]

hey, you can do something like this, it should work

#include "quill/Backend.h"
#include "quill/Frontend.h"
#include "quill/LogMacros.h"
#include "quill/Logger.h"
#include "quill/sinks/ConsoleSink.h"

#include <iostream>
#include <cstddef>
#include <string>
#include <utility>
#include <span>

#include "quill/core/Codec.h"
#include "quill/core/DynamicFormatArgStore.h"

#include "quill/bundled/fmt/format.h"
#include "quill/bundled/fmt/ostream.h"

// to format span
#include "quill/bundled/fmt/ranges.h"

struct Foo
{
  bool b;
  char foo[5] = "test";
  size_t num{0};  // total number of Payload
  int dynamic[0]; // dynamic part from here...
};

struct Payload
{
  // the dynamic portion appending after the fixed part
  double d;
};

struct FooView
{
  Foo* view;
};

template <>
struct fmtquill::formatter<Foo> : fmtquill::ostream_formatter {};

template <>
struct fmtquill::formatter<Payload> : fmtquill::ostream_formatter
{
};

template <>
struct fmtquill::formatter<FooView> : fmtquill::ostream_formatter
{
};

std::ostream& operator<<(std::ostream& os, Payload const& p)
{
  os << p.d;
  return os; // do output here...
}

std::ostream& operator<<(std::ostream& os, const Foo & c) {
  os << c.b << " ";
  os << c.foo;

  os << c.num  << " =[";
  for (size_t i = 0; i < c.num; ++i ) {//going through each dynamic part output
    os << reinterpret_cast<Payload const*>(c.dynamic)[i]  << " ";
  }
  os << "]";

  return os;
}

std::ostream& operator<<(std::ostream& os, FooView const& f)
{
  os << *f.view;
  return os; // do output here...
}

template <>
struct quill::Codec<Foo>
{
  static size_t compute_encoded_size(detail::SizeCacheVector&, Foo const& f) noexcept
  {
    // compute the dynamic size plus storage for num
    return sizeof(Foo) + f.num * sizeof(Payload);
  }

  static void encode(std::byte*& buffer, detail::SizeCacheVector const&,
                     uint32_t&, Foo const& f) noexcept
  {
    std::memcpy(buffer, &f, sizeof(f));
    buffer += sizeof(f);

    // now copy the dynamic part
    size_t const dyn_bytes = f.num * sizeof(Payload);
    std::memcpy(buffer, &f.dynamic[0], dyn_bytes);

    buffer += dyn_bytes;
  }

  static auto decode_arg(std::byte*& buffer)
  {
    FooView fv;
    fv.view = reinterpret_cast<Foo*>(buffer);
    buffer += sizeof(Foo) + fv.view->num * sizeof(Payload);
    return fv;
  }

  static void decode_and_store_arg(std::byte*& buffer, DynamicFormatArgStore* args_store)
  {
    args_store->push_back(decode_arg(buffer));
  }
};

// test program
void test_log()
{
  quill::Backend::start(quill::BackendOptions{});
  auto console_sink = quill::Frontend::create_or_get_sink<quill::ConsoleSink>("sink_id_1");
  quill::Logger* logger = quill::Frontend::create_or_get_logger("root", std::move(console_sink));

  char buffer[256];
  auto f = new (buffer)(Foo);
  f->b = true;
  f->num = 4;

  char* ptr = buffer + sizeof(Foo);
  auto b = new (ptr) Payload;
  b->d = 1.0;

  char* ptr2 = buffer + sizeof(Foo) + sizeof(Payload);
  auto b2 = new (ptr2) Payload;
  b2->d = 20.0;

  char* ptr3 = buffer + sizeof(Foo) + sizeof(Payload) + sizeof(Payload);
  auto b3 = new (ptr3) Payload;
  b3->d = 120.0;

  char* ptr4 = buffer + sizeof(Foo) + sizeof(Payload) + sizeof(Payload) + sizeof(Payload);
  auto b4 = new (ptr4) Payload;
  b4->d = 400.0;

  std::cout << f << std::endl;
  LOG_INFO(logger, "{}", *f);
  LOG_INFO(logger, "{}", *f);
  LOG_INFO(logger, "{}", *f);
}

int main()
{
  test_log();
  return 0;
}

[p-selini]

Nice. This very close to my original idea that zero copy and generically handles all nested type issues. I'm almost tempted to implicitly create the view class so that user doesn't have to, till I see I can't get away hide the view's fmtquill specialization. But this will do(no downstream change needed any more).

static auto & decode_arg(std::byte*& buffer){
    return *std::bit_cast<Foo*>(buffer);//can I just pass a reference back
              // so that I can do zero copy buffer, without heap allocation
  }