Can encoders append to std::string instead of overwriting?

[emptyVoid]

The following code overwrites string's contents:

    std::string str = "FOO";
    std::vector<uint8_t> data = {1,2,3,4,5,6,7,8,9,10};
    cppcodec::base64_rfc4648::encode( str, data );

Can encoder be instructed to append encoded result?

[jpetso]

Yes, that would be possible by specializing the data access functions from cppcodec/data/access.hpp. The clearest / most straightforward option in that regard is probably by wrapping the string in another object, and implementing the standard result container methods that are used by default.

cppcodec/data/raw_result_buffer.hpp has such an implementation, if you have the option to wrap the type before passing it to encode() then I'd something similar.

I'll look into writing up an example.

[jpetso]

I created an example app and added it to my current pull request, to go into a new example/ folder. Until that's merged, check it out as part of this commit in my fork and let me know if that works for you.

[emptyVoid]

Thanks!

Though, IMO support for std::back_inserter would be more elegant/C++ish and wouldn't require code duplication (such as init() and finish() must be declared in the cppcodec::data namespace) for applications that do not require performance quirks to generate gigabytes of encoded strings per second.

[emptyVoid]

And to the sad part: ATL's Base64Encode works 2.5 times faster than cppcodec::base64_rfc4648::encode.

Update: debug build is horrendously slow - works 40+ times slower than Base64Encode.

[jpetso]

I plan to add support for streams / std::back_inserter in v0.3, definitely a valid concern.

Thanks for reporting back on performance. I figure this is on master? What compiler are you using? Did you use my example verbatim or change it to use the faster C++17 direct data access for std::string as mentioned in the comments of the example?

Sorry about the poor debug build performance, I fear that this is an inherent downside of the generic / template-based approach. I spent a decent amount of energy on optimizing the release build and almost got pretty far on most compilers, but debug builds are a lost cause as I can't get the compiler to optimize things out. This is where simplicity has a definitive edge, but that's not what cppcodec is trying to do. I think it may be a good idea to add a warning about this to the README.

[emptyVoid]

Yes, I've run performance comparison on yesterday's master using Visual C++ 2017 (15.7.5). And just using the simplest version str = cppcodec::base64_rfc4648::encode( data ); with 1MB and 1GB data sizes.

[jpetso]

Thanks, that's more or less congruent with my findings from 7ef02f1 where GCC optimizes optimally and beats both MSVC and Clang by about a factor of 3 on release builds. I've put a lot of effort into getting close to the best-performing implementations, but I've mostly run out of ideas of how to get Clang and MSVC to produce simpler machine code even with templates involved. Here's to hoping they'll do a better job over time.

For better or worse, generalized templates is the direction that cppcodec has picked, so my main responsibility is documenting the trade-offs in order to avoid surprises by users. I'll close the issue and merge the new example in a bit. Thanks for the inspiring exchange!