Looking for more Collaborators!

[webmakersteve]

Hi everybody.

First off, I just want to thank the community for all of the support this repository has gotten. A few years ago when I started working on what would become node-rdkafka, I never could have imagined it getting this big! It wouldn't be nearly as big without all of you - your contributions, your bug reports, and the inspiration to keep me working on it.

This project originated on Blizzard's data team. We built out kafka support in node.js because we were using node for message streaming from HTTP to kafka to be processed by our downstream services. A few months after that, we stopped using node for new projects so we started using the library less. I still supported it as best I could, and we had still had existing services that used it that were made better with every release!

About 6 months ago I moved to another team that doesn't use any node.js, and the amount of time I've had to commit on this repository has gone to its lowest ever. This would be more OK, but my free weekend-time has also been at an all time low.

So, this has all been a long way of saying I need help! I'd like to give some of the responsibility for the management of this project to some of you. If you are interested at all please email me at sparente@blizzard.com!

I'm hoping to get more active again as best I can to give this project the support it deserves. There's a lot of work to be done on it to make it even better! If that excites you, send me an email!

Thank you everyone for your continued patience. Its totally undeserved, but appreciated :)

[JaapRood]

I think I speak on behalf of a lot of people to say: 🎉thank you for all your effort so far and in the future❤️! So many people, including myself personally and my team professionally, have benefited from all of your hard work, getting this setup in the first place.

I really hope someone with experience in both C++ and Node is willing to take on a contributor role, as especially for high-throughput cases it's hard to beat binding with the excellent librdkafka. Are there any companies / teams out there (apart from Blizzard) with these skillsets that use node-rdkafka to run any production services?

Unfortunately, that excludes myself, as all my C++ experience is just reading and hacking some other people's code. We're only a 2 person indie team, so lacking the resources to address the issues the project has been facing as of late. It pushed us towards adopting a client written 100% in JS for all new stream processing tasks, but without any of the luxuries of librdkafka's throughput, streams, near-mirrored consumer API's, etc.

[rusty0412]

What kind of help do you need? Do you have a roadmap of things to improve, modernize? Do you know of bottlenecks that need to be optimized? Would you be open to dropping support for unsupported node releases (<8.0)? Would you be open to modernizing tooling (TypeScript, eslint, prettier commit hooks)?

[webmakersteve]

@rusty0412

There isn't specifically a road-map. So far there has been very little process involved in what we are upgrading or doing at any given time. Previously it was driven by what I was having to work with at the time (since I was actually using the library), but nowadays I've done very little to actually push the library forward and rely on just merging PRs that come in and providing feedback.

Here is what I'd like to see done in the future:

1. Drop all support for < 8
2. Add promise compatible APIs for async functions, or just an easy Promisify wrapper available to wrap all the method calls so they can be used with async await
3. Upgrade NAN and get rid of all of the deprecations we have. This is a pretty big effort as the async worker handling has been hugely changed.
4. A lot of C++ refactoring - the code is very old and looks it. There is a lot of stuff that can be consolidated to reduce the huge amount of sprawl.
5. Get rid of the bundled librdkafka as part of the base package in favor of having a dependency on it being provided as a shared library in the box. I can be convinced that this is a bad idea, but so many of the problems with this library deal with the fact that it is built from source at install time.
6. First-class typescript support
7. A lot of CI improvements. Maybe migrate to circle CI? But mainly I want the builder to publish the package on every master build with pre-release tags and auto publish tagged builds.
8. Better testing. Need to make a mocked version of the thin librdkafka classes and inject those around.
9. Continue implementing new features of librdkafka. There are ways we can do FD polling for more efficient consumption in streams, etc.

If this sounds interesting to you, let me know! But I can also announce that I've added @iradul, @ankon , and @codeburke as collaborators, so thank you guys for volunteering! I appreciate all the help.

[codeburke]

Looking forward to collaborating on this project @webmakersteve, @iradul, @ankon!

I'm not going to be much help in the C/C++ world, but willing to help how I can. I can definitely start chipping away on some modernization efforts like dropping <8 support, async/await/Promise support, and improvements to CI. Seems like we should open an Issue for each of these to start discussion about what we are planning on changing and let people provide feedback/PRs.

[iradul]

But I can also announce that I've added @iradul, @ankon , and @codeburke as collaborators, so thank you guys for volunteering! I appreciate all the help.

Thank you for adding us to the team!

6. First-class typescript support

Should we consider rewriting the whole project in TypeScript? I think that would bring huge benefits.

Also, it might be helpful if we start using Projects feature to manage and track progress of all the things you'd like to see done in the future.

[IdanAdar]

@webmakersteve
I think the first thing needed is a more direct line of communication; getting replies is very sporadic and puts the package at high risk. For example right now it's just broken in 2.71, but there's no quick way to reach you - since there is no one else, or very very few, who can merge and kick off new releases - so teams around the world that rely on this package must wait for long period of time. We need to communicate. Lets solve this first.

How about a gitter channel? I've created this: https://gitter.im/node-rdkafka/community
Lets join.

[codeburke]

Good to meet you Ivan! Love the idea of using Project to help coordinate bigger changes. I’d love to hear the advantages of entirely Typescript. Personally I’m not sold on “strong typing” on the internals of a project, but would love to look into how to use Typescript more consistently on all the public interfaces. Working on interface definitions and getting those linked to documentation seems like a big win for the users. I know that I’ve had to sift through the source code to understand some apis recently. Ryan

…

On Thu, Jul 25, 2019 at 2:09 PM Idan Adar ***@***.***> wrote: I think the first thing needed is a more direct line of communication; getting replies is very sporadic and puts the package at high risk. For example right now it's just broken in 2.71, but there's no quick way to reach you - since there is no one else who can merge - so teams around the world that rely on this package must wait for long period of time. We need to communicate. Lets solve this first. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#628?email_source=notifications&email_token=AA4MURHFXV53AFNLUOYVS5DQBH257A5CNFSM4HSXN6UKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD22PKXY#issuecomment-515175775>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4MURGGPWVURXKADNDPZG3QBH257ANCNFSM4HSXN6UA> .

[SoyYoRafa]

Hi,
I am interested in contributing to a TypeScript migration. This project has no need for any dynamic typing behavior and the library needs to be robust. Static type checking will drive robustness since it is a type of test. It will help contributors understand the code layout. TypeScript will allow the project to use more modern syntax than allowed by supported runtimes. It will also guarantee high quality, error-free typing declarations to library users. If it's not TypeScript native, the declarations and API will drift and become inconsistent.

[codeburke]

Forgive my naivety on Typescript for libraries. Will that mean that library users will all need to transpile? I can go along with moving to Typescript for the reasons you mentioned, as long as we support native ES6 syntax in the released assets. I don’t think that we want to require the library users to setup Typescript transpiling of their projects just to use node-rdkafka.

…

On Thu, Jul 25, 2019 at 10:24 PM Rafael Fernández ***@***.***> wrote: Hi, I am interested in contributing to a TypeScript migration. This project has no need for any dynamic typing behavior and the library needs to be robust. Static type checking will drive robustness since it is a type of test. It will help contributor understand the code layout. It will allow the project to use more modern syntax than allowed by supported runtimes. TypeScript native would also guarantee high quality, error-free typing declarations to library users. If it's not native, the declarations and API will drift and become inconsistent. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#628?email_source=notifications&email_token=AA4MURDHLC4BWRPP4ZP7RJTQBJU75A5CNFSM4HSXN6UKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD23MF3A#issuecomment-515293932>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4MURB7NJMTKBQI3KQFATTQBJU75ANCNFSM4HSXN6UA> .

[SoyYoRafa]

Will that mean that library users will all need to transpile?

No. node-rdkafka's build script would invoke the TypeScript compiler. The TypeScript compiler would then generate javascript files and type declarations. node-rdkafka would publish the javascript files and type declarations files. Library users just run the javascript files, they are unaware the project is written in TypeScript. TypeScript users also run the javascript files but will use the generated type declaration files in their code.

Additionally, you can use syntax newer than es6 in TypeScript and configure the compiler to generate es6-compatible javascript. When each version of node gets deprecated, you just bump up the target in the TypeScript compiler config and you get the native code for that target.

[JaapRood]

From my experience to contributing to node-rdkafka, I'd have to underline this mention of @webmakersteve:

3. Upgrade NAN and get rid of all of the deprecations we have. This is a pretty big effort as the async worker handling has been hugely changed.
4. A lot of C++ refactoring - the code is very old and looks it. There is a lot of stuff that can be consolidated to reduce the huge amount of sprawl.
5. Get rid of the bundled librdkafka as part of the base package in favor of having a dependency on it being provided as a shared library in the box. I can be convinced that this is a bad idea, but so many of the problems with this library deal with the fact that it is built from source at install time.

Those are the definitely the points causing a bunch of problems at the moment, the ones that have been causing stability issues thus far and threaten to do so even more in the future. The binding to librdkafka is what sets node-rdkafka apart with a unique value proposition: piggy backing of an very highly optimized native library that achieves order of magnitude higher throughput than the official Kafka client. That's the bit worth trying to save, imo.

It's not that the Javascript side couldn't do with some love, it definitely could. But if what you're looking for is a modern JS code base, Typescript support, Promise support, strong test suite and CI, I wonder if you're time is not better spent on the newer KafkaJS project, which already has all of those things and is at a high rate of further development and progress. It's what we as a tiny 2-person indie team had to move to, and since we don't have much C++/C expertise, it's 100% Node codebase makes our contributions a whole bunch easier and more productive.

That's not to say that I think node-rdkafka shouldn't be kept alive / revived, not at all! Given it's bindings to librdkafka I could see it as a simpler client that's useful if you need some really serious throughput; it could offer levels of performance hard to match with a fully Javascript-implemented client. But then the focus should at least be on that part, that unique strength. Otherwise, all we're doing is fragmenting the already small Node + Kafka ecosystem even further and just duplicating efforts :(

[codeburke]

Speaking for myself, the reason I focused on the JS portion was because that is where my skillset primarily resides and where I feel I could provide immediate progress. But I do 100% agree that the differentiator here is the use of `librdkafka` as the performant underlying implementation. As I mentioned, I'm more than happy to help where I can in the C++ layer, but would be good if someone with much more experience could help lead that design and guidance on implementation. Is that you Jaap? I'm curious people's thoughts on moving toward N-API instead NAN? From what I understand that is the C++-binding layer for Node 8+. Personally, I'm excited to get started working with ya'll to expand my knowledge in other areas and push this project forward. Ryan

…

On Fri, Jul 26, 2019 at 3:19 AM Jaap van Hardeveld ***@***.***> wrote: From my experience to contributing to node-rdkafka, I'd have to underline this mention of @webmakersteve <https://github.com/webmakersteve>: 1. Upgrade NAN and get rid of all of the deprecations we have. This is a pretty big effort as the async worker handling has been hugely changed. 2. A lot of C++ refactoring - the code is very old and looks it. There is a lot of stuff that can be consolidated to reduce the huge amount of sprawl. 3. Get rid of the bundled librdkafka as part of the base package in favor of having a dependency on it being provided as a shared library in the box. I can be convinced that this is a bad idea, but so many of the problems with this library deal with the fact that it is built from source at install time. Those are the definitely the points causing a bunch of problems at the moment, the ones that have been causing stability issues thus far and threaten to do so even more in the future. The binding to librdkafka is what sets node-rdkafka apart with a unique value proposition: piggy backing of an very highly optimized native library that achieves order of magnitude higher throughput than the official Kafka client. That's the bit worth trying to save, imo. It's not that the Javascript side couldn't do with some love, it definitely could. But if what you're looking for is a modern JS code base, Typescript support, Promise support, strong test suite and CI, I wonder if you're time is not better spent on the newer KafkaJS project <https://kafka.js.org/>, which already has all of those things and is at a high rate of further development and progress. It's what we as a tiny 2-person indie team had to move to, and since we don't have much C++/C expertise, it's 100% Node codebase makes our contributions a whole bunch easier and more productive. That's not to say that I think node-rdkafka shouldn't be kept alive / revived, not at all! Given it's bindings to librdkafka I could see it as a simpler client that's useful if you need some really serious throughput: it could offer levels of performance hard to match with a fully Javascript-implemented client. But then the focus should at least be on that part, that unique strength. Otherwise, all we're doing is fragmenting the already small Node + Kafka ecosystem even further and just duplicating efforts :( — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#628?email_source=notifications&email_token=AA4MURFZF6UPRACGB7SSJ4DQBKXRTA5CNFSM4HSXN6UKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD233ZGY#issuecomment-515357851>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4MURB6ZZ6FLZJ2RP246W3QBKXRTANCNFSM4HSXN6UA> .

[JaapRood]

@codeburke unfortunately I can't be, through lack of experience with that side of the code base as well:

Unfortunately, that excludes myself, as all my C++ experience is just reading and hacking some other people's code. We're only a 2 person indie team, so lacking the resources to address the issues the project has been facing as of late.

[SoyYoRafa]

We can expect that users' runtimes will support N-API by January 2020 when Node 8 is deprecated. N-API guarantees API stability and ABI compatibility across major versions of node. So, moving to N-API has the big benefit of being able to compile node-rdkafka in node 10 and run it in future major versions of node (12, 14, etc) without recompiling the shared objects. As a result, I don't think upgrading NAN is worthwhile. Separately, we should use node-addon-api, not N-API directly. N-API is a low level C interface. node-addon-api is the C++, object oriented wrapper on N-API. So, around November/December would be a good time to start working on migrating to node-addon-api.