Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for keys other than strings #1544

Closed
chuck-alt-delete opened this issue Mar 1, 2023 · 2 comments
Closed

Add support for keys other than strings #1544

chuck-alt-delete opened this issue Mar 1, 2023 · 2 comments

Comments

@chuck-alt-delete
Copy link

Is your feature request related to a problem? Please describe.
I would like to produce kafka record keys that are not strings (eg ints). I'm using KafkaJS version 2.2.3.

Describe the solution you'd like
A clear and concise description of what you want to happen. Please include links to other clients' solutions and/or KIPs for Kafka features if applicable.

Additional context
Here is a minimally reproducible example:

const { Partitioners } = require('kafkajs');
const { faker } = require('@faker-js/faker');
const kafkaConfig = require('./src/kafka/kafkaConfig');

async function produce(k,v) {
    let kafka = await kafkaConfig();
    const producer = kafka.producer({
        createPartitioner: Partitioners.DefaultPartitioner
    });
    await producer.connect();

    try {
        await producer.send({
            topic: "chuck_test",
            messages: [{
                key: k,
                value: v
            }]
        }); 
    
        console.log(`key:${k}\nvalue:${v}`)
    } catch (error) {
        console.error(error)
    }
    
    await producer.disconnect()
}

let my_key = faker.datatype.number(10);
let my_value = faker.internet.email();

produce(my_key,my_value);
produce("0",my_value);

The first record gets an error while the second is fine.

{"level":"ERROR","timestamp":"2023-03-01T20:26:43.464Z","logger":"kafkajs","message":"[Producer] The \"string\" argument must be of type string or an instance of Buffer or ArrayBuffer. Received type number (5)","retryCount":0,"retryTime":266}
...
key:0
value:Rafael17@hotmail.com
@Nevon
Copy link
Collaborator

Nevon commented Mar 2, 2023

So the keys in Kafka are always just bytes. The fact that we accept a string is just convenience because there's a default way to encode that string into a buffer (see Buffer.from). But for numbers there are many ways you could choose to serialize them, big endian vs little endian, for example. That's why the input type is string | Buffer | null:

kafkajs/types/index.d.ts

Lines 109 to 110 in ff3b111

export interface Message {
key?: Buffer | string | null

If you want to use a number as a key you have to decide for yourself how you want that serialized. For example:

const key = Buffer.alloc(8)
key.writeBigInt64BE(BigInt(9007199254740991))

@Nevon Nevon closed this as completed Mar 2, 2023
@chuck-alt-delete
Copy link
Author

“Bring your own serializers”. Got it! Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants