Generates random JSON messages and sends them to Apache Kafka.
It uses Apache Zookeeper to discover the available brokers and partitions for a given topic.
Usage:
$ java -jar target/synthetic-producer-1.4.0-selfcontained.jar
usage: java -jar synthetic-producer.jar
-c,--config <arg> config file path
-h,--help show this help
-r,--rate <arg> messages rate per second
-t,--threads <arg> number of producer threads
-z,--zookeeper <arg> zookeeper connect stringThe config file specifies the Zookeeper connection string, the rate of messages generated, the threads to use, the topic where you want to produce, and the messages' schema.
For example, take a look at the following config:
topic: testTopic
partitionKey: myPartitionKeyField
fields:
myConstant:
type: constant
value: 'Hello, World!'
randomNumber:
type: integer
min: 0
max: 100Using a file like that will make the producer will connect to Zookeeper at "localhost:2181" in order create two producing threads, and send messages to the topic "testTopic" at a constant rate of 100 msgs per second, which would look like the following:
{ "myConstant": "Hello, World!", "randomNumber": 4}
{ "myConstant": "Hello, World!", "randomNumber": 12}
{ "myConstant": "Hello, World!", "randomNumber": 94}
{ "myConstant": "Hello, World!", "randomNumber": 33}You can find a full example of config file in this file
You can use a variety of different field types to generate your random message. Some types accepts one or more parameters that modify the field randomness. For example, the constant type always generates the same value for a field, but the integer type generates a random number between 'min' and 'max' for each field.
The available types, with their parameters, are the following:
Generates a constant value
Parameters:
- value: The value to be generated
Example:
myField:
type: constant
value: "This is a constant"Generates a random integer
Parameters:
- min: The minimum integer to be generated (Default 0)
- max: The maximum integer to be generated (Default Integer.MAX_VALUE)
- negative: True if the generated number must be negative (It will be multiplied by -1)
Example:
myField:
type: integer
min: 0
max: 50Generates a random double
Parameters:
- min: The minimum double to be generated (Default 0.00)
- max: The maximum double to be generated (Default Double.MAX_VALUE)
- negative: True if the generated number must be negative (It will be multiplied by -1)
- truncate: The decimals numbers (Default 10000, four decimals)
Example:
myField:
type: double
min: 0.00
max: 50.00Generates an integer counter between two values
Parameters:
- min: Minimum value that counter can return (Default 0)
- max: Maximum value that counter can return (Default Integer.MAX_VALUE)
- direction: True if the counter goes from minimum to maximum. False if it goes from maximum to minimum. (Default true)
Example:
myField:
type: counter
min: 1
max: 10Generates a random IP address
Parameters:
- min: The minimum ip to be generated (Default 0.0.0.0)
- max: The maximum ip to be generated (Default 255.255.255.255)
Example:
myField:
type: ip
min: 192.168.0.1
max: 192.168.0.255Generates a random MAC address
Parameters:
- min: The minimum ip to be generated (Default 00:00:00:00:00:00)
- max: The maximum ip to be generated (Default FF:FF:FF:FF:FF:FF)
- separator: The string using to split the mac on the output.
Example:
myField:
type: mac
min: 00:00:00:00:00:11
max: 00:00:00:00:00:44Generates a field with the current timestamp of the machine
Example:
myField:
type: timestampGenerates a random element from the given collection of elements
Parameters:
- values: An array of elements
Example:
myField:
type: collection
values:
- Campus A
- Campus B
- Campus C
- Campus D
- Campus EGenerates a random field based on a composition of two or more random fields
Parameters:
- separator: The character that will be used as the separator between the two generated values
- components: An array of types definitions that will be generated before joining the result with the separator
Examples:
myField:
type: composition
separator: ':'
components:
- type: integer
min: 0
max: 11
- type: integer
min: 0
max: 101This will create values like the following: 2:34, 6:11, 1:75, 10:62... Of course, you can use a composition type as a component for another composition type.
Generate a fixed number of sets with multiple fixed field per set.
Parameters:
- numbers: The different numbers of sets
- components: The components that form a set
set123:
type: set
numbers: 10
components:
client_mac:
type: mac
min: 00:00:00:00:00:11
max: 00:00:00:00:00:44
src:
type: ip
min: 80.82.34.12
max: 80.82.34.72This will create 10 sets form by a client_mac of type mac and src of type ip. The sets are making on the init and them don't change after. A message has 1 set that is choose randomly from 10 sets.
Choose one component randomly among all the components inside it.
Parameters:
- components: List of the components that will be picked randomly.
You can find a simple example below:
mySimpleRandom:
type: random
components:
-
myField:
type: integer
min: 101
max: 110
-
myField:
type: integer
min: 201
max: 210
-
myField:
type: integer
min: 301
max: 310This will generate values from 101 to 110, from 201 to 210 and from 301 to 310. Also you can nest random types in order to get more complex combinations.
Generate a new json inside other json.
Parameters:
- components: The components that form a set
engine_id_name:
type: constant
value: IANA-L4
myNewJson:
type: json
components:
src:
type: ip
min: 192.168.0.1
max: 192.168.255.255
application_id:
type: composition
separator: ':'
components:
- type: integer
min: 0
max: 11
- type: integer
min: 0
max: 101
myNewJsonInside:
type: json
components:
dst:
type: ip
min: 192.168.0.1
max: 192.168.255.255This will generate messages like this:
{"myNewJson":{"myNewJsonInside":{"dst":"192.168.231.69"},"src":"192.168.161.220","application_id":"9:32"},"engine_id_name":"IANA-L4"}Generate a random latitude and longitude inside a circle that is defined using radius.
- latPoint: The latitude center point.
- longPoint: The longitude center point.
- radius: The radius of the circle.
- latitudeDim: The field name of the latitude value.
- longitudeDim: The field name of the longitude value.
- truncate: The decimals numbers (Default 10000, four decimals)
Example:
coordinates:
type: coordinate
latPoint: 30.000
longPoint: -40.000
radius: 1000
latitudeDim: latitude
longitudeDim: longitude
truncate: 1000Generate array with different components.
- components: The different types that build the array
Example:
myArray:
type: array
components:
field1:
type: json
components:
dst:
type: ip
min: 192.168.0.1
max: 192.168.255.255
field2:
type: constant
value: 5This example generate an array like:
{"myArray":[5,{"dst":"192.168.142.39"}]}
{"myArray":[5,{"dst":"192.168.121.126"}]}
{"myArray":[5,{"dst":"192.168.189.54"}]}Generate one component each time. The sequential restart when reach the end.
- components: The different types that build the sequence
sequentialField:
type: sequential
components:
-
seq1:
type: set
numbers: 1
components:
longitude:
type: constant
value: 1
latitude:
type: constant
value: -1
-
seq2:
type: set
numbers: 1
components:
longitude:
type: constant
value: 2
latitude:
type: constant
value: -2
-
seq3:
type: set
numbers: 1
components:
longitude:
type: constant
value: 3
latitude:
type: constant
value: -3This example generate an array like:
{"longitude":1,"latitude":-1}
{"longitude":2,"latitude":-2}
{"longitude":3,"latitude":-3}
{"longitude":1,"latitude":-1}
{"longitude":2,"latitude":-2}- Fork it
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create a new Pull Request