Writing multiple iterations to a single file

[AaronJessen]

First off, thanks so much for creating the json-data-generator. It's exactly what I needed at this point in my project, and it will be invaluable going forward. Thanks for your hard work!

Can you recommend the best way (i.e. the correct place in the code) to modify the workflows so as to write all events/iterations to a single file, as opposed to a single file for each workflow iteration? I'm not fluent in Java, so I'm not certain where/how to perform this change.

Thanks.

[andrewserff]

Hey @AaronJessen did you figure this out? Just saw you closed it. I haven't had a chance to look at it yet, so let me know if you solved this.

[AaronJessen]

Hi Andrew,

I did manage to solve this by adding all generated Events to an ArrayList (without committing them to a file/logger), and then performing the logEvent iteration outside of the while loop.

I also added a quantity parameter to each configuration step, so I could easily generate 1,000 or even a million instances of a given config step that all get added to a single file for easy importing into a NoSQL database.

// Store all events in this ArrayList for future processing by each logger type
ArrayList<String> events = new ArrayList<String>();

Iterator<WorkflowStep> it = workflow.getSteps().iterator();

    while (running && it.hasNext()) {
        WorkflowStep step = it.next();

        // custom "quantity" parameter added to each configuration step to allow for looping
        int stepQuantity = step.getQuantity();
        int stepQuantityRun = 0;

        while (stepQuantityRun < stepQuantity) {
            executeStep(step);
            stepQuantityRun++;
        }

        if (!it.hasNext() && workflow.isRepeatWorkflow()) {
            it = workflow.getSteps().iterator();

            try {
                performWorkflowSleep(workflow);
            } catch (InterruptedException ie) {
                //wake up!
                running = false;
                break;
            }
        }
    }
    
    // Log all events here, instead, by concatenating the Events ArrayList from above
    for (EventLogger l : eventLoggers) {
        l.logEvent(String.join(",", events)
                , null);
    }

Here's the executeStep method:

try {
                String event = generateEvent(wrapper);
               
                // Add the events to the Events Array
                events.add(event);

                try {
                    performEventSleep(workflow);
                } catch (InterruptedException ie) {
                    //wake up!
                    running = false;
                    break;
                }
            } catch (IOException ioe) {
                log.error("Error generating json event", ioe);
            }

I'm building a CRM system, and I'd like the mock data to have relationships built in, despite the records continuing to be selected randomly. My goal is to have an Account/Contact class that not only randomly pulls a Name from the list of companies/contacts, but when company "XYZ" is selected, an ID field of "123" would always be included as well (instead of a random alphaNumeric type). In that way, the mock data has relationships built into it and is much more useful for intricate systems, such as a CRM app.

I'm hacking around and am getting closer to a solution, but again, I'm completely new to Java, so the progress is slow. Any suggestions would be greatly appreciated.

[Nelly-Barret]

Hi Andrew,

I did manage to solve this by adding all generated Events to an ArrayList (without committing them to a file/logger), and then performing the logEvent iteration outside of the while loop.

I also added a quantity parameter to each configuration step, so I could easily generate 1,000 or even a million instances of a given config step that all get added to a single file for easy importing into a NoSQL database.

...

Hello @AaronJessen and @andrewserff ,

I'm very interested in your open source software to generate large JSON datasets to evaluate my own software. As far as I looked at it, it is very well coded (even if it lacks some comments)! Like @AaronJessen , I need to generate one file which contains all the generated objects and the quantity parameter would be very useful for me.

Unfortunately, I'm not able to use @AaronJessen pieces of code to do it on my own. Have you forked the project and made a version with your improvements? If you have, I will be very grateful if you can share it with me.

Sorry for re-opening this old issue but I need some insights.

[UPDATE] I finally managed to use the quantity parameter as an argument and managed to write the events in one file.

Best,

Nelly