Question: Dynamic number of steps

[arpadgabor]

I'm currently using OpenWorkflow and it's all fine, but our app needs to get data from a third party API and that data can become pretty big so storing/retrieving the step result is quite slow.

Currently this data fetching happens in a single step, but I was thinking about splitting it into multiple steps (we call the same api's multiple times with different input data).

Would having a dynamic number of steps feasible? Would this not conflict somehow with the problems mentioned in the versioning documentation?

e.g. code

await Promise.all(
  myDynamicLengthArray.map((data, index) => {
    return step.run({ name: `fetch-data-$index{}` }, () => { ... })
  }
)

[jamescmartinez]

Would having a dynamic number of steps feasible?

yes, having a dynamic number of steps is supported and works well when step names are deterministic. I'll make sure to document this better.

I'd recommend using a unique ID from the data itself for the step name instead of the index.

That said, if you don't have that option, you have the right idea using the array index in the name (fetch-data-${index}), but be sure the order of myDynamicLengthArray does not change between the initial run and a replay, since "step 0" needs to always correspond to the same item. If the array order changes, the workflow would return the wrong cached result for that item, which would break the workflow.

that data can become pretty big so storing/retrieving the step result is quite slow

Splitting the work into multiple steps helps, but OpenWorkflow still persists the output of every step to the database. If the total amount of data is high, your workflow history will still be heavy.

Instead, I'd offload the data elsewhere. Something more like this:

1. fetch the data inside the step
2. write the heavy data to an object store like S3
3. store only a reference to the file in S3

const storedItem = await step.run(
  { name: `fetch-data:${item.id}` },
  async () => {
  // 1. get the data
  const data = await thirdPartyApi.fetch(item.id);
  
  // 2. upload the data to S3
  const s3Key = `workflow-data/${item.id}.json`;
  await s3.send(new PutObjectCommand({
    Bucket: "your-bucket",
    Key: s3Key,
    Body: JSON.stringify(data),
    ContentType: "application/json",
  }));

  // 3. store only the reference in your DB, not the data itself
  return { id: item.id, s3Key };
);

[jamescmartinez]

docs PR: #278

[arpadgabor]

Awesome, thanks a lot for the answer and also for updating the docs!

Also, in my case a singular payload isn't that big (about 20-30kb), but when added up it can get quite big (had some cases where the json reached about 10mb). I noticed the payloads are stored in separate rows in the db so it would be enough in my case. Thanks!

[jamescmartinez]

great!

One thing to keep in mind: when a workflow resumes or replays, it fetches the history of completed steps to reconstruct the state. So if you have 10MB of total payload data across those steps, that 10MB will still be loaded from Postgres -> Node.js on replay.

For 10MB that is usually totally fine, just something to be aware of if it grows significantly larger.

Closing this out since the docs are updated, but free to reopen or post in our Discord if you run into issues.