Can Spring Cloud Data Flow be used for implementing data mesh architecture?

[samuelstein]

Thanks for the great structured microsite about data mesh architecture.
I was wondering if spring cloud data flow could be also used as a tech stalk to implement data mesh architectures? what do you think?

[jochenchrist]

In my opinion, Spring Cloud Data Flow can be a solid choice for implementing data products.

I would consider this tech stack:

• if you have a strong Java/Kotlin/Spring developer base for your operational systems
• if you like to implement some data products with streaming-approach (you will always have batch data products as well)
• start your data mesh journey from source-aligned teams

Also, consider these aspects:

• data scientists, data engineers are not used to work with Spring, you might need an additional technology stack for them (Spark Engine, ...)
• Spring Data has no strong support for analytical data stores (S3, Datawarehouses, ...), I would consider combining it with DuckDB and write files on bucket stores.
• Spring Cloud Data Flow has no data product support (as an abstraction / module level), you need governance policies to define, what a data product is and how data contracts should look like.

To give a recommendation, I would need additional information on your context.

[samuelstein]

Thank you very much @jochenchrist for your assessment and the things also to consider.
This gives us the flexibility for starting small and evolve when necessary.