[KED-1106] Add Roadmap, Objectives and better actual use cases

[arita37]

Description

Roadmap and objective of the framework seems missing to see which direction it goes.
Expecially, what will be improved/supported and (what Not).
It allows to plan how to "potentiall" integrate into existing framework workflow.

Data Connector seems actively developed.
BUT machine learning model lifefcyle is not handle (and will or will not be).
Objectives and Roadmap will help identifying.

All major frameworks have clear roadmaps : TensorFlow, Pandas, ...

2. Use case example :
   Example are not well chosen (ie space flight, not really relevant for most people).
   If your targets are : Data Engineer, Data Science, please five example using :
   Specific Kaggle Dataset.
   Specific dataset (embed with the repo) to see the full process.

   It will raise more attention to the community.

[lorenabalan]

Hi @arita37, thank you for your feedback! Please see the responses below.

In terms of public roadmap and objectives, we’re a bit too young and nimble for this. :) There is a list of prioritised goals we adhere to internally, but it is flexible in terms of change, and we quite like it that way at this stage of Kedro. Maybe when we're closer to ruling the world like pandas or TensorFlow this can be revisited. Meanwhile, things that are unlikely to change and easier to grasp can be found here:
https://kedro.readthedocs.io/en/latest/06_resources/02_architecture_overview.html
https://kedro.readthedocs.io/en/latest/06_resources/01_faq.html#how-does-kedro-compare-to-other-projects
https://kedro.readthedocs.io/en/latest/06_resources/01_faq.html#what-is-the-philosophy-behind-kedro

The Spaceflight tutorial is meant to be a simple, easy-to-follow example even for data practitioners of little experience. We also didn’t want to force users to download some large datasets every time they install or upgrade Kedro, or when creating a Kedro project, which is why we keep it in a separate repository. Also, this makes it easier for tutorial-only contributions! You’re welcome to contribute a more advanced tutorial to https://github.com/quantumblacklabs/kedro-examples :)

[arita37]

Hello, This is not related whether its rule the world or not.... If there is no roadmap, no direction where the framework goes, how can we rely on it for actual use case in company.... Smaller projects have also roadmap because can identify what objectives they want to reach and lay out the overall features objectives. (Goal plan and achievement rather than a stack of hacky code put together...) Does this framework aim to be used (and potentially improved by the community) or its an open release for internal needs.

…

On Oct 9, 2019, at 23:02, Lorena Bălan ***@***.***> wrote:  Hi @arita37, thank you for your feedback! Please see the responses below. In terms of public roadmap and objectives, we’re a bit too young and nimble for this. :) There is a list of prioritised goals we adhere to internally, but it is flexible in terms of change, and we quite like it that way at this stage of Kedro. Maybe when we're closer to ruling the world like pandas or TensorFlow this can be revisited. Meanwhile, things that are unlikely to change and easier to grasp can be found here: https://kedro.readthedocs.io/en/latest/06_resources/02_architecture_overview.html https://kedro.readthedocs.io/en/latest/06_resources/01_faq.html#how-does-kedro-compare-to-other-projects https://kedro.readthedocs.io/en/latest/06_resources/01_faq.html#what-is-the-philosophy-behind-kedro The Spaceflight tutorial is meant to be a simple, easy-to-follow example even for data practitioners of little experience. We also didn’t want to force users to download some large datasets every time they install or upgrade Kedro, or when creating a Kedro project, which is why we keep it in a separate repository. Also, this makes it easier for tutorial-only contributions! You’re welcome to contribute a more advanced tutorial to https://github.com/quantumblacklabs/kedro-examples :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[lorenabalan]

It has never been implied that "there is no roadmap" or "no direction", but a matter of how we choose to share this with the public, in a constructive way. This item is on our backlog and we'll have a think about how to approach this in the future. We would also never open-source if we didn't expect/want the community to contribute and get involved in making this project even better. I believe we've been doing decently well in proving that throughout our engagement with the OS community, and we've been nothing but welcoming and open to contributions & feedback. :)

[yetudada]

Thanks for this feedback @arita37. We've noted your concerns and will look at different ways to raise the themes that we're exploring for Kedro. It might be that we make this public via GitHub Projects so that we can get community input on what we're tackling next.

You're right that this needs to be evidenced. I'll reopen this issue, complete that task and then close it again.

[arita37]

Thanks. In that case, If this is a “future feature”, this is useful. Especially, showing in which objectives will aim to be covered. (And Which one will not be covered). Ex: AbstractDataset expanded to cover more datasources .... Define AbstractPipeline scheme ( for Ml lifecycle and DS lifecycle). The workflow between DS and ML are complex with many moving parts, and very different people needs...

…

On Oct 10, 2019, at 18:10, Lorena Bălan ***@***.***> wrote:  It has never been implied that "there is no roadmap" or "no direction", but a matter of how we choose to share this with the public, in a constructive way. This item is on our backlog and we'll have a think about how to approach this in the future. We would also never open-source if we didn't expect/want the community to contribute and get involved in making this project even better. I believe we've been doing decently well in proving that throughout our engagement with the OS community, and we've been nothing but welcoming and open to contributions & feedback. :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[yetudada]

Hi @arita37, I'm getting to create some actionable tasks from this issue so that I can close it.

You will see, our uploaded product themes here: https://github.com/quantumblacklabs/kedro/projects/1

And in terms of your thoughts on the extending the AbstractDataSet, that is something that our user have been doing. We have a contrib system, see here that users have been adding new data connecters and in some cases users have built their own model classes like the MlflowDataset, see #113.

What do you have in mind for the AbstractPipeline?

And lastly, have you been able to pick up Kedro at all? Or did you go for Mlflow?

[arita37]

Thanks for the start of roadmap.
This is indeed useful to see the visbility over 1 year.
Main point of MLFlow, besides the functionality, is we have confidence that DataBricks will support
the Open Source for some time (ie as they do for Spark) and also integrate their external users concerns.

Kedro being 1/2 proprietary and 1/2 open,
it might end up like GraphLab (bought by Apple)
Check the story of sframe.... https://github.com/turi-code/SFrame

Setting the mid-term ojectives of Kedros will help a lot, especially where it stands between the framework.

Thanks !

[yetudada]

@arita37 Thanks for the comment. This isn't even the roadmap for the year but rather the quarter with possible extensions into Q1 2020. In terms of support model, the Kedro team is fully staffed to support it because the tool is so heavily used internally and by our clients.

We define the vision of where we see Kedro going in our contribution docs. I'm going to close this ticket for now. Thank you!