Describe integration with MLflow #3856

astrojuanlu · 2024-05-06T11:35:47Z

Description

Development notes

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

Read the contributing guidelines
Signed off each commit with a Developer Certificate of Origin (DCO)
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes
Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

astrojuanlu · 2024-05-06T11:47:53Z

Preview: https://kedro--3856.org.readthedocs.build/en/3856/integrations/mlflow.html

Closes #3541. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

astrojuanlu · 2024-05-07T14:00:34Z

Note to reviewers:

The idea of this page is to complement kedro-mlflow documentation, and in fact it contains several references to it. What this adds then is:

References to MLflow in the Kedro docs so users can easily find it
Short examples on how to create custom integrations using hooks
Fill some gaps in kedro-mlflow

The idea is for this page to serve as brief collection of MLOps use cases, and to use this as a template for future integrations.

I'm paging @stichbury as well because I was somewhat careless with the prose in certain parts.

I ~~confess~~ disclose that Lilli wrote the first 2 paragraphs.

docs/source/integrations/mlflow.md

noklam

Is it possible to keep kedro-mlflow as its own section? The current structure is by feature, "tracking", "artifact". The rest of our docs usually start from "basic" -> "advance".

docs/source/conf.py

docs/source/integrations/mlflow.md

astrojuanlu · 2024-05-10T08:52:42Z

Is it possible to keep kedro-mlflow as its own section? The current structure is by feature, "tracking", "artifact". The rest of our docs usually start from "basic" -> "advance".

I went back and forth several times on how to structure this page. I like how I ended up making 1 section per use case but I agree there might be other ways. Awaiting for @stichbury's take on this.

docs/source/integrations/mlflow.md

stichbury · 2024-05-10T11:26:07Z

I went back and forth several times on how to structure this page. I like how I ended up making 1 section per use case but I agree there might be other ways. Awaiting for @stichbury's take on this.

I found the sectioning useful but agree with @noklam that we usually break into simple and advanced usage. Can I suggest we do the same here and have the structure as follows, but will leave you to decide where the simple/advanced sections fall?

Maybe something like this?

However, I'm not that attached to this and if you want to stick with what you have, I'd say that's fine, but omit the basic second level "Use cases" header and promote all the following (currently 3rd level) to 2nd level.

Header

Prerequisites

Simple use cases

Tracking Kedro pipeline runs in MLflow using Hooks

Artifact tracking in MLflow using hooks

Advanced use cases

Complete tracking of Kedro runs in MLflow using `kedro-mlflow`

Tracking Kedro in MLflow using the Python API

Artifact tracking in MLflow using `kedro-mlflow`

Model registry in MLflow using `kedro-mlflow`

astrojuanlu · 2024-05-10T11:31:52Z

My biggest gripe with this is that it seems wrong to declare custom hooks as "basic" and using kedro-mlflow, which is objectively fewer lines of code (just a pip install kedro-mlflow away), "advanced". If anything, the former are more "custom", "ad-hoc", or "homegrown", whereas the latter is more "off-the-shelf".

I can totally see how someone starts with the custom hook ("basic"), then they start making it more complex because they need more functionality, and in the end it becomes way more difficult than just pip install kedro-mlflow and let the plugin take care of it for you, assuming the plugin does more or less exactly what you want to do.

stichbury · 2024-05-10T11:48:38Z

I can't work on your branch so I've forked and made a PR to commit back to it #3862

Please take a look, merge what you want, and I can review again when you have the entire page in your preferred final state (see comment about sectioning above).

* Some proposed edits Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Fix some Vale warnings Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

astrojuanlu · 2024-05-13T10:01:03Z

I'd say that's fine, but omit the basic second level "Use cases" header and promote all the following (currently 3rd level) to 2nd level.

We agreed to do this 👍🏼 Will make the change today

DimedS

Thank you, @astrojuanlu , for the excellent manual. Everything is working well. It's a great starting point for exploring Kedro+MLFlow. I've left a few minor comments.

docs/source/integrations/mlflow.md

astrojuanlu · 2024-05-16T06:19:52Z

Thanks for the review @Galileo-Galilei 🙏🏼 will address your comments ASAP.

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

astrojuanlu · 2024-06-04T09:52:24Z

I significantly reworked the order of the sections, but the content is largely the same. I think the flow is much nicer now - wouldn't have reached this stage without @Galileo-Galilei's insightful comments.

Please do have a look again.

astrojuanlu · 2024-06-04T09:52:42Z

Preview: https://kedro--3856.org.readthedocs.build/en/3856/integrations/mlflow.html

docs/source/integrations/mlflow.md

stichbury

Looks really great, well done 🌟 I made just 2 very minor suggestions.

DimedS

Fantastic work, @astrojuanlu! 🚀🚀🚀 I really like the new description flow, starting from easy use cases and progressing to more complex ones like Hooks and Session management. I left a few minor comments.

docs/source/integrations/mlflow.md

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Juan Luis Cano Rodríguez <hello@juanlu.space>

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

astrojuanlu · 2024-06-04T13:17:47Z

docs/source/integrations/mlflow.md

+$ kedro run --to-outputs=X_test,y_test
+...
+$ kedro run --from-nodes=evaluate_model_node --params mlflow_run_id=4cba84...


This was somewhat clunky, dumped some thoughts in #3922

I agree, but it's great that you added that section! It's an interesting functionality.

astrojuanlu · 2024-06-04T13:18:46Z

docs/source/integrations/mlflow.md

+
+```
+(.venv) $ pip install mlflow
+(.venv) $ mlflow ui --backend-store-uri ./mlflow_runs


Why not leaving the default location? Well, because it's called mlruns, which could be conflated with https://github.com/mlrun/mlrun 😬 This requires us to write a mlflow.yml in the first step, but I think it's not the end of the world

It's actually better to show the mlflow.yml to the users :)

astrojuanlu · 2024-06-06T05:07:46Z

@Galileo-Galilei in the interest of putting this in front of users already and given that I addressed your initial comments, I'm going ahead with merging this - but if you spot any glaring mistakes or further areas for improvement in a post-merge review, do leave a comment and I'll send another PR with the amendments 🙏🏼

Galileo-Galilei · 2024-06-06T06:20:11Z

Sorry Juan, readthedocs preview did not render well in my phone for some unknown reasons so it was hard to review. It does work on the latest branch though, so I had a chance to read it. It's much easier to read now, thanks for the work!

Two minor comments I'll address in a further PR:

it feels a bad idea to customize the name of the local backend store, because there is 99% chances that user will forget to add it to gitignore and will leak the data. The default mlruns is already in kedro's template .gitignore
if you don't customize the name, you do not need a custom mlflow.yml, hence it simplifies the setup.

astrojuanlu · 2024-06-06T07:37:53Z

Thanks!

The default mlruns is already in kedro's template .gitignore

Oh, I remembered #3765, and that we didn't follow suit with the starters...

About the customisation, I was hesitating about it myself, see #3856 (comment) I'd rather want to avoid mlrun appearing in a Ctrl+K/F of our docs for the time being, but I see your point around defaults. Maybe we can add mlflow_runs to our .gitignores?

Galileo-Galilei · 2024-06-07T20:25:25Z

I really think this is a bad idea to create our own standard for many reasons:

MLflow tries hard to create a local mlruns folder : if one use the debugger, runs the kedro pipeline before using the UI command, use runs a node interactively (e.g. in a notebook)... without having the proper setup he will likely ends ups with both an auto created mlruns folder and your custom mlflow_runs. He will see only partials runs in the UI, and eventually we will end up having some questions in slack because of this weird setup.
some teams have their own template, and for sure mlflow_runs is not inside their .gitignore ;)
eventually, some users can seek help outside the kedro community, and the setup will confuse coworkers / stackoverflow members...

I don't like the name either, but the point of having a standard is having everyone sticking to it and I think we should keeo this well established one (and I don't think having mlrun in the docs is a big deals, mlflow users usually know about mlruns folder)

astrojuanlu · 2024-06-10T07:43:46Z

I see your points, fair enough. @Galileo-Galilei do you want to send the PR yourself?

* Describe integration with MLflow Closes kedro-org#3541. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * [Docs] Update MLflow docs page in kedro-org#3856 (kedro-org#3862) * Some proposed edits Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Fix some Vale warnings Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Simplify MLflow launch Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Add label to runtime params section Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Restructure MLflow document Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> * Apply suggestions from code review Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Juan Luis Cano Rodríguez <hello@juanlu.space> * Apply suggestions from code review Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> --------- Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Juan Luis Cano Rodríguez <hello@juanlu.space> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com> Signed-off-by: bpmeek <bpmeek.developer@gmail.com>

astrojuanlu force-pushed the docs/kedro-mlflow branch 2 times, most recently from a8256cf to df922b7 Compare May 7, 2024 12:51

Describe integration with MLflow

24c25c2

Closes #3541. Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

astrojuanlu force-pushed the docs/kedro-mlflow branch from df922b7 to 24c25c2 Compare May 7, 2024 13:57

astrojuanlu requested review from noklam and Galileo-Galilei May 7, 2024 13:57

astrojuanlu marked this pull request as ready for review May 7, 2024 13:57

astrojuanlu requested a review from yetudada as a code owner May 7, 2024 13:57

astrojuanlu commented May 7, 2024

View reviewed changes

docs/source/integrations/mlflow.md Outdated Show resolved Hide resolved

noklam reviewed May 9, 2024

View reviewed changes

docs/source/conf.py Show resolved Hide resolved

docs/source/integrations/mlflow.md Outdated Show resolved Hide resolved

stichbury reviewed May 10, 2024

View reviewed changes

docs/source/integrations/mlflow.md Outdated Show resolved Hide resolved

stichbury reviewed May 10, 2024

View reviewed changes

docs/source/integrations/mlflow.md Outdated Show resolved Hide resolved

[Docs] Update MLflow docs page in #3856 (#3862)

aedab89

* Some proposed edits Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Fix some Vale warnings Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com>

DimedS approved these changes May 13, 2024

View reviewed changes

docs/source/integrations/mlflow.md Outdated Show resolved Hide resolved

docs/source/integrations/mlflow.md Outdated Show resolved Hide resolved

docs/source/integrations/mlflow.md Outdated Show resolved Hide resolved

Galileo-Galilei reviewed May 15, 2024

View reviewed changes

noklam and others added 5 commits May 20, 2024 11:54

Merge branch 'main' into docs/kedro-mlflow

7a0228f

Simplify MLflow launch

db60ccf

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

Add label to runtime params section

abdf298

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

Restructure MLflow document

0b38f26

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

Merge branch 'main' into docs/kedro-mlflow

3c4e972

astrojuanlu requested review from DimedS and stichbury June 4, 2024 09:51

astrojuanlu requested review from noklam and Galileo-Galilei June 4, 2024 09:51

stichbury reviewed Jun 4, 2024

View reviewed changes

docs/source/integrations/mlflow.md Outdated Show resolved Hide resolved

stichbury reviewed Jun 4, 2024

View reviewed changes

docs/source/integrations/mlflow.md Outdated Show resolved Hide resolved

stichbury approved these changes Jun 4, 2024

View reviewed changes

DimedS approved these changes Jun 4, 2024

View reviewed changes

docs/source/integrations/mlflow.md Show resolved Hide resolved

docs/source/integrations/mlflow.md Show resolved Hide resolved

astrojuanlu and others added 2 commits June 4, 2024 15:04

Apply suggestions from code review

777a5f0

Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Juan Luis Cano Rodríguez <hello@juanlu.space>

Apply suggestions from code review

746ad72

Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>

astrojuanlu commented Jun 4, 2024

View reviewed changes

marrrcin approved these changes Jun 5, 2024

View reviewed changes

astrojuanlu merged commit 014d7a8 into main Jun 6, 2024
9 checks passed

astrojuanlu deleted the docs/kedro-mlflow branch June 6, 2024 05:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Describe integration with MLflow #3856

Describe integration with MLflow #3856

astrojuanlu commented May 6, 2024

astrojuanlu commented May 6, 2024

astrojuanlu commented May 7, 2024

noklam left a comment

astrojuanlu commented May 10, 2024

stichbury commented May 10, 2024

astrojuanlu commented May 10, 2024

stichbury commented May 10, 2024

astrojuanlu commented May 13, 2024

DimedS left a comment

astrojuanlu commented May 16, 2024

astrojuanlu commented Jun 4, 2024

astrojuanlu commented Jun 4, 2024

stichbury left a comment

DimedS left a comment

astrojuanlu Jun 4, 2024

DimedS Jun 4, 2024

astrojuanlu Jun 4, 2024

marrrcin Jun 5, 2024

astrojuanlu commented Jun 6, 2024

Galileo-Galilei commented Jun 6, 2024 •

edited

Loading

astrojuanlu commented Jun 6, 2024

Galileo-Galilei commented Jun 7, 2024 •

edited

Loading

astrojuanlu commented Jun 10, 2024

Describe integration with MLflow #3856

Describe integration with MLflow #3856

Conversation

astrojuanlu commented May 6, 2024

Description

Development notes

Developer Certificate of Origin

Checklist

astrojuanlu commented May 6, 2024

astrojuanlu commented May 7, 2024

noklam left a comment

Choose a reason for hiding this comment

astrojuanlu commented May 10, 2024

stichbury commented May 10, 2024

Header

Prerequisites

Simple use cases

Tracking Kedro pipeline runs in MLflow using Hooks

Artifact tracking in MLflow using hooks

Advanced use cases

Complete tracking of Kedro runs in MLflow using kedro-mlflow

Tracking Kedro in MLflow using the Python API

Artifact tracking in MLflow using kedro-mlflow

Model registry in MLflow using kedro-mlflow

astrojuanlu commented May 10, 2024

stichbury commented May 10, 2024

astrojuanlu commented May 13, 2024

DimedS left a comment

Choose a reason for hiding this comment

astrojuanlu commented May 16, 2024

astrojuanlu commented Jun 4, 2024

astrojuanlu commented Jun 4, 2024

stichbury left a comment

Choose a reason for hiding this comment

DimedS left a comment

Choose a reason for hiding this comment

astrojuanlu Jun 4, 2024

Choose a reason for hiding this comment

DimedS Jun 4, 2024

Choose a reason for hiding this comment

astrojuanlu Jun 4, 2024

Choose a reason for hiding this comment

marrrcin Jun 5, 2024

Choose a reason for hiding this comment

astrojuanlu commented Jun 6, 2024

Galileo-Galilei commented Jun 6, 2024 • edited Loading

astrojuanlu commented Jun 6, 2024

Galileo-Galilei commented Jun 7, 2024 • edited Loading

astrojuanlu commented Jun 10, 2024

Complete tracking of Kedro runs in MLflow using `kedro-mlflow`

Artifact tracking in MLflow using `kedro-mlflow`

Model registry in MLflow using `kedro-mlflow`

Galileo-Galilei commented Jun 6, 2024 •

edited

Loading

Galileo-Galilei commented Jun 7, 2024 •

edited

Loading