Migrate to XGBoost mainline repository

[mrocklin]

@RAMitchell (an xgboost maintainer) was mentioning that it might be possible to migrate the whole of the dask-xgboost codebase into xgboost itself.

Any thoughts or concerns about this?

I think that the proposed API change would be to add a dask_client= keyword (or something similar) to the official train and predict methods.

[TomAugspurger]

No objections here, though I think the CI is failing with the latest xgboost. I have a half-written tutorial copying https://xgboost.readthedocs.io/en/latest/jvm/xgboost4j_spark_tutorial.html. I’ll try to finish that up in a couple weeks.

[javabrett]

Someone should verify that this code with license (BSD 3-clause) can be absorbed into and under their license (ASF 2.0) without changes.

I expect this is fine as BSD-3-clause is permissive, but best make sure they have checked before they accept the code, that no relicensing is required.

[jakirkham]

As xgboost now has native Dask support in mainline ( dmlc/xgboost#4473 ), what would like to do with this repo?

[TomAugspurger]

At least update the readme, possible add a warning on import.

…

On May 29, 2019, at 11:22, jakirkham ***@***.***> wrote: As xgboost now has native Dask support in mainline ( dmlc/xgboost#4473 ), what would like to do with this repo? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[TomAugspurger]

Will want to wait until there’s an XGBoost release though.

…

On May 29, 2019, at 11:22, jakirkham ***@***.***> wrote: As xgboost now has native Dask support in mainline ( dmlc/xgboost#4473 ), what would like to do with this repo? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[jakirkham]

That sounds reasonable. How do we handle new issues that are reported here?

[mrocklin]

Just to be clear, the solution in XGBoost is quite a bit different than what is implemented in this repository. Folks may want to familiarize themselves with their solution before deprecating this repository entirely.

…

On Fri, May 31, 2019 at 9:47 AM jakirkham ***@***.***> wrote: That sounds reasonable. How do we handle new issues that are reported here? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#39?email_source=notifications&email_token=AACKZTB2OAIN5DZCBFXTDLTPYFJCPA5CNFSM4HJ7HKPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWVYQ5Q#issuecomment-497780854>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACKZTAEGDR6ZNEDJRX7UFTPYFJCPANCNFSM4HJ7HKPA> .

[ksangeek]

@mrocklin Could you please elaborate a bit on how the implementation in this repository is different from the one included in dmlc/xgboost? And if you have some more time on what circumstances this one could be preferred?

[mrocklin]

I don't know the other implementation particularly well. I recommend reading their docs to get a more authoritative perspective. https://xgboost.readthedocs.io/en/latest/python/python_api.html#module-xgboost.dask

…

On Tue, Jun 18, 2019 at 5:39 PM ksangeek ***@***.***> wrote: @mrocklin <https://github.com/mrocklin> Could you please elaborate a bit on how the implementation in this repository is different from the one included in dmlc/xgboost? And if you have some more time on what circumstances this one could be preferred? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39?email_source=notifications&email_token=AACKZTGHKLYOJYJCFTNHSL3P3D6RJA5CNFSM4HJ7HKPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODX7BT6Y#issuecomment-503192059>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACKZTBTMTY4J6YR4ETU6C3P3D6RJANCNFSM4HJ7HKPA> .

[RAMitchell]

@ksangeek See demos here: https://github.com/dmlc/xgboost/tree/master/demo/dask

The xgboost dask integration is currently slightly more low-level than what is here. We are currently considering if we can extend the API to provide more high level functionality currently available in dask-xgboost without duplicating existing xgboost APIs. If this can be achieved we should be able to definitively move users from dask-xgboost to native xgboost integration and deprecate this without loss of user experience.

[ksangeek]

Thank you @mrocklin and @RAMitchell for the useful information.

[kylejn27]

Has this been thought about further? There's functionality in this library that I'd like to implement that's already been done in dmlc/xgboost's native dask integration

[TomAugspurger]

From what I understand, the implementation in xgboost is closer to what we have here now.

Personally, I'd like to see this handled within xgboost itself, to reduce the maintenance load here :)

Are there any features within dask-xgboost not handled within xgboost itself?