clarify semantics in order_by docstring #518
Conversation
Deploying datachain-documentation with
|
| Latest commit: |
db48bfa
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://f99d6d47.datachain-documentation.pages.dev |
| Branch Preview URL: | https://fix-477.datachain-documentation.pages.dev |
mattseddon
changed the title
| Order is not guaranteed when steps are added after an `order_by` statement. | ||
| I.e. when using `from_dataset` an `order_by` statement should be used if | ||
| the order of the records in the chain is important. | ||
| Using `order_by` directly before `limit` will give expected results. |
There was a problem hiding this comment.
I assume collect, show, to_parquet - are not considered as steps and will give also a "desired" output?
There was a problem hiding this comment.
collect - should but not guranteed (explanation in the original ticket #477)
show - yes
to_parquet - should but not guaranteed (uses collect_flatten)
There was a problem hiding this comment.
yep, probably needs to be a separate ticket then - e.g. combine things like order_by().collect() into a single statement, wdyt? (can be / should be implemented separately)
There was a problem hiding this comment.
Do you mean adding methods like collect_ordered OR adding an explicit order_by param to those methods or patching those functions? Is there a specific use case we're targeting or are we just trying to avoid confusion?
There was a problem hiding this comment.
I think it's a basic expectation that if you do order_by().collect() that you are getting a sorted result? collect is a way to iterate on the results I guess.
I mean that collect probably should be implemented similar to show() - something like ``order_by().collect()` should be a single SQL query probably
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #518 +/- ##
=======================================
Coverage 87.87% 87.87%
=======================================
Files 96 96
Lines 9873 9873
Branches 1349 1349
=======================================
Hits 8676 8676
Misses 857 857
Partials 340 340
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Closes #477