Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[spark] Fix docs, fix bug that spark sort for order should be non-global #2328

Merged
merged 4 commits into from
Nov 17, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/content/engines/spark3.md
Original file line number Diff line number Diff line change
Expand Up @@ -441,9 +441,9 @@ This section introduce all available spark procedures about paimon.
<tbody style="font-size: 12px; ">
<tr>
<td>compact</td>
<td><nobr>CALL [paimon.]sys.compact('&ltidentifier&gt','&ltpartitions&gt','&ltsort_type&gt','&ltcolumns&gt')</nobr><br>CALL [paimon.]sys.compact(table => '&ltidentifier&gt' [,partitions => '&ltpartitions&gt'] [, order_strategy =>'&ltsort_type&gt'] [,order_by => '&ltcolumns&gt'])</td>
<td><nobr>CALL [paimon.]sys.compact(table => '&ltidentifier&gt' [,partitions => '&ltpartitions&gt'] </nobr><br>[, order_strategy =>'&ltsort_type&gt'] [,order_by => '&ltcolumns&gt'])</td>
<td>identifier: the target table identifier. Cannot be empty.<br><br><nobr>partitions: partition filter. Left empty for all partitions.<br> "," means "AND"<br>";" means "OR"</nobr><br><br>order_strategy: 'order' or 'zorder' or 'none'. Left empty for 'none'. <br><br><nobr>order_columns: the columns need to be sort. Left empty if 'order_strategy' is 'none'. </nobr><br><br>If you want sort compact two partitions date=01 and date=02, you need to write 'date=01;date=02'<br><br>If you want sort one partition with date=01 and day=01, you need to write 'date=01,day=01'</td>
<td><nobr>SET spark.sql.shuffle.partitions=10; --set the sort parallelism</nobr> <nobr>CALL paimon.sys.compact('my_db.Orders1','f0=0,f1=1;f0=1,f1=1', 'zorder', 'f1,f2');</nobr><br><nobr>CALL paimon.sys.compact(table => 'T', partitions => 'p=0', order_strategy => 'zorder', order_by => 'a,b')</nobr></td>
<td><nobr>SET spark.sql.shuffle.partitions=10; --set the compact parallelism</nobr><br><nobr>CALL sys.compact(table => 'T', partitions => 'p=0', order_strategy => 'zorder', order_by => 'a,b')</nobr></td>
</tr>
</tbody>
</table>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,6 @@ public OrderSorter(FileStoreTable table, List<String> orderColNames) {
@Override
public Dataset<Row> sort(Dataset<Row> input) {
Column[] sortColumns = orderColNames.stream().map(input::col).toArray(Column[]::new);
return input.repartitionByRange(sortColumns).sort(sortColumns);
return input.repartitionByRange(sortColumns).sortWithinPartitions(sortColumns);
}
}
Loading