Spreadsheet view, sorting, grouping #363

pavish · 2021-07-14T16:26:29Z

Related to #343, #344

This PR contains:

Spreadsheet layout with virtual scroll
Feching data based on scroll position
Single column sort with url sync
Basic grouping without count

Note:

The implementation for sorting and grouping are basic and not fully complete. These basic functionalities were implemented in the same PR, inorder to handle layout changes which are essential for them.
THIRDPARTY file has been added with references to components that we derive our virtual list component from. The file itself contains the MIT license text. @kgodey Please let me know if we also need to include a separate LICENSES folder.

Checklist

My pull request has a descriptive title (not a vague title like Update index.md).
My pull request targets the master branch of the repository
My commit messages follow best practices.
My code follows the established code style of the repository.
I added tests for the changes I made (if applicable).
I added or updated documentation (if applicable).
I tried running the project locally and verified that there are no
visible errors.

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

…frontend_sorting

…optimize render during scroll

…frontend_sorting

dhruvkb

Not a formal review, but I'm curious about why did the third party code was copied over rather than installed via npm and imported?

pavish · 2021-07-16T13:19:56Z

@dhruvkb They were not copied, they were ported from React to Svelte, and stripped down to bare essentials. We also have additional functionality over them to better suit our needs.

dhruvkb · 2021-07-16T13:27:07Z

Isn't react-window itself a port of svelte-window?

pavish · 2021-07-16T14:18:59Z

@dhruvkb react-window is the original work, it was created by the same author who created react-virtualized.

svelte-window is a port of react-window (It's mentioned on the readme of svelte-window). Our component is also a port of react-window.

kgodey

@pavish I only reviewed the licensing portions of the PR, I'll review the functionality separately.

kgodey · 2021-07-16T16:00:24Z

README.md

@@ -4,6 +4,7 @@ Mathesar is a project to make databases easier to use for non-technical users. O

 We are currently in early development and hope to release an alpha version by late 2021. Please see the [Mathesar wiki](https://wiki.mathesar.org/) for more information about the project.

+Mathesar is open source under the GPLv3 license - see [LICENSE](LICENSE). It also contains derivatives of third-party open source modules. See the list and respective licenses in [THIRDPARTY](THIRDPARTY).


Suggested change

Mathesar is open source under the GPLv3 license - see [LICENSE](LICENSE). It also contains derivatives of third-party open source modules. See the list and respective licenses in [THIRDPARTY](THIRDPARTY).

Mathesar is open source under the GPLv3 license - see [LICENSE](LICENSE). It also contains derivatives of third-party open source modules licensed under the MIT license. See the list and respective licenses in [THIRDPARTY](THIRDPARTY).

Please move this under a "License" header at the bottom of the file.

Changes made in 2e798b1

kgodey · 2021-07-16T16:01:22Z

THIRDPARTY

+Mathesar contains derivatives of the following components that are subject to 
+separate license terms. Your use of these derivatives is subject to the separate
+license terms applicable to each component, in addition to Mathesar's license.
+The full text of all referenced licenses is appended at the end of this file.


I think it's better to have a LICENSES folder where we copy the actual LICENSE file from the project that includes the copyright information.

I think that would best meet this clause from the MIT license.

The above copyright notice and this permission notice (including the next
paragraph) shall be included in all copies or substantial portions of the
Software.

Changes made in 2e798b1

kgodey · 2021-07-16T16:02:49Z

THIRDPARTY

+    /mathesar_ui/src/sections/table-view/virtual-list/detectElementResize.ts
+
+
+ALL LICENSE TEXTS:


Please remove this when you add the LICENSES folder.

Changes made in 2e798b1

kgodey · 2021-07-16T16:09:14Z

The functionality looks good to me! It's nice to be able to do things with the data, the grouping seems to work well.

It's confusing to have to click on "Group by Column" to ungroup the data, but I assume that's just a consequence of the implementation not being complete.

I'd very much like @zackkrida to take a look at this PR since it's a big frontend structural change.

…frontend_sorting

kgodey

Looks good on the license front.

@zackkrida Can you get to this today? If not, I'll merge it and we can address review comments in future PRs.

dhruvkb · 2021-07-20T04:47:42Z

From our A8C discussions, @zackkrida is AFK for the coming few days (not sure why I didn't remember this sooner, he's been AFK last week as well). I would recommend merging to prevent blocking further work.

kgodey · 2021-07-20T10:22:30Z

Thanks @dhruvkb! Merged.

zackkrida · 2021-07-21T03:38:47Z

@kgodey @pavish Some retroactive thoughts:

This is lovely work! I noticed a small alignment issue that is resolved by removing overflow: hidden from the .table-content class:

before

after

Otherwise, virtualization and server-loaded pagination seem to be working well together. I'd like to see the initial result count loaded much larger, and perhaps even a larger 'buffer' of items visualized above and below the virtualized items currently in the viewport. I do feel like while browsing through a large table I see a lot of empty rows

I don't think it's unreasonable to load 100s or even 1000s of rows initially on page load, but these sorts of # are going to need to be continually updated anyway to best balance perceived user performance with metrics like the average row size, column count, row count, etc.

pavish · 2021-07-21T06:42:13Z

Hey @zackkrida , thank you so much :)

Regarding the alignment issue, could you please share the browser you're using and it's version. The table-content class is not supposed to allow overflow, I'll need to figure out what is causing the issue.

On data loading, I agree that there are a lot of empty rows while scrolling. Currently, it works the following way:

An initial set of 50 rows are loaded, total count is obtained in the result, let's say 160. We render the 50 filled rows, and 110 empty rows.
When user scrolls, and stops at a position, we identify the range and request the server for the empty rows in that range, along with around 10 buffer rows above and below.
- If there are no empty rows in the range, request is not sent.
- If there are only a few empty rows in the range, request is only sent for them.
We cache the result on the client after each request.
It might take a few milliseconds in showing the rows, due to network delay. We can alleviate this using websockets to an extent.
However, while scrolling is in motion, we do not send any requests. In this case, empty rows would be visible. I'm still trying to figure out a way to handle this.
- Loading all the 'out of range' rows in the background when a table is open, is not a good idea if the count is very large.
- Any suggestion on this would be very helpful.

pavish added 18 commits July 12, 2021 16:07

Sort table on clicking header

11be7d9

Merge branch 'master' of https://github.com/centerofci/mathesar into …

220afba

…frontend_sorting

URL sync for sorting

ae734ae

Grouping records by columns

1289d00

VirtualList Implementation

99d5e31

Added virtual scroll to existing table

6f1425d

Perfect scrollbar used with virtual scroll for better control and to …

40d0181

…optimize render during scroll

Merge branch 'master' of https://github.com/centerofci/mathesar into …

66faefb

…frontend_sorting

Fetch table data on scroll

d54000a

Preserve scroll positions on tab change

f433e86

Merge branch 'master' of https://github.com/centerofci/mathesar into …

cd9ad33

…frontend_sorting

Sorting functionality for spreadsheet

e6ec5a9

Fetch empty records on height change

ea0f484

Exporting itemKey closure function prop

35afca8

Cancelling previous record requests upon full reload

46e3a84

Basic grouping without count

7e6df6d

Reset groupIndex on reload

cd324ec

Thirdparty information

5e06ad3

pavish changed the title ~~Spreadsheet view, sorting~~ Spreadsheet view, sorting, grouping Jul 16, 2021

pavish marked this pull request as ready for review July 16, 2021 13:08

pavish requested review from a team, kgodey and mathemancer July 16, 2021 13:08

github-actions bot requested review from dhruvkb, eito-fis, ghislaineguerin and zackkrida July 16, 2021 13:08

dhruvkb reviewed Jul 16, 2021

View reviewed changes

kgodey requested changes Jul 16, 2021

View reviewed changes

pavish added 2 commits July 19, 2021 11:13

Merge branch 'master' of https://github.com/centerofci/mathesar into …

2471011

…frontend_sorting

Updated thirdparty licenses

2e798b1

pavish force-pushed the frontend_sorting branch from 92018be to 2e798b1 Compare July 19, 2021 06:46

pavish requested a review from kgodey July 19, 2021 06:49

Merge branch 'master' of https://github.com/centerofci/mathesar into …

091c91a

…frontend_sorting

kgodey approved these changes Jul 19, 2021

View reviewed changes

Merge branch 'master' into frontend_sorting

5b53112

kgodey merged commit 0211da8 into master Jul 20, 2021

kgodey deleted the frontend_sorting branch July 20, 2021 10:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spreadsheet view, sorting, grouping #363

Spreadsheet view, sorting, grouping #363

pavish commented Jul 14, 2021 •

edited

Loading

dhruvkb left a comment

pavish commented Jul 16, 2021 •

edited

Loading

dhruvkb commented Jul 16, 2021

pavish commented Jul 16, 2021

kgodey left a comment

kgodey Jul 16, 2021

kgodey Jul 16, 2021

pavish Jul 19, 2021

kgodey Jul 16, 2021

kgodey Jul 16, 2021

pavish Jul 19, 2021

kgodey Jul 16, 2021

pavish Jul 19, 2021

kgodey commented Jul 16, 2021

kgodey left a comment

dhruvkb commented Jul 20, 2021

kgodey commented Jul 20, 2021

zackkrida commented Jul 21, 2021

pavish commented Jul 21, 2021

		@@ -4,6 +4,7 @@ Mathesar is a project to make databases easier to use for non-technical users. O

		We are currently in early development and hope to release an alpha version by late 2021. Please see the [Mathesar wiki](https://wiki.mathesar.org/) for more information about the project.

		Mathesar is open source under the GPLv3 license - see [LICENSE](LICENSE). It also contains derivatives of third-party open source modules. See the list and respective licenses in [THIRDPARTY](THIRDPARTY).

		/mathesar_ui/src/sections/table-view/virtual-list/detectElementResize.ts


		ALL LICENSE TEXTS:

Spreadsheet view, sorting, grouping #363

Spreadsheet view, sorting, grouping #363

Conversation

pavish commented Jul 14, 2021 • edited Loading

Checklist

Developer Certificate of Origin

dhruvkb left a comment

Choose a reason for hiding this comment

pavish commented Jul 16, 2021 • edited Loading

dhruvkb commented Jul 16, 2021

pavish commented Jul 16, 2021

kgodey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kgodey commented Jul 16, 2021

kgodey left a comment

Choose a reason for hiding this comment

dhruvkb commented Jul 20, 2021

kgodey commented Jul 20, 2021

zackkrida commented Jul 21, 2021

pavish commented Jul 21, 2021

pavish commented Jul 14, 2021 •

edited

Loading

pavish commented Jul 16, 2021 •

edited

Loading