Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(sqllab): empty large query results from localStorage #23302

Merged
merged 3 commits into from
Mar 13, 2023
Merged

fix(sqllab): empty large query results from localStorage #23302

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4d0de19
fix(sqllab): empty large query results from localStorage
justinpark Mar 7, 2023
74ff22b
retest e2e
justinpark Mar 7, 2023
194d928
replace test spec by constants
justinpark Mar 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions superset-frontend/src/SqlLab/constants.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ export const BYTES_PER_CHAR = 2;
// browser's localStorage max usage constants
export const LOCALSTORAGE_MAX_QUERY_AGE_MS = 24 * 60 * 60 * 1000; // 24 hours
export const LOCALSTORAGE_MAX_USAGE_KB = 5 * 1024; // 5M
export const LOCALSTORAGE_MAX_QUERY_RESULTS_KB = 1 * 1024; // 1M
export const LOCALSTORAGE_WARNING_THRESHOLD = 0.9;
export const LOCALSTORAGE_WARNING_MESSAGE_THROTTLE_MS = 8000; // danger type toast duration

Expand Down
37 changes: 37 additions & 0 deletions superset-frontend/src/SqlLab/utils/emptyQueryResults.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,11 @@ import {
import { LOCALSTORAGE_MAX_QUERY_AGE_MS } from 'src/SqlLab/constants';
import { queries, defaultQueryEditor } from '../fixtures';

jest.mock('src/SqlLab/constants', () => ({
...jest.requireActual('src/SqlLab/constants'),
LOCALSTORAGE_MAX_QUERY_RESULTS_KB: 1,
}));

describe('reduxStateToLocalStorageHelper', () => {
const queriesObj = {};
beforeEach(() => {
Expand All @@ -45,6 +50,38 @@ describe('reduxStateToLocalStorageHelper', () => {
expect(emptiedQuery[id].results).toEqual({});
});

it('should empty query.results if query,.results size is greater than LOCALSTORAGE_MAX_QUERY_RESULTS_KB', () => {
const reasonableSizeQuery = {
...queries[0],
startDttm: Date.now(),
results: { data: [{ a: 1 }] },
};
const largeQuery = {
...queries[1],
startDttm: Date.now(),
results: {
data: [
{
test: 1123123123123,
stringValue:
'Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industrys standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can maybe just create a very large array and fill?

new Array(1024*512+1).fill(0).join("")

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow. it's smart.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated with this suggestion

jsonValue:
'{"a":1234234234234234,"b":21234123412341234,"c":31234123412341234,"str":"something long text goes here"}',
},
],
},
};
expect(Object.keys(largeQuery.results)).toContain('data');
const emptiedQuery = emptyQueryResults({
[reasonableSizeQuery.id]: reasonableSizeQuery,
[largeQuery.id]: largeQuery,
});
expect(emptiedQuery[largeQuery.id].results).toEqual({});
expect(emptiedQuery[reasonableSizeQuery.id].results).toEqual(
reasonableSizeQuery.results,
);
});

it('should only return selected keys for query editor', () => {
const queryEditors = [defaultQueryEditor];
expect(Object.keys(queryEditors[0])).toContain('schemaOptions');
Expand Down
21 changes: 17 additions & 4 deletions superset-frontend/src/SqlLab/utils/reduxStateToLocalStorageHelper.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,12 @@
* specific language governing permissions and limitations
* under the License.
*/
import { LOCALSTORAGE_MAX_QUERY_AGE_MS } from '../constants';
import {
BYTES_PER_CHAR,
KB_STORAGE,
LOCALSTORAGE_MAX_QUERY_AGE_MS,
LOCALSTORAGE_MAX_QUERY_RESULTS_KB,
} from '../constants';

const PERSISTENT_QUERY_EDITOR_KEYS = new Set([
'remoteId',
Expand All @@ -36,13 +41,21 @@ const PERSISTENT_QUERY_EDITOR_KEYS = new Set([
'hideLeftBar',
]);

function shouldEmptyQueryResults(query) {
const { startDttm, results } = query;
return (
Date.now() - startDttm > LOCALSTORAGE_MAX_QUERY_AGE_MS ||
((JSON.stringify(results)?.length || 0) * BYTES_PER_CHAR) / KB_STORAGE >
LOCALSTORAGE_MAX_QUERY_RESULTS_KB
);
}

export function emptyQueryResults(queries) {
return Object.keys(queries).reduce((accu, key) => {
const { startDttm, results } = queries[key];
const { results } = queries[key];
const query = {
...queries[key],
results:
Date.now() - startDttm > LOCALSTORAGE_MAX_QUERY_AGE_MS ? {} : results,
results: shouldEmptyQueryResults(queries[key]) ? {} : results,
};

const updatedQueries = {
Expand Down