-
-
Notifications
You must be signed in to change notification settings - Fork 46
Re-add SQLite support #1527
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Re-add SQLite support #1527
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work @snowteamer! Some more minor issues and after these are addressed I think it should be good to go.
backend/database-sqlite.js
Outdated
reject(err) | ||
} | ||
}) | ||
db.run('CREATE TABLE IF NOT EXISTS Data(key TEXT UNIQUE NOT NULL, value TEXT NOT NULL)', [], (err) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What was the verdict on adding a third column for metadata for dealing with lastModified
header for fs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That turned out to be a bit complex because lastModified
has to be stored somewhere and kept up-to-date for all three storage backends, and also because we want a consistent API regardless the data type but only want that field for files.
|
||
export function exportToJSON (): Promise<Object> { | ||
return new Promise((resolve, reject) => { | ||
db.all('SELECT key, value FROM Data', [], (err, rows) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This would read the entire database contents into memory, right? That seems like an extreme amount of data to read into memory (potentially hundreds of gigabytes worth). What is this function for?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Copying a note from @snowteamer from Slack about what this is:
These are database migration helpers, which I was planning to maybe use either in the preloading code or in chel migrate.
I've left them here so that they can be unit-tested in database.test.js and reviewed, but yeah actually they are not used (at least yet).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work @snowteamer! I left a few comments.
Also, I tried testing this with GI_PERSIST=sqlite grunt dev
, and ran into two problems:
- I couldn't load the site, neither on port
:3000
nor:8000
- I didn't see the message
[chelonia.db] Preloaded ${newKeys.length} new entries
in the console either
This is (some of) the output I got:
Running "backend:relaunch" task
backend: forking...
Running "keepalive" task
[Browsersync] Proxying: http://127.0.0.1:8000
[Browsersync] Access URLs:
----------------------------------------
Local: http://localhost:3000
External: http://192.168.100.201:3000
----------------------------------------
UI: http://localhost:3002
UI External: http://localhost:3002
----------------------------------------
[Browsersync] Watching files...
[SBP WARN]: registering unsafe selector: 'chelonia/db/get' (remember to lock after overwriting)
[SBP WARN]: registering unsafe selector: 'chelonia/db/set' (remember to lock after overwriting)
[SBP WARN]: registering unsafe selector: 'chelonia/db/delete' (remember to lock after overwriting)
Connected to the groupincome.db SQLite database.
[SBP WARN]: registering unsafe selector: 'chelonia/db/get' (remember to lock after overwriting)
[SBP WARN]: registering unsafe selector: 'chelonia/db/set' (remember to lock after overwriting)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approved! Great work @snowteamer!!
* Require login with invite links. Fix bug with next parameter in login guard. * Stop Cypress upon first failed spec (#1528) * Substitute shareable keys * Issues in Modal and Tab (#1519) * feat: removed padding in modal tablet and mobile view * fix: error in unexpected clickable areas in tabWrapper * fix: error in setting clickable area to upload avatar * feat: navigation style updated * fix: position error of textarea when page is full of messages * chore: removed unnecessary comment and travis retry * feat: added i18n * feat: set the correct version * feat: created acknowledge page * feat: made acknowledgements page * feat: added correct urls and travis retry * chore: travis retry * feat: make animation when message is deleted * feat: optimize Promise * feat: added test case for animation of deleting message * feat: resolved conflicts * feat: added initiating spinner * feat: optimized codes * feat: added comment for better understanding * chore: added TODO comment * Wrong due date shown in UI (#1533) * feat: fix confusion in dueDate * feat: display nextDistributionDate in DateTime format * chore: revert changes in the previous version of group contract * feat: optimize to get nextDistributionDateTime * feat: fix error in working with toLocaleDateString * feat: remove unnecessary updates * chore: travis retry * Updated OP_KEY_ADD to keep track of key requests * Re-add SQLite support (#1527) * Add sqlite support * Apply review feedback * Remove unnecessary try statements * Use readFile and writeFile from fs/promises * Fix an inconsistency and some Flow typings * Make 'chelonia/db/set' return Promise<void> * Add storage backend unit tests * Use prepared statements for performance * Update .travis.yml to use Ubuntu Xenial * fixup! Use prepared statements for performance * Use path.join in readString * Drop 'backend/db/readFile' and '-writeFile' * Rename to x-backend.js files to database-x.js * Update Node to v18 LTS * Store uploads in Chelonia DB * Validate args in 'chelonia/db/set' * Bump Ubuntu to v20 LTS in Travis * Fix Flow errors * Remove initDB() call in backend.test.js * Add DB import and export functions * Remove 'GET contract/' route * Use primary key * Improve Chelonia DB preloading * Disallow unprintable characters in keys * Use sequential preloading and skip avatars * Chelonia dashboard frontend (#1483) * create a new Gruntfile.js for dashboard * add various placeholder files and assets in backend/dashboard folder * configure & register 'exec'/'clean'/'copy' tasks * add fonts / create a task 'esbuild' * split builds into mainJs and mainCss * setup browser-sync watch * fix the linter error * few bug-fixes regarding esbuild css / modify backend/routes.js / make linters ignore /dist-dashboard * basic scss / Vue-router setup * turn off browsersync 'ws' option for console error * prep build script / style router-link for match * setup flowRemoveTypesPlugin * remove unused alias * ignore backend/dashboard for now for flow type check * add type annotation for router obj * random comment for cypress failure * random comment change * work on some of the change requests * add translation to the project * temp setup for fetching translation data / build config * create 'prep-translation' task * cleanup some linter error * simplify translation-related build flow * remove cross-env * re-enable translation for development * random comment update for flaky cypress test * fix the linter error * init Vuex / prep theming setup / replace 'Lato' to 'Inter' * replace icons from font-awesome to chelonia icons * add Toolbar.vue / prep custom-variables setup for dark-theme * link hamburger btn in toolbar with navigation menu * add PageTemplate.vue and replace things with it * build StatCard.vue / add some placeholder to Dashboard.vue * some cleanup work * fill the /dashboard page with dummy data and make sure the page is responsive to the viewport * design improvements for some UIs / add theming to the app * create a skeleton of /design-system page * create Dropdown.vue component * style the dropdown / place the dropdown on /contracts page * create some dummy data for /contracts page * add contract table to /contracts page * length 0 with no unit * make the contracts filter work in /contracts page * create modal skeleton / attach MODAL_OPEN, MODAL_CLOSE event listeners * make modal skeleton work with sbp event call * add style to ModalTemplate.vue * create a custom-pre UI and add that to the modal * complete modal-template design * build TextToCopy.vue skeleton * build TextToCopy.vue and add it to the modal * Implement /users page * Build StyledInput.vue and add css to it * run translation * add vError.js * create .radio class with basic styles * add dropdown to Accounts.vue * build InfoCard.vue / add it to the Accounts.vue page * setup form validation * couple of css bug-fixes * work on Billing.vue page * build a skeleton loading screen / make sure theming works in there * some work to the landing page * draw bar graphs * remove the typo * complete landing page animation * Landing.vue done / page-transition to <router-view /> * add Error.vue * implement lazy loading of the page components * adjust route basepath based on the build type * add pointer event effect * remove unused font files * work for some of the Greg's change requests * build Tooltip.vue and add it to Accounts page * update Error page * re-run translation * improve Tooltip.vue * fix typo * random change * work on alex review * fix the theme bug regarding the system preference * update translation * fix the linter error * random comment change for cypress haisenbug * missing fix * remove console.log * comment addition for a heisenbug * change request * Save pending joins to login state (broken) * Use prefixes only in 'chel/db/get', not in DB keys (#1552) * latest changes * Checkpoint: contract sync joins chatroom when logging in again * Recv Keys Event, group event listener * fix unnecessary CLOSE_EMOTICON event (#1561) * New design to create DM and optimize ChatMixin (#1534) * chore: removed breaklines * feat: focus and enter key press in CreateNewChannelModal * fix: error in handling undefined displayName * fix: style and submit action in search component * feat: default focus and submit action in EditChannelNameModal * feat: default focus on EditChannelDescriptionModal * fix: error in handling displayName * chore: simplified function to filter users * fix: DRY to filter users * feat: declare types * fix: skip unnecessary profile changes * feat: removed unncessary function * chore: added comments * chore: remove break lines * chore: updated comment * fix: error when loading private chatroom which is not part of * feat: optimized GroupChat redirects * feat: updated cypress assert for incorrect chatroom url * fix: errors in chat unread message and init MultiSearch component * feat: multiuserselect component step 1 * feat: hover effect in user comp in multiuserselector * chore: removed unnecessary function * feat: added keydown and keyup event handler * feat: MultiUserSelector step 2 completed * feat: filter two types of users in ChatMembers Modal * chore: removed unnecessary condition * feat: fix filtering users inNewDMModal * feat: wrong number of users in titles * chore: renamed component * feat: added autofocus and submit to UsersSelector component * feat: added focus style in UsersSelector and use it in NewDMModal * feat: fit the height * feat: added submit action in NewDMModal * feat: optimize UsersSelector component * feat: clear after selection * feat: open DM only when no users selected * feat: optimized NewDMModal * feat: openPriveDM only after users type keyword * feat: created DMMixin * feat: fix cypress error regarding new workflow * feat: implemented several ways to create and open DM in cypress test scenario * Optimize Chatmixin (#1542) * feat: removed padding in modal tablet and mobile view * fix: error in unexpected clickable areas in tabWrapper * fix: error in setting clickable area to upload avatar * feat: navigation style updated * fix: position error of textarea when page is full of messages * chore: removed unnecessary comment and travis retry * feat: added i18n * feat: set the correct version * feat: created acknowledge page * feat: made acknowledgements page * feat: added correct urls and travis retry * chore: travis retry * feat: make animation when message is deleted * feat: optimize Promise * feat: added test case for animation of deleting message * feat: resolved conflicts * feat: added initiating spinner * feat: optimized codes * feat: optimize ChatMixin Step 1 * feat: init members list on mounted * wip: setUnreadPosition while joining * feat: set unread position while syncing * feat: optimized ChatMixin Step 2 * fix: error of loading messages twice * chore: fixed DRY * fix: error in readUntil position after editMessage * chore: added comment and travis retry * fix: error in working with previous version of chatroom contract * feat: added box-shadow in card when it's dark them * chore: removed codes related to the Archive Channel feature * fix: travis error * chore: removed duplicated file * fix: travis error * fix: removed TODO * feat: removed unnecessary Optional Chaining * feat: fix travis error in the bypass mode * feat: clear vuex state on logout * fix: travis error regarding syncing contracts * fix: error in setting finishedLogin state * chore: reverted change regarding unnecessary finishedLogin in vuex * chore: reverted await using in identity login process * feat: init finishedLogin when logout * feat: update UnreadPosition only joining & first signing-in * chore: optimized and renamed for better understanding * feat: removed html-entities library * chore: added comment saying why v-model couldn't be used * chore: removed unnecessary attribute in profile wrapper * feat: set cursors * chore: removed duplicated style * feat: set initvalue when edit non-monetary pledge * feat: renamed function in order to reduce confusion * feat: removed syncContract and used isNewSyncing instead * feat: removed unreachable condition * chore: simplified chatroom contract * feat: added comment * feat: update rule to set order of recent dms * feat: removed isNewSyncing and improved isSyncing instead * fix: error in parameter declaration * feat: simplified codes * chore: revert box-shadow of card-component in dark-mode to * fix: error in using infinite loading --------- Co-authored-by: Ricardo Iván Vieitez Parra <ricardo@exactrealty.co.uk> Co-authored-by: snowteamer <64228468+snowteamer@users.noreply.github.com> Co-authored-by: Alex Jin <57976479+Silver-IT@users.noreply.github.com> Co-authored-by: Ricardo Iván Vieitez Parra <3857362+corrideat@users.noreply.github.com> Co-authored-by: Sebin Song <songxevin@gmail.com>
Closes #1484
Summary of changes
FormData
andBlob
support in backend tests without pulling in new dependencies).blob:${hash}
keys, whereas previously they were always stored in the./data
folder. This means that file uploads will no longer be persisted unless persistence is actually enabled.GI_PERSIST
can now be eitherfs
orsqlite
.fs
ifGI_PERSIST
is not set or empty.name:
prefix is now used in namespace keys instead ofname=
.head:
prefix is now used in contract log head keys instead of a-HEAD
suffix.'backend/db/readFile'
,'-writeFile'
, and'-writeFileOnce'
selectors have been removed.backend/database.js
module must now be initialized before use.test/database.test.js
. These tests apply to both thefs
andsqlite
backends in a single run, so that you do not have to do multiple test runs with differentGI_PERSIST
values.