Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(core): Switch binary filesystem mode to nested path structure #7307

Merged
merged 330 commits into from
Oct 10, 2023
Merged

feat(core): Switch binary filesystem mode to nested path structure #7307

merged 330 commits into from
Oct 10, 2023

Conversation

ivov
Copy link
Contributor

@ivov ivov commented Sep 29, 2023 •
edited
Loading

Depends on #7253 | Story: PAY-863

This PR introduces filesystem-v2 to store binary data in the filesystem in the same format as s3.

@ivov
Install aws4 and @types/aws4
745437f
@ivov
Add to config schema
ccd3abb
@ivov
Set up object store service
afcc474
@ivov
Set up temp dev code
0f909b0
@ivov
Comment out cache keys
96ea9ea
@ivov
Revert "Comment out cache keys"
3d76b21 
This reverts commit 96ea9ea.
@ivov
Rename dir to TempBinaryData
455c327
@ivov
Rename back to BinaryData
65b4fb0
@ivov
Merge master
3bf3430
@ivov
Clear timers on shutdown
abe6d9d
@ivov
Set timers only on main instance
2682b3d
@ivov
Also for clearing timers
1c788fe
@ivov
Merge parent branch
f248982
@ivov
Rename back to localStoragePath
04f4b83
@ivov
Remove unused arg
03264f9
@ivov
Fix lint
a6e33d4
@ivov
Merge parent branch
fa09435
@ivov
Make as enterprise edition
552860a
@ivov
Add logging, refactor for readability
8823554
@ivov
Ensure hard-deletion select includes soft-deleted rows
a5def40
@ivov
Switch info to debug
77955e4
@ivov
Fix tests
b930e3e
@ivov
Remove redundant checks for deletedAt being NULL
e91f3de 
The column `deletedAt` is marked `@DeleteDateColumn()` so all reads
from the repository automatically add a `WHERE` clause checking
that the column `IS NULL`.
@ivov
Fix lint
2c3704d
@ivov
Fix last test
215f58e
@ivov
More setup
fabecea
@ivov
Add sample path structure
33344bc
@ivov
Set up some errors
70ab53b
@ivov
Remove unused
e22f0bf
@ivov
Cleanup
d5c8935
valya
valya previously approved these changes Oct 4, 2023
View reviewed changes
Copy link
Contributor

@valya valya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple questions but other than that, I think it's good.

packages/core/src/BinaryData/FileSystem.manager.ts Outdated Show resolved Hide resolved
packages/cli/src/workflows/workflows.services.ts Show resolved Hide resolved
@cypress
Copy link

cypress bot commented Oct 4, 2023 •
edited
Loading

Passing run #2444 ↗︎

0 250 0 0 Flakiness 0

Details:

🌳 pay-863-switch-binary-filesystem-mode-to-nested-path-structure 🖥️ browsers:n...
Project: n8n Commit: 3a1fe52195
Status: Passed Duration: 08:51 💡
Started: Oct 10, 2023 7:19 AM Ended: Oct 10, 2023 7:28 AM

Review all test suite changes for PR #7307 ↗︎

@github-actions
Copy link
Contributor

github-actions bot commented Oct 4, 2023

✅ All Cypress E2E specs passed

Base automatically changed from pay-848-integrate-object-store-service-as-binary-data-manager to master October 5, 2023 13:25
@ivov ivov dismissed valya’s stale review October 5, 2023 13:25

The base branch was changed.

krynble
krynble previously approved these changes Oct 5, 2023
View reviewed changes
Copy link
Contributor

@krynble krynble left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested the feature for backwards compatibility and new behavior, all working nicely! Thanks 💯

@github-actions
Copy link
Contributor

github-actions bot commented Oct 6, 2023

Great PR! Please pay attention to the following items before merging:

Files matching packages/**:

  • If fixing bug, added test to cover scenario.
  • If addressing forum or Github issue, added link to description.

Files matching packages/**/*.ts:

  • Added unit tests to cover new or updated functionality.

Make sure to check off this list before asking for review.

@github-actions
Copy link
Contributor

✅ All Cypress E2E specs passed

@ivov ivov merged commit 0847623 into master Oct 10, 2023
55 checks passed
@ivov ivov deleted the pay-863-switch-binary-filesystem-mode-to-nested-path-structure branch October 10, 2023 08:06
MiloradFilipovic added a commit that referenced this pull request Oct 10, 2023
@MiloradFilipovic
Merge branch 'master' into ADO-1148-remote-options-fix
c5cc371 
* master: (42 commits)
  fix(editor): Make workflow history button available only for dev builds (#7392)
  refactor: Upgrade to pnpm 8.9 (no-changelog) (#7393)
  ci: Identify NPM releases as `stable` (no-changelog)
  fix(editor): Implement canvas zoom UX improvements (#7376)
  feat(core): Switch binary filesystem mode to nested path structure (#7307)
  fix(editor): Fix completions for `.json` on quoted node name in Code node (#7382)
  fix(Notion Node): Handle empty values correctly for Notion selects + multi selects (#7383)
  feat(editor): Make PDF and Audio binary-data viewable in the UI (#7367)
  fix(Google Sheets Node): Fix "Maximum call stack size exceeded" error on too many rows (#7384)
  refactor(core): Refactor nodes loading (no-changelog) (#7283)
  fix(core): Add an option to enable postgres ssl with default certs (#6889)
  feat(editor): Workflow history [WIP]- Add restore and clone into new workflow actions (no-changelog) (#7359)
  fix(HTML Node): Update property fields to not use expressions on drag (#7379)
  fix: Add role check for upgrade path (#7374)
  fix(editor): Remove excess margin below run data editor (#7372)
  fix(Google Drive Trigger Node): Add Shared Drives support (#7369)
  feat(core): Add Job Summary to Worker response (#7360)
  feat(Execute Workflow Node): Run once for each item mode (#7289)
  refactor(core): Move `copyInputItems` to node helpers (no-changelog) (#7299)
  refactor(core): Create controller for binary data (no-changelog) (#7363)
  ...

# Conflicts:
#	cypress/e2e/5-ndv.cy.ts
@github-actions github-actions bot mentioned this pull request Oct 11, 2023
Merged
netroy added a commit that referenced this pull request Oct 11, 2023
@github-actions @netroy
🚀 Release 1.11.0 (#7402)
e6ba841 
# [1.11.0](https://github.com/n8n-io/n8n/compare/n8n@1.10.0...n8n@1.11.0)
(2023-10-11)


### Bug Fixes

* **core:** Add an option to enable postgres ssl with default certs
([#6889](#6889))
([789e1e7](789e1e7))
* **core:** Fix error on missing paired item data
([#7399](#7399))
([47e8953](47e8953))
* **core:** Missing pairing info
([#7326](#7326))
([e2c3c7a](e2c3c7a))
* **core:** Prevent object deletion request on no prefix match
([#7366](#7366))
([63e11e4](63e11e4))
* **editor:** Fix completions for `.json` on quoted node name in Code
node ([#7382](#7382))
([86e7ec7](86e7ec7))
* **editor:** Implement canvas zoom UX improvements
([#7376](#7376))
([7e06b31](7e06b31))
* **editor:** Make workflow history button available only for dev builds
([#7392](#7392))
([7ed466d](7ed466d))
* **editor:** Remove excess margin below run data editor
([#7372](#7372))
([3fa2764](3fa2764))
* **editor:** Sanitize HTML binary-data before rendering in the UI
([#7400](#7400))
([2b075bf](2b075bf))
* **editor:** Use display option's @Version specifier
([#7351](#7351))
([afbf0c3](afbf0c3))
* **Google BigQuery Node:** Location default to jobReference
([#7354](#7354))
([97bb703](97bb703))
* **Google Drive Trigger Node:** Add Shared Drives support
([#7369](#7369))
([3e7a4d3](3e7a4d3))
* **Google Sheets Node:** Fix "Maximum call stack size exceeded" error
on too many rows ([#7384](#7384))
([732b15a](732b15a))
* **HTML Node:** Update property fields to not use expressions on drag
([#7379](#7379))
([77643e5](77643e5))
* **Notion Node:** Handle empty values correctly for Notion selects +
multi selects ([#7383](#7383))
([fbcd1d4](fbcd1d4))
* **Set Node:** Increase search priority
([#7358](#7358))
([e5ad1e7](e5ad1e7))
* **Webhook Node:** Backward compatible form-data parsing for non-array
files ([#7385](#7385))
([6479eb1](6479eb1))


### Features

* **core:** Add Job Summary to Worker response
([#7360](#7360))
([b8608ce](b8608ce))
* **core:** Integrate object store as binary data manager
([#7253](#7253))
([1a661e6](1a661e6))
* **core:** Switch binary filesystem mode to nested path structure
([#7307](#7307))
([0847623](0847623))
* **editor:** Make PDF and Audio binary-data viewable in the UI
([#7367](#7367))
([8187be1](8187be1))
* **editor:** Support autologin for upgrade path
([#7316](#7316))
([1dfa052](1dfa052))
* **Execute Workflow Node:** Run once for each item mode
([#7289](#7289))
([c8c14ca](c8c14ca))
* **Item Lists Node:** Split merge binary data
([#7297](#7297))
([965db8f](965db8f))
* **Loop Over Items (Split in Batches) Node:** Automatically add a loop
+ rename ([#7228](#7228))
([7b773cc](7b773cc))
* **Notion Node:** Fetch child blocks recursively
([#7304](#7304))
([193181a](193181a))

Co-authored-by: netroy <netroy@users.noreply.github.com>
@janober
Copy link
Member

janober commented Oct 11, 2023

Got released with n8n@1.11.0

elsmr pushed a commit that referenced this pull request Oct 19, 2023
@github-actions @netroy @elsmr
🚀 Release 1.11.0 (#7402)
f10b5f0 
# [1.11.0](https://github.com/n8n-io/n8n/compare/n8n@1.10.0...n8n@1.11.0)
(2023-10-11)


### Bug Fixes

* **core:** Add an option to enable postgres ssl with default certs
([#6889](#6889))
([789e1e7](789e1e7))
* **core:** Fix error on missing paired item data
([#7399](#7399))
([47e8953](47e8953))
* **core:** Missing pairing info
([#7326](#7326))
([e2c3c7a](e2c3c7a))
* **core:** Prevent object deletion request on no prefix match
([#7366](#7366))
([63e11e4](63e11e4))
* **editor:** Fix completions for `.json` on quoted node name in Code
node ([#7382](#7382))
([86e7ec7](86e7ec7))
* **editor:** Implement canvas zoom UX improvements
([#7376](#7376))
([7e06b31](7e06b31))
* **editor:** Make workflow history button available only for dev builds
([#7392](#7392))
([7ed466d](7ed466d))
* **editor:** Remove excess margin below run data editor
([#7372](#7372))
([3fa2764](3fa2764))
* **editor:** Sanitize HTML binary-data before rendering in the UI
([#7400](#7400))
([2b075bf](2b075bf))
* **editor:** Use display option's @Version specifier
([#7351](#7351))
([afbf0c3](afbf0c3))
* **Google BigQuery Node:** Location default to jobReference
([#7354](#7354))
([97bb703](97bb703))
* **Google Drive Trigger Node:** Add Shared Drives support
([#7369](#7369))
([3e7a4d3](3e7a4d3))
* **Google Sheets Node:** Fix "Maximum call stack size exceeded" error
on too many rows ([#7384](#7384))
([732b15a](732b15a))
* **HTML Node:** Update property fields to not use expressions on drag
([#7379](#7379))
([77643e5](77643e5))
* **Notion Node:** Handle empty values correctly for Notion selects +
multi selects ([#7383](#7383))
([fbcd1d4](fbcd1d4))
* **Set Node:** Increase search priority
([#7358](#7358))
([e5ad1e7](e5ad1e7))
* **Webhook Node:** Backward compatible form-data parsing for non-array
files ([#7385](#7385))
([6479eb1](6479eb1))


### Features

* **core:** Add Job Summary to Worker response
([#7360](#7360))
([b8608ce](b8608ce))
* **core:** Integrate object store as binary data manager
([#7253](#7253))
([1a661e6](1a661e6))
* **core:** Switch binary filesystem mode to nested path structure
([#7307](#7307))
([0847623](0847623))
* **editor:** Make PDF and Audio binary-data viewable in the UI
([#7367](#7367))
([8187be1](8187be1))
* **editor:** Support autologin for upgrade path
([#7316](#7316))
([1dfa052](1dfa052))
* **Execute Workflow Node:** Run once for each item mode
([#7289](#7289))
([c8c14ca](c8c14ca))
* **Item Lists Node:** Split merge binary data
([#7297](#7297))
([965db8f](965db8f))
* **Loop Over Items (Split in Batches) Node:** Automatically add a loop
+ rename ([#7228](#7228))
([7b773cc](7b773cc))
* **Notion Node:** Fetch child blocks recursively
([#7304](#7304))
([193181a](193181a))

Co-authored-by: netroy <netroy@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@valya valya valya left review comments

@krynble krynble krynble approved these changes

Assignees
No one assigned
Labels
core Enhancement outside /nodes-base and /editor-ui n8n team Authored by the n8n team Released
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ivov @janober @krynble @valya