Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File System Watcher #28593

Closed
Tracked by #28378
damen-dotcms opened this issue May 17, 2024 · 12 comments · Fixed by #28710, #29067 or #29353
Closed
Tracked by #28378

File System Watcher #28593

damen-dotcms opened this issue May 17, 2024 · 12 comments · Fixed by #28710, #29067 or #29353

Comments

@damen-dotcms
Copy link
Contributor

damen-dotcms commented May 17, 2024

Parent Issue

#28378

Task

We need some sort of tool to monitor file systems for changes, and trigger a push when a change is detected.

The idea is to include a new parameter --watch to the existing push commands (the global push and the specific ones).

After a brainstorming session, we agreed the following:

  • Use Java nio watch service where possible to be notified of changes rather than polling and scanning the folder every time. The watch service only works if the OS supports it and on linux does not work on separate disk, so the code should fallback to polling if the Watch Service is not available.
  • When a new notification is received, we should wait 2 seconds by default before running the push logic to make sure there are no further changes over a minimum time. This would prevent the case where there are logs of changes and we end up having to run the sync multiple times where it could be covered with the once if we waited for a few seconds. The waiting time can be set using a parameter as well, and if not defined, it will default to 2 secs.
  • No explicit path will be provided, the service will use the same path being used by the push command (it includes the case when the --workspace is set)

Proposed Objective

Same as Parent Issue

Proposed Priority

Same as Parent Issue

Acceptance Criteria

Essentially we're aiming for automated deployment capabilities using the CLI.

External Links... Slack Conversations, Support Tickets, Figma Designs, etc.

No response

Assumptions & Initiation Needs

No response

Quality Assurance Notes & Workarounds

No response

Sub-Tasks & Estimates

No response

@wezell
Copy link
Contributor

wezell commented May 29, 2024

The watcher sync should probably be Debounced - here is the Debouncer code, not sure if it runs in Quarkus:

https://github.com/dotCMS/core/blob/master/dotCMS/src/main/java/com/dotcms/concurrent/Debouncer.java

fabrizzio-dotCMS added a commit that referenced this issue Jun 4, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 5, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 6, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 7, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 7, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 7, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 10, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 10, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jun 10, 2024
@fabrizzio-dotCMS fabrizzio-dotCMS linked a pull request Jun 10, 2024 that will close this issue
github-merge-queue bot pushed a commit that referenced this issue Jun 11, 2024
### Proposed Changes
* We're introducing Watch options in the Push Mixin so they become
available to every push subcommand
* We're adding a bit of logic in the `DotExecutionStrategy` to be able
to handle starting up in watch mode and continue to repeat execution of
the command of choice
* I'm making `DotPush` a descendant of `DotCommand` to help me select
all Push commands and still gain access to the mixins.
* Because of the above change I needed to tell the global Push command
from the others, so I introduced a new getter with such info.
* Symmetry-wise wise I made `DotPull` also a descendant of `DotCommand`.
But just for symmetry. All Pull sub commands got modified when I removed
the no longer-needed DotCommand interface. But there are no other
changes in logic.
* A DirectoryWatchService was introduced. this class uses a poll
interval to observe and filter events.
* The events captured by the service are filtered so, let's say a folder
move that in reality creates between 4 and 5 events (FOLDER_MODIFY,
FOLDER_DELETE, FOLDER_MODIFY, FOLDER_CREATE), These events are filtered
so that we only get the most recent one. This ensures we do not flood
the server side with many unneeded changes.
* Changes in documentation were made
* Tests were introduced for the `DirectoryWatchService`, The Global Push
running with the watch mode flag on, and finally another one to cover
the changes introduced in the `DotExecutionStrategy`
@fabrizzio-dotCMS fabrizzio-dotCMS removed their assignment Jun 11, 2024
@nollymar nollymar reopened this Jun 11, 2024
@nollymar
Copy link
Contributor

Internal QA: Passed

Tested by @spbolton

@bryanboza bryanboza added the Release : 24.07.19 Bug Fixing label Jul 12, 2024
@fabrizzio-dotCMS fabrizzio-dotCMS self-assigned this Jul 17, 2024
@fabrizzio-dotCMS
Copy link
Contributor

@bryanboza it is not clear what you did here you ran
dotcli push content-types --watch and dotcli push /Users/bryanboza/Desktop/dotCLI/content-types --watch
and then what ? where did you drop the files ? under what folder ?

Can you please add that bit of info ?

@bryanboza
Copy link
Member

bryanboza commented Jul 17, 2024

@fabrizzio-dotCMS I just made a modification to one of the content-types files to ensure we are triggering the job after any change inside the content-types folder.

@bryanboza bryanboza removed the Release : 24.07.19 Bug Fixing label Jul 19, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jul 24, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jul 29, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jul 29, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jul 29, 2024
fabrizzio-dotCMS added a commit that referenced this issue Jul 29, 2024
@nollymar nollymar linked a pull request Jul 29, 2024 that will close this issue
github-merge-queue bot pushed a commit that referenced this issue Jul 29, 2024
### Proposed Changes

1. When the` files push ` command gets called from the global push,
passing an existing folder but outside the files folder an error is
thrown
2. This change makes the command default to the workspace root folder
and continues processing when the scenario described in the first point
occurs
@fabrizzio-dotCMS fabrizzio-dotCMS removed their assignment Jul 30, 2024
@spbolton
Copy link
Contributor

Internal QA. Tested global push and pull changing site with and without watch option as well as running from sites folder and specifying path. No error messages and changes were pushed as expected

Screenshot 2024-08-15 at 14 08 05

@bryanboza
Copy link
Member

Fixed and tested on the latest CLI version, and now it is working as expected. A new error regarding the exceptions has been opened here: #29629

We can close this one for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment