Skip to content

[backend/frontend] move background tasks execution to worker (#10274) #10699

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
JeremyCloarec merged 11 commits into from
Jun 4, 2025
Merged

[backend/frontend] move background tasks execution to worker (#10274) #10699

JeremyCloarec merged 11 commits into from
Jun 4, 2025

Conversation

@JeremyCloarec
Copy link
Contributor

@JeremyCloarec JeremyCloarec commented Apr 16, 2025 •
edited
Loading

Proposed changes

  • background tasks are no longer processed in task manager, but are sent to worker for processing
  • task manager now only serves as a provisioning manager: it's job is now to build bundles with the approriate operation to send to the worker for ingestion
  • Tasks and Rules are updated to now display the progress of the work in addition to the display of the task
  • Containers are no longer shared synchronously: we now diplay a loading spinner to indicate to the user that a sharing task is ongoing

client PR: OpenCTI-Platform/client-python#883

Related issues

Checklist

  • I consider the submitted work as finished
  • I tested the code for its functionality
  • I wrote test cases for the relevant uses case (coverage and e2e)
  • I added/update the relevant documentation (either on github or on notion)
  • Where necessary I refactored code to improve the overall quality

Further comments

@github-actions github-actions bot added the filigran team use to identify PR from the Filigran team label Apr 16, 2025
@JeremyCloarec JeremyCloarec added the multi-repository For contribution that requires PR in several repository label Apr 23, 2025
@JeremyCloarec JeremyCloarec force-pushed the issue/10274 branch 5 times, most recently from c9e8e23 to 061a23e Compare May 14, 2025 13:50
@codecov
Copy link

codecov bot commented May 14, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 21.72285% with 627 lines in your changes missing coverage. Please review.

Project coverage is 64.83%. Comparing base (b37d230) to head (8862c67).
Report is 1 commits behind head on release/current.

Files with missing lines Patch % Lines
...latform/opencti-graphql/src/manager/taskManager.js 5.64% 418 Missing ⚠️
...tform/opencti-graphql/src/domain/stixCoreObject.js 13.63% 76 Missing ⚠️
...latform/opencti-graphql/src/manager/ruleManager.ts 7.93% 58 Missing ⚠️
...pencti-platform/opencti-graphql/src/domain/stix.js 25.00% 24 Missing ⚠️
...rm/opencti-graphql/src/resolvers/stixCoreObject.js 8.33% 11 Missing ⚠️
...latform/opencti-graphql/src/database/repository.js 0.00% 10 Missing ⚠️
...-platform/opencti-graphql/src/database/rabbitmq.js 63.15% 7 Missing ⚠️
...ncti-platform/opencti-graphql/src/database/smtp.js 36.36% 7 Missing ⚠️
...tform/opencti-graphql/src/domain/backgroundTask.js 40.00% 3 Missing ⚠️
...src/utils/filtering/filtering-stix/stix-testers.ts 40.00% 3 Missing ⚠️
... and 6 more
Additional details and impacted files 
@@                 Coverage Diff                 @@
##           release/current   #10699      +/-   ##
===================================================
- Coverage            65.31%   64.83%   -0.48%     
===================================================
  Files                  716      716              
  Lines                70156    70316     +160     
  Branches              7683     7611      -72     
===================================================
- Hits                 45822    45589     -233     
- Misses               24334    24727     +393

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JeremyCloarec JeremyCloarec mentioned this pull request May 15, 2025
Merged
5 tasks
@JeremyCloarec JeremyCloarec force-pushed the issue/10274 branch 7 times, most recently from d4fcbcb to 8c0ef89 Compare May 27, 2025 15:44
@JeremyCloarec JeremyCloarec marked this pull request as ready for review May 27, 2025 16:07
@JeremyCloarec JeremyCloarec changed the title [backend/frontend] move background tasks execution to worker [backend/frontend] move background tasks execution to worker (#10274) May 28, 2025
aHenryJard
aHenryJard reviewed May 28, 2025
View reviewed changes
aHenryJard
aHenryJard reviewed May 28, 2025
View reviewed changes
aHenryJard
aHenryJard reviewed May 28, 2025
View reviewed changes
opencti-platform/opencti-front/src/private/components/data/tasks/TasksList.jsx
const progressFullText = `${t_i18n('Progress')}${progressNumberDisplay}${provisioningNumberDisplay}`;
let progressValue = 0;
if (task.work) {
if (task.work.status === 'complete') {
Copy link
Member

@aHenryJard aHenryJard May 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to use enum or constants to avoid having a string for status checks ?

aHenryJard
aHenryJard reviewed May 28, 2025
View reviewed changes
aHenryJard
aHenryJard reviewed May 28, 2025
View reviewed changes
opencti-platform/opencti-graphql/src/database/smtp.js

const USE_SSL = booleanConf('smtp:use_ssl', false);
const REJECT_UNAUTHORIZED = booleanConf('smtp:reject_unauthorized', false);
const SMTP_ENABLE = booleanConf('smtp:enabled', true);
Copy link
Member

@aHenryJard aHenryJard May 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change seems unrelated to background tasks.

Copy link
Contributor Author

@JeremyCloarec JeremyCloarec May 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a leftover from Julien's PR. I agree that it's unrelated, but I didn't remove it thinking that it maybe could be useful in some cases

Copy link
Member

@aHenryJard aHenryJard May 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually I think it's usefull for test run on CI at least.

aHenryJard
aHenryJard reviewed May 28, 2025
View reviewed changes
opencti-platform/opencti-graphql/src/database/stix-2-1-converter.ts
if (isStixSightingRelationship(type)) {
return 'sighting';
}
if (isBasicRelationship(type)) {
Copy link
Member

@aHenryJard aHenryJard May 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this change, is this fixing a bug ?

Copy link
Contributor Author

@JeremyCloarec JeremyCloarec May 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A leftover from Julien's PR also. I assumed that it was a fix/improvement in the stix model (that way all relationships will be typed as relationship instead of ref relationships being typed as their entity_type)

aHenryJard
aHenryJard reviewed May 28, 2025
View reviewed changes
Copy link
Member

@aHenryJard aHenryJard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like there are several bug fix included in this PR, would be better to have them in dedicated PRs IMO.

Since the taskManager it rewritten, have you tried to move it to typescript ?

@JeremyCloarec
Copy link
Contributor Author

JeremyCloarec commented May 28, 2025

I feel like there are several bug fix included in this PR, would be better to have them in dedicated PRs IMO.

Since the taskManager it rewritten, have you tried to move it to typescript ?

Haven't worked on a TS refacto, but I could look into it to see if it is a lot of work or not

@JeremyCloarec JeremyCloarec force-pushed the issue/10274 branch 4 times, most recently from bef0e58 to a4e1ce7 Compare June 2, 2025 09:40
aHenryJard
aHenryJard reviewed Jun 3, 2025
View reviewed changes
aHenryJard
aHenryJard approved these changes Jun 4, 2025
View reviewed changes
Copy link
Member

@aHenryJard aHenryJard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to go for me, let's merge that ! Really great job.

@JeremyCloarec JeremyCloarec merged commit 39126e6 into release/current Jun 4, 2025
6 checks passed
@JeremyCloarec JeremyCloarec deleted the issue/10274 branch June 4, 2025 08:02
@marieflorescontact marieflorescontact mentioned this pull request Jun 11, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aHenryJard aHenryJard aHenryJard approved these changes

Assignees

No one assigned

Labels

filigran team use to identify PR from the Filigran team multi-repository For contribution that requires PR in several repository

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@JeremyCloarec @aHenryJard @richard-julien