Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a Job with the relevant logic from ImportUtil and ImportContentletsAction to successfully import large files #29498

Open
Tracked by #29482
fabrizzio-dotCMS opened this issue Aug 7, 2024 · 0 comments

Comments

@fabrizzio-dotCMS
Copy link
Contributor

fabrizzio-dotCMS commented Aug 7, 2024

Parent Issue

#29482

Task

Create a Job following the implementation of the Epic
This Job has to use the new BufferedCvsReader to extract the content data from a file to avoid memory issues
It is desirable to be able to start or restart the import from a given row number skipping all previous
The new Job can be instructed to perform a db commit after n rows are saved.
The new method should return an Immutable ImportSumary class instead of a HashMap reporting the results.
Consume a single class with all the required parameters instead of taking a large number of arguments. Right now it takes 15 parameters. When the maximum number of allowed params should be 7. Any private methods created here have to meet these requirements too.
Optionally we can Refactor the method ImportUtil.importFile to reduce its complexity and make it clearer to read and understand. Currently, it is 115 lines and the recommended is 15

Proposed Objective

Core Features

Proposed Priority

Priority 2 - Important

Acceptance Criteria

  1. The Job should be able to import successfully a large number of content items. e.g. 10K items.
  2. The Job should never get stuck or run out of memory.
  3. The same options passed to the Structs Action (preview, import, fields, and key selections should be accepted by this job) Since the functionality should remain as it is now.

External Links... Slack Conversations, Support Tickets, Figma Designs, etc.

No response

Assumptions & Initiation Needs

No response

Quality Assurance Notes & Workarounds

No response

Sub-Tasks & Estimates

No response

@fabrizzio-dotCMS fabrizzio-dotCMS changed the title Create a Job with the relevant logic from ImportUtil and ImportContentletsAction to successfully imports large files Create a Job with the relevant logic from ImportUtil and ImportContentletsAction to successfully import large files Aug 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: In Progress
Development

No branches or pull requests

2 participants