[Bug][gitextractor] Out of memory, malloc failed (tried to allocate 1462922845 bytes)

[Mandla-SM]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

"Collect data in full refresh", for bitbucket server in beta4 and beta5 throw the following error:

attached stack trace -- stack trace: | github.com/apache/incubator-devlake/server/services.runPipeline | /app/server/services/pipeline_runner.go:79 | [...repeated from below...] Wraps: (2) Error running pipeline 39. Wraps: (3) attached stack trace -- stack trace: | github.com/apache/incubator-devlake/server/services.RunTasksStandalone | /app/server/services/task.go:217 | github.com/apache/incubator-devlake/server/services.(pipelineRunner).runPipelineStandalone.func1 | /app/server/services/pipeline_runner.go:42 | github.com/apache/incubator-devlake/core/runner.runPipelineTasks | /app/core/runner/run_pipeline.go:90 | github.com/apache/incubator-devlake/core/runner.RunPipeline | /app/core/runner/run_pipeline.go:54 | github.com/apache/incubator-devlake/server/services.(pipelineRunner).runPipelineStandalone | /app/server/services/pipeline_runner.go:38 | github.com/apache/incubator-devlake/server/services.runPipeline | /app/server/services/pipeline_runner.go:76 | github.com/apache/incubator-devlake/server/services.RunPipelineInQueue.func1 | /app/server/services/pipeline.go:349 | runtime.goexit | /usr/local/go/src/runtime/asm_amd64.s:1598 Wraps: (4) attached stack trace | -- stack trace: | | github.com/apache/incubator-devlake/server/services.RunTasksStandalone.func1 | | /app/server/services/task.go:189 | Wraps: (2) Error running task 4877. | Wraps: (3) attached stack trace | -- stack trace: | | github.com/apache/incubator-devlake/core/runner.RunPluginSubTasks | | /app/core/runner/run_task.go:333 | | [...repeated from below...] | Wraps: (4) subtask Clone Git Repo ended unexpectedly | Wraps: (5) attached stack trace | -- stack trace: | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(GitcliCloner).execCommand | | /app/plugins/gitextractor/parser/clone_gitcli.go:323 | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(GitcliCloner).git | | /app/plugins/gitextractor/parser/clone_gitcli.go:311 | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(GitcliCloner).gitClone | | /app/plugins/gitextractor/parser/clone_gitcli.go:273 | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(GitcliCloner).shallowClone | | /app/plugins/gitextractor/parser/clone_gitcli.go:224 | | github.com/apache/incubator-devlake/plugins/gitextractor/parser.(GitcliCloner).CloneRepo | | /app/plugins/gitextractor/parser/clone_gitcli.go:182 | | github.com/apache/incubator-devlake/plugins/gitextractor/tasks.CloneGitRepo | | /app/plugins/gitextractor/tasks/repo_cloner.go:60 | | github.com/apache/incubator-devlake/core/runner.runSubtask | | /app/core/runner/run_task.go:425 | | github.com/apache/incubator-devlake/core/runner.RunPluginSubTasks | | /app/core/runner/run_task.go:330 | | github.com/apache/incubator-devlake/core/runner.RunPluginTask | | /app/core/runner/run_task.go:165 | | github.com/apache/incubator-devlake/core/runner.RunTask | | /app/core/runner/run_task.go:139 | | github.com/apache/incubator-devlake/server/services.runTaskStandalone | | /app/server/services/task_runner.go:114 | | github.com/apache/incubator-devlake/server/services.RunTasksStandalone.func1 | | /app/server/services/task.go:187 | | runtime.goexit | | /usr/local/go/src/runtime/asm_amd64.s:1598 | Wraps: (6) git cmd [git clone https://:**@./bitbucket/scm/oma/<project_name>.git /tmp/gitextractor4184564856 --depth=1 --bare] in failed: Cloning into bare repository '/tmp/gitextractor4184564856'... | | fatal: Out of memory, malloc failed (tried to allocate 1462922845 bytes) | | fatal: unpack-objects failed | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

What do you expect to happen

Refreshes to run through without failing.

How to reproduce

Re-transform Data or Collect Data or Collect data in full refresh a project with bitbucket repos with different amounts of commits and branches.
The project also has Jira and Bamboo.

Anything else

The bitbucket project in this ticket had only one initial commit. In another case, the bibucket project had > 100 commits and many branches. The shortened errors are:
| fatal: Out of memory, malloc failed (tried to allocate 361795585 bytes) | fatal: index-pack failed Error types: (1)
| fatal: Out of memory, malloc failed (tried to allocate 1462922845 bytes) | fatal: unpack-objects failed Error types: (1)
| fatal: Out of memory, malloc failed (tried to allocate 481953334 bytes) | fatal: index-pack failed Error types: (1)

Version

v1.0.2-beta4 and v1.0.2-beta5

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

The out-of-memory error you're encountering during the cloning process with the gitextractor plugin could be related to how the plugin handles large repositories. Here are a few suggestions to help mitigate this issue:

1. Shallow Cloning: Ensure that shallow cloning is enabled, as it reduces memory usage by only fetching the latest commits. The shallowClone method is designed to perform a clone with a depth of 1, which minimizes the data fetched ^[1].

2. Configuration Options: Adjust the configuration options to optimize memory usage:

   • Set SkipCommitStat to true to skip commit statistics.
   • Set SkipCommitFiles to true to skip processing commit files.
   • Ensure NoShallowClone is set to false to allow shallow cloning.
   • Consider setting UseGoGit to false if applicable ^[2].

3. Batch Processing: The plugin uses a batching mechanism to handle data operations, which can help manage memory usage. Ensure that the BatchSaveDivider is configured correctly to process data in smaller chunks ^[3].

4. Double Cloning: If shallow cloning is not sufficient, the doubleClone method can be used to perform a full clone into an intermediary directory followed by a shallow clone, which can help reduce memory usage ^[1].

By adjusting these settings and methods, you may be able to reduce the memory footprint during the cloning process and avoid the out-of-memory errors.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[Mandla-SM]

Please provide a guide on how to implement these suggestions in DevLake.

[dosubot]

To implement the suggestions for mitigating the out-of-memory error in DevLake's gitextractor plugin, follow these steps:

1. Enable Shallow Cloning:

   • Ensure that the NoShallowClone option is set to false. This allows the use of the shallowClone method, which fetches only the latest commits.
   • Check the CloneRepoConfig struct in the code to verify this setting ^[1].

2. Adjust Configuration Options:

   • Modify the configuration file or API request to set SkipCommitStat and SkipCommitFiles to true. This reduces the amount of data processed.
   • Verify that UseGoGit is set to false by checking the configuration settings. If not explicitly set, it defaults to false ^[2].

3. Configure Batch Processing:

   • Adjust the BatchSaveDivider batch size in the NewDatabase function. The default batch size is 100, but you can modify it to a smaller value if needed to manage memory usage ^[3].

4. Consider Double Cloning:

   • If shallow cloning is insufficient, you might need to implement the doubleClone method. This involves performing a full clone into an intermediary directory followed by a shallow clone. Ensure that the NoShallowClone option is set appropriately to enable this method ^[1].

By following these steps, you can optimize the gitextractor plugin's performance and reduce memory usage during the cloning process.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[Mandla-SM]

The issue has gone away. It seems like it may have been a bitbucket repo name clash.
In bitbucket_projectA there are 5 repos with the exact same name as in bitbucket_projectB. Both bitbucket projects were linked to the same DevLake project.
I double checked that only the repos that are active are linked(bitbucket_projectA/<repo_name>) and the ones that are dormant were removed from Connections.
I suspect the issue may still occur for people who have this situation but their bitbucket repos are active.

Re-transform data ran succesfully after this.

Collect data in full refresh has 503 errors but I don't think they are related.