migrate os.listdir() to os.scandir() to increase performance

[xsuchy]

As recommended in https://peps.python.org/pep-0471/

[xsuchy]

I did very simple and trivial benchmark on walking through the kernel tree:

• previously: 45 second
• after this PR: 36 second

[xsuchy]

If this gets merged, I will open another PR for the remaining occurrences of listdir() in other functions.

[xsuchy]

Any review comments?

[xsuchy]

@AyanSinhaMahapatra or @pombredanne can I kindly ask you for a review?

[AyanSinhaMahapatra]

@xsuchy Thanks++ for the PR.

Thanks for the references, we should definitely move towards using scandir() allright in this context. I've tested out this branch with scancode-toolkit latest too, and everything looks good, and we do get some performance improvements there on the codebase collection step similarly as indicated by you, thanks for doing and including those too.

1. Some more CI updates and other fixes were also added to main, could you rebase to latest main?
2. Please go ahead and use scandir() in the other instances of listdir() use in commoncode too.

This looks ready otherwise, and apologies for the late review.

We might want to do the same in other places too which would have some effect on the performance, https://github.com/search?q=org%3Aaboutcode-org+os.listdir&type=code maybe this could be useful in extractcode/fetchcode/deltacode too