Enhancement/Leverage httplib2shim for GCP Client Libraries

[x4v13r64]

Description

Test fix for #665 & #443, leveraging https://github.com/GoogleCloudPlatform/httplib2shim.

Ran with no errors on my end, with the additional improvements:

• Completed in 36 seconds vs 4.26 minutes
• Used 85MB or memory vs 190MB

Type of change

Select the relevant option(s):

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works (optional)
• New and existing unit tests pass locally with my changes

[x4v13r64]

@rtomlinson-latacora, @kareem-DA, @nycnewman please test.

[kareem-DA]

I ran `test/httplib2shim` and I got the following error: ``` 2020-03-17 14:35:59 blackduck-test scout[3822] ERROR base.py L110: Unable to list accessible Projects: 'Resource' object has no attribute 'list' Traceback (most recent call last): File "/home/kareem_khan_digitalasset_com/ScoutSuite/ScoutSuite/providers/gcp/facade/base.py", line 110, in _get_projects_recursively folder_request = resourcemanager_client_v2.folders().list(parent='%ss/%s' % (parent_type, parent_id)) AttributeError: 'Resource' object has no attribute 'list' ```

[rtomlinson-latacora]

I ran with --all-projects and it only took ~5 minutes with expected errors. Huge improvement. Nice find @j4v .
What command did you use @kareem-DA? That looks like it might just be a bug in the _get_projects_recursively logic for folders.

[x4v13r64]

I ran with --all-projects and it only took ~5 minutes with expected errors. Huge improvement. Nice find @j4v .

If you look at the cprofile a huge % of the time was spent just creating the clients so I'm not surprised. What was the previous run time like?

[x4v13r64]

I ran test/httplib2shim and I got the following error:

That's odd - the same command runs fine in develop or master? Can you provide full input/output please... The changes in this PR shouldn't affect the functionality you describe.

See below.

[kareem-DA]

python scout.py gcp --service-account /opt/scoutcreds/gcp.json --organization-id *** Removed *** --debug --services computeengine 2>&1 |tee ../scout.log

[x4v13r64]

Ok bug confirmed, fixing.

@kareem-DA should now be fixed, please retest?

[rtomlinson-latacora]

ScoutSuite was running for 2-3 hours with my semaphore fix for a large organization.

[x4v13r64]

ScoutSuite was running for 2-3 hours with my semaphore fix for a large organization.

2-3h to 5 minutes???

[rtomlinson-latacora]

ScoutSuite was running for 2-3 hours with my semaphore fix for a large organization.

2-3h to 5 minutes???

Haha, yeah. I was being pretty conservative with the semaphores so I probably could have cut that time down.

[x4v13r64]

ScoutSuite was running for 2-3 hours with my semaphore fix for a large organization.

2-3h to 5 minutes???

Haha, yeah. I was being pretty conservative with the semaphores so I probably could have cut that time down.

Well great, that's a tremendous improvement. Could you confirm that the report generated is the same? A quick diff against the results JSON should only show 2-3 differences.

[rtomlinson-latacora]

ScoutSuite was running for 2-3 hours with my semaphore fix for a large organization.

2-3h to 5 minutes???

Haha, yeah. I was being pretty conservative with the semaphores so I probably could have cut that time down.

Well great, that's a tremendous improvement. Could you confirm that the report generated is the same? A quick diff against the results JSON should only show 2-3 differences.

Yup. Running it the old way with the semaphore now since I already overwrote the existing report. It'll be a little while but I'll get you a response.

[kareem-DA]

This is much better. Ran the entire org in 3 minutes. All services and all projects. The results look accurate as well. Thanks Kareem

…

On Mar 17, 2020, at 11:25 AM, rtomlinson-latacora ***@***.***> wrote: ScoutSuite was running for 2-3 hours with my semaphore fix for a large organization. 2-3h to 5 minutes??? Haha, yeah. I was being pretty conservative with the semaphores so I probably could have cut that time down. Well great, that's a tremendous improvement. Could you confirm that the report generated is the same? A quick diff against the results JSON should only show 2-3 differences. Yup. Running it the old way with the semaphore now since I already overwrote the existing report. It'll be a little while but I'll get you a response. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#676 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOFRDMXGJIHLDAE4D4BFSQ3RH6I5JANCNFSM4LNHLNKQ>.

-- This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at  http://www.digitalasset.com/emaildisclaimer.html <http://www.digitalasset.com/emaildisclaimer.html>. If you are not the intended recipient, please delete this message.

[x4v13r64]

This is much better. Ran the entire org in 3 minutes. All services and all projects. The results look accurate as well.

Great, I'll run some tests on my end and integrate into develop. Thanks both!

[rtomlinson-latacora]

Findings are comparable on my end as well.

[x4v13r64]

Thanks @rtomlinson-latacora & @kareem-DA, this has now been merged into develop. I'll release v5.8 ASAP.