Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle binaries and sourcecode data section parts at PDS and SecHub side #1316

Merged
merged 19 commits into from
Jun 1, 2022
Merged

Handle binaries and sourcecode data section parts at PDS and SecHub side #1316

merged 19 commits into from
Jun 1, 2022

Conversation

de-jcup
Copy link
Member

@de-jcup de-jcup commented May 6, 2022 •
edited
Loading

This PR handles most parts from #729. It ...

@de-jcup
Start implementing support for binaries on PDS side #1167
5f9ff7a 
- Introduce apache commons compress library for tar archives # 1167
- implementing PDSTarSupport which archive inspection and filtering
  inclusive path transformation
@de-jcup de-jcup changed the title Start implementing support for binaries on PDS side #1167 Handle binaries and sourcecode data section parts at PDS and SecHub side May 9, 2022
@de-jcup
Moved gradle sub parts to gradle folder #1321 + introduce archive
4403785 
- all gradle sub files from rootfolder are now moved
  to `gradle` subfolder
- introduced `sechub-commons-archive` gradle submodule for handling
  issues #1320, #1319 and #1167
@de-jcup
Replaced PDSScanType with ScanType #1322,consolidate archive parts #1323
e207666 


- replaced PDSScanType with ScanType
- moved new and old archive parts for #1167 to
  sechub-commons-archive
@de-jcup
Combined Tar and ZipSupport into one single codebase #1323
32a60fc 
- to make things easier Tar and ZipSupport are now both inside
  ArchiveSupport
- also integrated PDSFileUnzipSupport into ArchiveSupport
- moved test resources for tar and zipfile to new gradle sub project
@de-jcup de-jcup force-pushed the feature-1167-handle-binaries-on-PDS-server-side branch from 17298cf to f34d4c0 Compare May 11, 2022 06:20
@de-jcup
Implemented transformation while extracting ZIP and Tars #1167
e7f99c1 
- implemented transformation while extracting
  same logic used for TAR and ZIP
- some refactoring/renaming
- simplified some parts
- added tests
- added special logging for sechub client execution on
  integration tests: elapsed time is now calculated and shown
@de-jcup
made integration tests faster #1340
d9169a3 
- introduced id inside mock adapter configuration class
- changed log behaviour for mock adapters
- changed IntegrationTestMockMode so the currently used
  variants will use the "zero wait" definitions instead
- reduced times to wait inside adapters
- changed behavior of role checking mechanism inside
  integration test framework
- some integration test methods did run very long but did
  execute the same action, only tests were different.
  So integrated them into one test, so much faster because
  precondition checks, scenario intialization and action to test
  are done only one time and then all tests happens.
  time
@de-jcup de-jcup force-pushed the feature-1167-handle-binaries-on-PDS-server-side branch from f34d4c0 to d9169a3 Compare May 13, 2022 15:30
@de-jcup
Using now always growing scenarios to speed up execution time #1340
b57c749 
- some minor changes
- created new gradle task `cleanIntegrationTestData`
  and integrated in to `integrationtest` task so will be executed
  on every server build automaticaly and ensures growing ids
  are always reset
- Growing scenarios abstract implementation does now create
  main prefix automatically
- added test case
- every `CleanScenario` has been changed to `GrowingScenario` and
  inherits now AbstractGrowingSecHubServerTestScenario
@de-jcup
Integration test enhancements #1340
0c1e4e8 
- Introducing marker interfaces for tests being necessary for
  separating documentation and "normal" integration tests.
- Implemented separation automation for test execution
@de-jcup
Created javadoc overview and details about profiles for testing #1167
ab078f4 
- for the integration tests for binary scans with PDS it was necessary
  to have a better overview of the current profiles and test scenarios
@de-jcup
Implemented integration test for binary PDS handling #1167
55b70f1
@de-jcup
Merge branch 'develop' into feature-1167-handle-binaries-on-PDS-serve…
3b05a35 
…r-side
@de-jcup
Implementation changes for #1167
c0f208d 
- refactored adapter config strategies,
  introduced two interfaces without generics, simplifed much
- Every PDS executor does now use the PDSAdapterConfigurationStrategy
  so we have one central configuration option for all common PDS parts
- The integration tests fails because green is returned, but no longer
  execution exceptions
@de-jcup
Implementation changes for #1167
ef49e51 
- to avoid implementing always same builder logic for every PDS
  adapter a refactoring was done:

  - PDSAdapterConfigData is contained
  by every PDSAdapterConfig variant. PDS relevant config data is stored
  there now.

  - PDSAdapterConfiguratoris also contained now in every pds adapter
    config variant and is used for configuration by common
    PDSAdapterConfigurationStrategy

  - The data and the configuration is implemented in one class:
    PDSAdapterDataConfigurator
 - documented PDS adapter configuration in architecture concepts
@de-jcup
Introduce environment variables + handle uploads different #1308
9e82d60 
- introduced environment variables PDS_JOB_HAS_SOURCES and
  PDS_JOB_HAS_BINARIES #1308
- added tar extraction #1286
- PDS now calculates the additional environment variables
- Multiple changes for binary handling
@de-jcup de-jcup force-pushed the feature-1167-handle-binaries-on-PDS-server-side branch from 916178f to fc23c0c Compare May 25, 2022 06:25
@de-jcup
Use binary upload mechanism introduced in sechub for pds uploads #1351
3390eca 
- renamed also the assertSechubReport parts to assertReportUnordered
  and removed deprecation. Reason: we still need this. because
  integration pds script tests which do a filewalk and return
  multiple results can have different orderings. And than we
  need still the old implementation. (we could change this later
  to the new object based approach as well)
- fixed flaky access level test
- enhanced test api
@de-jcup de-jcup force-pushed the feature-1167-handle-binaries-on-PDS-server-side branch from fc23c0c to 3390eca Compare May 25, 2022 07:25
@de-jcup de-jcup requested review from 4w31ss3 and winzj and removed request for 4w31ss3 May 25, 2022 09:36
@de-jcup de-jcup marked this pull request as ready for review May 25, 2022 09:44
@de-jcup de-jcup force-pushed the feature-1167-handle-binaries-on-PDS-server-side branch from d622c6a to 5386162 Compare May 25, 2022 10:00
@de-jcup
Added archive extraction documentation #1177
be41d9c 
- documentation
- minor changes
- added missing spdx headers
winzj
winzj requested changes May 31, 2022
View reviewed changes
Copy link
Member

@winzj winzj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very good to me. A lot of useful changes and improvements added with this pull request.

I found some smaller things that might need some changes. Please look at my comments for detailed information.

...ub-adapter-pds/src/main/java/com/mercedesbenz/sechub/adapter/pds/PDSAdapterConfigurator.java Show resolved Hide resolved
...ub-commons-archive/src/main/java/com/mercedesbenz/sechub/commons/archive/ArchiveSupport.java Show resolved Hide resolved
sechub-developertools/scripts/container/postgres/postgresql.conf Outdated Show resolved Hide resolved
sechub-doc/src/docs/asciidoc/diagrams/diagram_pds_adapter_configuration_overview.puml Show resolved Hide resolved
sechub-doc/src/docs/asciidoc/documents/shared/concepts/concept_archive_extraction.adoc Outdated Show resolved Hide resolved
...integrationtest/src/main/java/com/mercedesbenz/sechub/integrationtest/internal/MockData.java Outdated Show resolved Hide resolved
sechub-pds/src/main/java/com/mercedesbenz/sechub/pds/UploadSizeConfiguration.java Outdated Show resolved Hide resolved
sechub-scan/src/main/java/com/mercedesbenz/sechub/domain/scan/product/ProductExecutorData.java Outdated Show resolved Hide resolved
sechub-sereco/src/test/resources/nessus/nessus_7.0.2.result.xml Outdated Show resolved Hide resolved
sechub-testframework/src/main/java/com/mercedesbenz/sechub/test/TestURLBuilder.java Outdated Show resolved Hide resolved
@de-jcup de-jcup force-pushed the feature-1167-handle-binaries-on-PDS-server-side branch from 5253ff8 to c6f99aa Compare June 1, 2022 14:14
@de-jcup
Review changes for PR #1316
ce0ac1f 
- removed unnecessary /irritating javadoc
- marked method as deprecated
- reverted unnecessary replacement of "archive" to "tar"
- updated apply-header.sh so plantuml and puml files do now have
  spdx headers automatically as well
- fixed typos
- removed duplicated diagram
- changed upload size variables for PDS
- improved javadoc
- renamed variables
@de-jcup de-jcup force-pushed the feature-1167-handle-binaries-on-PDS-server-side branch from c6f99aa to ce0ac1f Compare June 1, 2022 14:15
@de-jcup de-jcup merged commit e257143 into develop Jun 1, 2022
@de-jcup de-jcup deleted the feature-1167-handle-binaries-on-PDS-server-side branch June 1, 2022 14:51
@de-jcup de-jcup mentioned this pull request Jun 2, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@winzj winzj winzj requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@de-jcup @winzj