Fixed an edge case for incremental run failures on cloud file-systems

[bashir2]

Description of what I changed

The bug was discovered as part of this report (see item 4). The actual bug is not what is reported there but for whatever [other] reasons, an incremental-run has failed. That failure has blocked the following incremental-runs (which are supposed to repeat the old failed one). The reason this edge-case fails on cloud files-systems (e.g., GCS, S3), is because there is not really a concept of "directory" there. We have buckets and files in them. As a consequence of this bug-fix, the incremental_run directories now have a timestamp suffix as well.

E2E test

TESTED:

Beside updating unit-tests, I rant the controller locally with dwhRootPrefix set to a s3://... path; then created an artificial failure by creating an error.log in an incremental run (and removing some other files). I reran the incremental-build and verified that the controller can recover from that situation (previously it could not).

Checklist: I completed these to help reviewers :)

• I have read and will follow the review process.

• I am familiar with Google Style Guides for the language I have coded in.

  No? Please take some time and review Java and Python style guides.

• My IDE is configured to follow the Google code styles.

  No? Unsure? -> configure your IDE.

• I have added tests to cover my changes. (If you refactored existing code that was well tested you do not have to add tests)

• I ran mvn clean package right before creating this pull request and added all formatting changes to my commit.

• All new and existing tests passed.

• My pull request is based on the latest changes of the master branch.

  No? Unsure? -> execute command git pull --rebase upstream master

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 65.85366% with 14 lines in your changes missing coverage. Please review.

Project coverage is 51.42%. Comparing base (72d1113) to head (a30663c).

Files with missing lines	Patch %	Lines
...ava/com/google/fhir/analytics/PipelineManager.java	12.50%	7 Missing ⚠️
.../main/java/com/google/fhir/analytics/DwhFiles.java	79.31%	3 Missing and 3 partials ⚠️
...ava/com/google/fhir/analytics/DwhFilesManager.java	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #1185      +/-   ##
============================================
+ Coverage     51.34%   51.42%   +0.07%     
- Complexity      711      712       +1     
============================================
  Files            95       95              
  Lines          5889     5886       -3     
  Branches        782      781       -1     
============================================
+ Hits           3024     3027       +3     
+ Misses         2578     2573       -5     
+ Partials        287      286       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[chandrashekar-s]

Thanks @bashir2 for the changes, they look good to me. There is only one minor suggestion, you can merge the changes after resolving it.

[bashir2]

Thanks @chandrashekar-s for the review.