[SYSINFRA-3912] Handle EBS resize edge case (grailbio/grail!20013)

Approved-by: Jim Clune <jclune@grailbio.com>

GitLab URL: https://gitlab.com/grailbio/grail/-/merge_requests/20013

fbshipit-source-id: 3f36a25

Revert "Merge: Download each file once (grailbio/grail!15922)" (grail…

…bio/grail!16436)

Approved-by: Pedro Fialho <pfialho@grailbio.com>

GitLab URL: https://gitlab.com/grailbio/grail/-/merge_requests/16436

fbshipit-source-id: 76a8d8c

Resolve SYSINFRA-2039 "/run reflow without taskdb" (grailbio/grail!7409)

Approved-by: Pedro Fialho <pfialho@grailbio.com>

GitLab URL: https://gitlab.com/grailbio/grail/-/merge_requests/7409

fbshipit-source-id: 625523f

Revert "Merge: [SYSINFRA-554] Deprecate nocacheextern flag (grailbio/…

…grail!3658)" (grailbio/grail!5650)

Approved-by: Jim Clune <jclune@grailbio.com>

GitLab URL: https://gitlab.com/grailbio/grail/-/merge_requests/5650

fbshipit-source-id: cf457c2

[SYSINFRA-1870]Reflow exec failures include error hints in error mess…

…age (grailbio/grail!3811)

Approved-by: Jim Clune <jclune@grailbio.com>

GitLab URL: https://gitlab.com/grailbio/grail/-/merge_requests/3811

fbshipit-source-id: c226f65

[SYSINFRA-1673] reduce reflow cpu usage in scheduler.run (grailbio/gr…

…ail!902)

reduce reflow cpu/mem usage

Various changes to reduce reflow cpu/mem usage:
- Use a higher (`50`) than default (`2`) number of idle conns per reflowlet.
- Make sure to close `http` calls to release connections
- Don't update status in `repository/manager` for non-console
  use-cases (eg: `pi-evaluator`), and don't set a default status.

Co-authored-by:
Approved-by: Approved-by: Jim Clune <jclune@grailbio.com>

GitLab URL: https://gitlab.com/grailbio/grail/-/merge_requests/902

fbshipit-source-id: 4af52c5

[SYSINFRA-1501] base/digest: Use buffered writer for computing digests

Summary:
Using a buffered writer for computing digests reduces the amount
of memory used and allocations made while computing digests.

Since digest computation is used widely, we introduce `NewWriterShort`
for computing digests of relatively short inputs and use it where appropriate.

Performance improvement is mainly due to addition of `WriteString`.
Various benchmarks show modest CPU improvements but significant memory/alloc reduction (reduced by **90%** in some cases)

Test Plan:
Unit and Integration tests.

Benchmarking some digest computations (fileset assertions in particular shown below):
- before
```
BenchmarkDigest
BenchmarkDigest/writeDigestOld_N100
BenchmarkDigest/writeDigestOld_N100-8         	    2750	    401574 ns/op	  177809 B/op	    1232 allocs/op
BenchmarkDigest/writeDigestNew_N100
BenchmarkDigest/writeDigestNew_N100-8         	    3904	    269149 ns/op	   51968 B/op	    1310 allocs/op
BenchmarkDigest/writeDigestOld_N1000
BenchmarkDigest/writeDigestOld_N1000-8        	     183	   6039835 ns/op	 1770788 B/op	   12113 allocs/op
BenchmarkDigest/writeDigestNew_N1000
BenchmarkDigest/writeDigestNew_N1000-8        	     352	   3031291 ns/op	  515594 B/op	   13010 allocs/op
BenchmarkDigest/writeDigestOld_N10000
BenchmarkDigest/writeDigestOld_N10000-8       	      16	  70172874 ns/op	22929233 B/op	  121179 allocs/op
BenchmarkDigest/writeDigestNew_N10000
BenchmarkDigest/writeDigestNew_N10000-8       	      31	  37214169 ns/op	 5153214 B/op	  130010 allocs/op
BenchmarkDigest/writeDigestOld_N100000
BenchmarkDigest/writeDigestOld_N100000-8      	       2	 960212048 ns/op	207093356 B/op	 1209659 allocs/op
BenchmarkDigest/writeDigestNew_N100000
BenchmarkDigest/writeDigestNew_N100000-8      	       2	 561615654 ns/op	51450236 B/op	 1300010 allocs/op
PASS
```
- after
```
BenchmarkDigest
BenchmarkDigest/writeDigestOld_N100
BenchmarkDigest/writeDigestOld_N100-8         	    3322	    339663 ns/op	  136214 B/op	      34 allocs/op
BenchmarkDigest/writeDigestNew_N100
BenchmarkDigest/writeDigestNew_N100-8         	    5930	    191718 ns/op	   10168 B/op	     112 allocs/op
BenchmarkDigest/writeDigestOld_N1000
BenchmarkDigest/writeDigestOld_N1000-8        	     294	   4650456 ns/op	 1316342 B/op	     115 allocs/op
BenchmarkDigest/writeDigestNew_N1000
BenchmarkDigest/writeDigestNew_N1000-8        	     483	   2245917 ns/op	   61336 B/op	    1012 allocs/op
BenchmarkDigest/writeDigestOld_N10000
BenchmarkDigest/writeDigestOld_N10000-8       	      21	  63469161 ns/op	18344572 B/op	    1176 allocs/op
BenchmarkDigest/writeDigestNew_N10000
BenchmarkDigest/writeDigestNew_N10000-8       	      43	  27897608 ns/op	  572248 B/op	   10012 allocs/op
BenchmarkDigest/writeDigestOld_N100000
BenchmarkDigest/writeDigestOld_N100000-8      	       2	 753470408 ns/op	161193760 B/op	    9590 allocs/op
BenchmarkDigest/writeDigestNew_N100000
BenchmarkDigest/writeDigestNew_N100000-8      	       3	 359851338 ns/op	 5607640 B/op	  100012 allocs/op
PASS
```

Reviewers: pboyapalli, ghorrell

Reviewed By: pboyapalli

Subscribers: dnicolaou

Differential Revision: https://phabricator.grailbio.com/D73250

fbshipit-source-id: 39208f6

[SYSINFRA-1419] reflow/flow: Avoid creating copy of Fileset

Summary: Use a pointer instead of value to avoid creating a copy of the resolved fileset.

Test Plan: Unit and Integration tests.

Reviewers: pboyapalli

Reviewed By: pboyapalli

Subscribers: dnicolaou

Differential Revision: https://phabricator.grailbio.com/D72110

fbshipit-source-id: 43f83e8

[SYSINFRA-595] reflow: Refactoring and cleanup

Summary:
Most of the changes here are done to facilitate the eventual move of `tool.Runner`
into the `reflow/runtime` package.  This change cleans up the dependency chain
so that the move itself is easier to manage.

There is no functionality change in this revision.

Test Plan: Unit and Integration tests.

Reviewers: ghorrell

Reviewed By: ghorrell

Subscribers: dnicolaou, pboyapalli

Differential Revision: https://phabricator.grailbio.com/D70851

fbshipit-source-id: 61dc1a9

[SYSINFRA-1028] reflow/flow: Reflect resolved filesets of Exec depend…

…encies in log output.

Summary:
Since the `scheduler` resolves filesets of the dependencies of an `Exec` flow
only during the `Load` stage, the resolved filesets are never reflected in the flow graph
(but only available in the scheduler task's config).

This change makese these resolved filesets available in the `Flow` for subsequent logging.

Test Plan:
Unit and integration tests.

See [[ https://jira.ti-apps.aws.grail.com/browse/SYSINFRA-1029 | SYSINFRA-1029 ]].

Reviewers: pboyapalli

Reviewed By: pboyapalli

Subscribers: dnicolaou

Differential Revision: https://phabricator.grailbio.com/D69257

fbshipit-source-id: 8cf72c6