pga: safe downloads #69

smola · 2018-06-15T09:57:49Z

pga: download files to temporary file first, fixes Make PGA downloader safe to cancel #39
make get/list cancellable: Control+C will make pga to shutdown gracefully and clean up after itself, leaving no traces of temporary downloads

smola · 2018-06-15T09:58:23Z

Note that I have not tested this with HDFS yet.
@bzz If you happen to use pga for a new download, could you test from this branch?

bzz · 2018-06-15T10:09:54Z

Sounds awesome, going to start full download using this branch's pga get to the same cluster and then let's compare results after a single run and report the results here on Mon 18.06.2018

bzz · 2018-06-18T12:15:57Z

High-level overview

hdfs dfs -du -h hdfs://hdfs-namenode/pga2
2.5 T  hdfs://hdfs-namenode/pga2/siva

root@hadoop-alex:/# hdfs dfs -du -h hdfs://hdfs-namenode/pga
2.4 T  hdfs://hdfs-namenode/pga/siva

😕 and seems like a pod has been re-started so there are no logs ....

@smola number of files seems to be different, when using this branch

$ hdfs dfs -ls -R hdfs://hdfs-namenode/pga2 | grep -c "\.siva$"
239733

$ hdfs dfs -ls -R hdfs://hdfs-namenode/pga/ | grep -c "\.siva$"
239807

but number of .siva file in index is

zgrep -o "[0-9a-z]*\.siva" ~/.pga/latest.csv.gz | sort | uniq | wc -l
239807

bzz · 2018-06-18T13:28:10Z

So, the second run using this branch resulted in:

overall bigger results size in FS
74 .siva files missing and were not downloaded

campoy

I'm concerned with the way you're handling the errors from the deferred functions.

Seems a bit too "smart" for me.

campoy · 2018-06-18T18:57:10Z

PublicGitArchive/pga/cmd/cache.go

@@ -97,3 +143,9 @@ func getIndex() (io.ReadCloser, error) {
 	}
 	return gzip.NewReader(f)
 }
+
+func checkClose(name string, c io.Closer, err *error) {


I'm not sure about this piece of code, tbh.
This is changing the value of the error in an almost magical way.

I understand why you're doing this, but I'd rather check have slightly more verbose code than this subtle interaction.

campoy · 2018-06-18T18:57:33Z

PublicGitArchive/pga/cmd/cache.go

+}
+
+func cancelableCopy(ctx context.Context, dst io.Writer, src io.Reader) (
+	int64, error) {


put all in a line

campoy · 2018-06-18T18:58:16Z

PublicGitArchive/pga/cmd/cache.go

+		default:
+		}
+
+		w, err := io.CopyN(dst, src, 512*1024)


why 512*1024? should this be a named constant?

Yes, that was an arbitrary number for testing. I'll change it.

smola · 2018-06-19T12:23:43Z

@campoy Done.

#69 (comment)

The CheckClose idiom is something we use frequently in our codebase to ensure we always handle close errors properly. Handling it explicitly can be quite cumbersome and error prone. Anyway, for this case, since there's not so many calls, I've changed to explicit handling.

campoy · 2018-06-20T17:47:58Z

PublicGitArchive/pga/cmd/cache.go

+const copyBufferSize = 512 * 1024
+
+func cancelableCopy(ctx context.Context, dst io.Writer, src io.Reader) (int64, error) {
+


drop blank line

campoy · 2018-06-20T17:55:48Z

PublicGitArchive/pga/cmd/cache.go

+
+const copyBufferSize = 512 * 1024
+
+func cancelableCopy(ctx context.Context, dst io.Writer, src io.Reader) (int64, error) {


I'm almost tempted to propose this function to Go's stdlib.

That would be great, but I doubt it gets accepted, since nothing in io or io/ioutil is cancellable. Also, there are prominent voices against that usage: Context isn’t for cancellation.

campoy

Small detail, but otherwise LGTM

campoy · 2018-07-02T16:35:16Z

PublicGitArchive/pga/cmd/cache.go

-	if err := wc.Close(); err != nil {
-		return fmt.Errorf("could not close %s: %v", dest.Abs(name), err)
+
+	if err := rc.Close(); err != nil {


I'd say that failing to close a reader is not important, so it's ok to simply do defer rc.Close(),
but I'll let you decide on this.

That depends on the reader implementation. Some of them signal unexpected EOF on Close. You could argue that it is their fault to not use UnexpectedEOF or some other errors on Read though...

Signed-off-by: Santiago M. Mola <santi@mola.io>

Control+C will make pga to shutdown gracefully and clean up after itself, leaving no traces of temporary downloads. Signed-off-by: Santiago M. Mola <santi@mola.io>

Signed-off-by: Santiago M. Mola <santi@mola.io>

smola · 2018-07-20T09:36:41Z

Merging, since we already verified that with this PR we avoid corrupt files.

smola requested a review from campoy June 15, 2018 09:57

bzz mentioned this pull request Jun 15, 2018

Update the CSV file in pga.sourced.tech #66

Open

smola force-pushed the safe-downloads branch from fb2704b to d6e7ebe Compare June 15, 2018 13:21

bzz mentioned this pull request Jun 18, 2018

Question on full PGA integrity verification #53

Open

smola force-pushed the safe-downloads branch from d6e7ebe to 91af4cb Compare June 18, 2018 14:08

campoy suggested changes Jun 18, 2018

View reviewed changes

smola force-pushed the safe-downloads branch from ddc417f to aadca19 Compare June 19, 2018 12:22

campoy approved these changes Jun 20, 2018

View reviewed changes

smola force-pushed the safe-downloads branch from aadca19 to 511ad17 Compare July 2, 2018 16:07

campoy approved these changes Jul 2, 2018

View reviewed changes

bzz mentioned this pull request Jul 17, 2018

Engine fails on siva files from pipeline-staging cluster src-d/jgit-spark-connector#414

Closed

smola added 3 commits July 18, 2018 14:31

pga: download files to temporary file first, fixes src-d#39

9fbda35

Signed-off-by: Santiago M. Mola <santi@mola.io>

pga: make get/list cancellable

a2bec54

Control+C will make pga to shutdown gracefully and clean up after itself, leaving no traces of temporary downloads. Signed-off-by: Santiago M. Mola <santi@mola.io>

explicit Close() error handling

e0126da

Signed-off-by: Santiago M. Mola <santi@mola.io>

smola force-pushed the safe-downloads branch from 511ad17 to e0126da Compare July 18, 2018 12:31

smola merged commit ce68993 into src-d:master Jul 20, 2018

smola deleted the safe-downloads branch July 20, 2018 09:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pga: safe downloads #69

pga: safe downloads #69

smola commented Jun 15, 2018

smola commented Jun 15, 2018

bzz commented Jun 15, 2018 •

edited

Loading

bzz commented Jun 18, 2018 •

edited

Loading

bzz commented Jun 18, 2018

campoy left a comment

campoy Jun 18, 2018

campoy Jun 18, 2018

campoy Jun 18, 2018

smola Jun 19, 2018

smola commented Jun 19, 2018

campoy Jun 20, 2018

campoy Jun 20, 2018

smola Jun 21, 2018

campoy left a comment

campoy Jul 2, 2018

smola Jul 10, 2018

smola commented Jul 20, 2018

		const copyBufferSize = 512 * 1024

		func cancelableCopy(ctx context.Context, dst io.Writer, src io.Reader) (int64, error) {

pga: safe downloads #69

pga: safe downloads #69

Conversation

smola commented Jun 15, 2018

smola commented Jun 15, 2018

bzz commented Jun 15, 2018 • edited Loading

bzz commented Jun 18, 2018 • edited Loading

bzz commented Jun 18, 2018

campoy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smola commented Jun 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

campoy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smola commented Jul 20, 2018

bzz commented Jun 15, 2018 •

edited

Loading

bzz commented Jun 18, 2018 •

edited

Loading