Add a nice little progress bar while downloading binary

[lidizheng]

I stuck for half a hour while installing node-sass as dependency of my project.
After read the source code, I figure out that my download speed from github.com is too slow. Then the problem solved by turning my VPN on.

So I decided to add a progress bar as a reminder, made life easier for people whose network connection is poor.

Downloading

    Start downloading binary at https://github.com/sass/node-sass/releases/download/v3.8.0/darwin-x64-48_binding.node
    Total 2602136 [===============          ] 1566256 60% 4.9s

Success

    Start downloading binary at https://github.com/sass/node-sass/releases/download/v3.8.0/darwin-x64-48_binding.node
    Total 2602136 [=========================] 2602136 100% 0.0s
    Binary downloaded and installed at /Users/skylarzheng/playground/node-sass/vendor/darwin-x64-48/binding.node

[xzyfer]

Thanks @andyxxsd. This has been on my todo list for a while. I'll take a closer look this weekend.

[xzyfer]

-total: parseInt(response.headers['content-length'])
+total: parseInt(response.headers['content-length'], 10),

[xzyfer]

One minor nitpick. Otherwise looks good to me.

[lidizheng]

I have tried couple times. These CI tests just keep failing...
So, at least, I fixed the minor pitfall. XD

[xzyfer]

I've been looking at the progress package. It's pretty much dead and unmaintained at this point. I would look at this alternative - https://github.com/AndiDittrich/Node.CLI-Progress

[saper]

I personally dislike progress bars like this (especially the npm@3 one), wouldn't just a message informing that download starts suffice?

[xzyfer]

I think these are super useful to majority of users. Our install process can be rather slow. I don't blame people for thinking npm has frozen. As an aside I think we might be publishing src/libsass to npm which probably isn't helping matters (I haven't confirmed this).

[xzyfer]

I'm keen to ship a 3.9.0 in the next couple days. We see steady background noise of network issues that I think this will really help with. We can probably handle network errors better in general but I don't think we can beat the user experience of a progress bar. User implicitly know what it means if the progress moves slowly or stops.

@saper what exactly are you concerns? We need a solution for better communicating networking issues, even if this is just temporary stop gap.

[saper]

Let me rant a bit. Actually this is worth a separate post online or something like that.

Very often I am using a debugging technique that requires treating program output captured verbose as the important information.

Because npm logging is notoriously unreliable (not getting compiler output in the log file on Windows etc.) I am using script(1) or GNU screen logging to capture total output from stdout, stderr and whatever else got to the terminal.

I am using this for two specific purposes:

1. capturing log files and comparing them (with diff(1))
2. searching for strange bytes/characters in the output.

For the purpose (1) to work, captured program output should be reliably repeatable and reproducible.
For purposes (1) and (2) to work the amount of strange control characters added to the output should be minimized - because diff tools get confused, and human eye gets confused.
A typical mild offense against reproducibility are timestamps, but they are usually formatted in a standard way designed to be stripped off easily.

People argue that those problems can be avoided by using isatty() and detecting if we are running on the true interactive terminal or if the output is being captured or not (like https://github.com/visionmedia/node-progress/blob/master/lib/node-progress.js#L120). This is wrong for two reasons:

A) it cannot detect situations like script(1) or screen(1) logging, because the two offer pseudoterminal and the program thinks it is running on a real terminal. I've also had situation (not with node) where I had to log remotely via ssh (i.e. log with the ssh client) and - again - the tool happily thinks it is "interactive".

B) it introduces another code path for "interactive" logging and "file" logging, which can be a source of discrepancies between those two (and bugs).
I am already always having those two options in .npmrc:

color=false
progress=false

to kill the worst offenders. If I don't, here's how the log output looks like in the text editor:

A hex dump looks even worse:

0003d2c0  39 6d 20 0d 0a e2 94 82  20 e2 94 82 20 e2 94 82  |9m ..... ... ...|
0003d2d0  20 e2 94 9c e2 94 80 e2  94 80 20 1b 5b 34 30 6d  | ......... .[40m|
0003d2e0  1b 5b 33 33 6d 6a 73 6f  6e 2d 73 63 68 65 6d 61  |.[33mjson-schema|
0003d2f0  40 30 2e 32 2e 32 1b 5b  33 39 6d 1b 5b 34 39 6d  |@0.2.2.[39m.[49m|

ANSI terminal control characters are embedded in the output everywhere and basically make it
unreadable by the text processing tools or humans. Sure, it looks fancy on the users' screen
if all goes well.

Progress bars are even worse, since with every invocation their output needs to be different
(subsequent download may be faster or slower), so basically one needs to try to carefully
remove all this junk before trying to compare logs.

I am aware of tools stripping ANSI sequences, that usually just do some regular expressions. After trying some I had to write my own :(

The next time I am going to debug the strange UTF-8 problem or compare two different npm install runs I'd like to have a consistent program output I can rely on.

I know we have moved on from the times of the ASR 33 teletypes but somehow we cannot get rid of tty-like terminals yet (and some believe it's a good thing).

Regarding networking problems, what worries me is that folk reporting timeouts do not apparently get the message we put to help them (#1681).

[xzyfer]

Would making the progress respect the npm "progress" config address your
use case?

On 26 Aug 2016 10:18 PM, "Marcin Cieślak" notifications@github.com wrote:

Let me rant a bit. Actually this is worth a separate post online or
something like that.

Very often I am using a debugging technique that requires treating program
output captured verbose as the important information.

Because npm logging is notoriously unreliable (not getting compiler output
in the log file on Windows etc.) I am using script(1)
http://manpages.ubuntu.com/manpages/precise/en/man1/script.1.html or GNU
screen logging
https://www.gnu.org/software/screen/manual/html_node/Log.html#Log to
capture total output from stdout, stderr and whatever else got to the
terminal.

I am using this for two specific purposes:

1. capturing log files and comparing them (with diff(1))
2. searching for strange bytes/characters in the output.

For the purpose (1) to work, captured program output should be reliably
repeatable and reproducible.
For purposes (1) and (2) to work the amount of strange control characters
added to the output should be minimized - because diff tools get confused,
and human eye gets confused.
A typical mild offense against reproducibility are timestamps, but they
are usually formatted in a standard way designed to be stripped off easily.

People argue that those problems can be avoided by using isatty()
sass/sassc#177 (comment) and
detecting if we are running on the true interactive terminal or if the
output is being captured or not (like https://github.com/
visionmedia/node-progress/blob/master/lib/node-progress.js#L120). This is
wrong for two reasons:

A) it cannot detect situations like script(1) or screen(1) logging,
because the two offer pseudoterminal and the program thinks it is running
on a real terminal. I've also had situation (not with node) where I had to
log remotely via ssh (i.e. log with the ssh client) and - again - the tool
happily thinks it is "interactive".

B) it introduces another code path for "interactive" logging and "file"
logging, which can be a source of discrepancies between those two (and
bugs yeoman/yo#68).
I am already always having those two options in .npmrc:

color=false
progress=false

to kill the worst offenders. If I don't, here's how the log output looks
like in the text editor:

[image: npmjunk]
https://cloud.githubusercontent.com/assets/248374/18004078/03bc654a-6b92-11e6-997e-a219df468e18.png

A hex dump looks even worse:

0003d2c0 39 6d 20 0d 0a e2 94 82 20 e2 94 82 20 e2 94 82 |9m ..... ... ...|
0003d2d0 20 e2 94 9c e2 94 80 e2 94 80 20 1b 5b 34 30 6d | ......... .[40m|
0003d2e0 1b 5b 33 33 6d 6a 73 6f 6e 2d 73 63 68 65 6d 61 |.[33mjson-schema|
0003d2f0 40 30 2e 32 2e 32 1b 5b 33 39 6d 1b 5b 34 39 6d |@0.2.2.[39m.[49m|

ANSI terminal control characters are embedded in the output everywhere and
basically make it
unreadable by the text processing tools or humans. Sure, it looks fancy on
the users' screen
if all goes well.

Progress bars are even worse, since with every invocation their output
needs to be different
(subsequent download may be faster or slower), so basically one needs to
try to carefully
remove all this junk before trying to compare logs.

I am aware of tools stripping ANSI sequences, that usually just do some
regular expressions. After trying some I had to write my own :(

The next time I am going to debug the strange UTF-8 problem
#1241 or compare two different npm
install runs I'd like to have a consistent program output I can rely on.

I know we have moved on from the times of the ASR 33 teletypes
https://www.youtube.com/watch?v=qv5b1Xowxdk but somehow we cannot get
rid of tty-like terminals
http://lists.gnu.org/archive/html/groff/2016-08/msg00013.html yet (and
some believe it's a good thing).

Regarding networking problems, what worries me is that folk reporting
timeouts do not apparently get the message we put to help them
#1681 (comment) (
#1681 #1681).

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#1649 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAjZWPFdWCK60jT4jGvIOll214CXwRyrks5qjtmVgaJpZM4JbXq8
.

[nschonni]

I think this looks fine, but maybe using the https://github.com/npm/npmlog package should be looked at since then it would all be using the same format

[xzyfer]

@saper's concerns are valid. There should be an opt-out and mirroring the npm config seems sane.

[artyfarty]

Why did you do this?

it even ignores npm --no-progress etc and still spams in logs.

[xzyfer]

We did this because people without the luxury of fast internet, like Australia where I live, can be stuck waiting for a binary to download without any indication of what is happening. This resulted in a steady stream of bugs that was exhausting to deal with.

As for the --no-progress we built in support for that from the beginning. There were some issues in earlier releases but the latest releases should respect it properly. If not please file a separate bug.