How does scoop download files ?

[shodanx2]

Hi,

I'm making an auto install script for my windows installer.
I want it to be fully automated including downloading the files

So I wanted to download this file

https://www.irfanview.info/files/iview473_x64.zip

But that doesn't work with curl, it downloads a file which is just html saying to press a button

So I looked in the .json in the extra bucket

Extras/bucket/irfanview.json

Line 4 in 05fbcce

"homepage": "https://www.irfanview.com",

and they're using the same URL as me

So I tried a variety of commands to try and do it like scoop does it

powershell -command "$ProgressPreference='SilentlyContinue'; Invoke-WebRequest 'https://www.irfanview.info/files/iview473_x64.zip' -OutFile 'iview473_x64.zip' -UseBasicParsing"
powershell -command "$url='https://www.irfanview.info/files/iview473_x64.zip'; $out='iview473_x64.zip'; $hash='433936d26b97057b16026ea6b6c2897e5941e38cd980e424172bc0d980242fd4'; for ($i=1; $i -le 5; $i++) { Remove-Item $out -ErrorAction SilentlyContinue; Invoke-WebRequest -Uri $url -OutFile $out -UseBasicParsing; $actual=(Get-FileHash $out -Algorithm SHA256).Hash; if ($actual -ieq $hash) { echo '✅ Success'; break } else { echo '❌ Hash failed, retrying...'; Start-Sleep -Seconds 2 } }"
powershell -command "$url='https://www.irfanview.info/files/iview473_x64.zip'; $out='iview473_x64.zip'; $headers=@{ 'User-Agent'='Scoop/1.0 (+https://github.com/ScoopInstaller/Scoop)'; 'Accept-Encoding'='gzip, deflate' }; Invoke-WebRequest -Uri $url -OutFile $out -Headers $headers -UseBasicParsing"
powershell -command "$url='https://www.irfanview.info/files/iview473_x64.zip'; $out='iview473_x64.zip'; $headers=@{ 'User-Agent'='Scoop/1.0 (+https://github.com/ScoopInstaller/Scoop)'; 'Accept-Encoding'='gzip, deflate'; 'Connection'='keep-alive' }; Invoke-WebRequest -Uri $url -OutFile $out -Headers $headers -UseBasicParsing"
powershell -command "$url='https://www.irfanview.info/files/iview473_x64.zip'; $out='iview473_x64.zip'; $headers=@{'User-Agent'='Mozilla/5.0'; 'Referer'='https://www.irfanview.com/'}; $ProgressPreference='SilentlyContinue'; Invoke-WebRequest -Uri $url -OutFile $out -Headers $headers -UseBasicParsing"
powershell -command "$url='https://www.irfanview.info/files/iview473_x64.zip'; $out='iview473_x64.zip'; $req=[System.Net.HttpWebRequest]::Create($url); $req.UserAgent='Mozilla/5.0'; $req.Referer='https://www.irfanview.com/'; $res=$req.GetResponse(); $stream=$res.GetResponseStream(); $fs=[System.IO.File]::OpenWrite($out); $buffer=New-Object byte[] 8192; while(($read=$stream.Read($buffer,0,$buffer.Length)) -gt 0){$fs.Write($buffer,0,$read)}; $fs.Close(); $stream.Close(); $res.Close()"
powershell -command "$url='https://www.irfanview.info/files/iview473_x64.zip'; $out='iview473_x64.zip'; $req=[System.Net.HttpWebRequest]::Create($url); $req.UserAgent='Scoop/1.0 (+https://github.com/ScoopInstaller/Scoop)'; $req.Referer='https://www.irfanview.com/'; try { $res=$req.GetResponse() } catch { $res=$_.Exception.Response }; while ($res.StatusCode -match '30[127]') { $url=$res.GetResponseHeader('Location'); $req=[System.Net.HttpWebRequest]::Create($url); $req.UserAgent='Scoop/1.0 (+https://github.com/ScoopInstaller/Scoop)'; $res=$req.GetResponse() }; $stream=$res.GetResponseStream(); $fs=[System.IO.File]::OpenWrite($out); $buffer=New-Object byte[] 8192; while (($read=$stream.Read($buffer,0,$buffer.Length)) -gt 0) { $fs.Write($buffer,0,$read) }; $fs.Close(); $stream.Close(); $res.Close()"
powershell -command "Invoke-WebRequest 'https://github.com/ScoopInstaller/Extras/releases/download/irfanview-4.73/iview473_x64.zip' -OutFile 'iview473_x64.zip'"
powershell -command "Invoke-WebRequest 'https://objects.scoop.sh/files/iview473_x64.zip' -OutFile 'iview473_x64.zip'"

But none of these worked

And I looked into the scoop files core.ps1 and download.ps1

https://github.com/ScoopInstaller/Scoop/blob/b588a06e41d920d2123ec70aee682bae14935939/lib/core.ps1
https://github.com/ScoopInstaller/Scoop/blob/b588a06e41d920d2123ec70aee682bae14935939/lib/download.ps1

and I can't find what they're doing different ?

And it does work

D:\guest\winkit\installers>scoop download irfanview
Updating Scoop...
Updating Buckets...
Scoop was updated successfully!
INFO  Downloading 'irfanview' [64bit] from extras bucket
iview473_x64.zip (3.8 MB) [====================================================================================================================================================================================================] 100%
Checking hash of iview473_x64.zip ... ok.
iview473_plugins_x64.zip (28.1 MB) [===========================================================================================================================================================================================] 100%
Checking hash of iview473_plugins_x64.zip ... ok.
'irfanview' (4.73) was downloaded successfully!

D:\guest\winkit\installers>

I will try and run tcpdump and see what is going on as I'm stumped, how does it even work ??

[z-Fng]

https://github.com/ScoopInstaller/Scoop/blob/b588a06e41d920d2123ec70aee682bae14935939/lib/download.ps1#L94-L96

Please check your request headers. It's always caused by a missing or incorrect Referer/User-Agent header; many servers block requests that don't originate from their own domain or don't come from a real browser.

curl -e https://www.irfanview.info/files -O https://www.irfanview.info/files/iview473_x64.zip

[shodanx2]

Thank you ! That worked for me !