fetchnvd Fails using example loop

[kitplummer]

Environment

Vuls

Hash : ____

To check the commit hash of HEAD
$ vuls -v

➜  Vuls docker run --rm -i vuls/vuls -v
vuls v0.1.7 6d82ad3

➜  Vuls docker run --rm -i vuls/go-cve-dictionary -v
go-cve-dictionary v0.1.0 2dd369d

Addition Details

Can you also please fill in each of the remaining sections.

Expected Behavior

For it to fetch all the years sequentially.

Actual Behavior

Fetches the first year, fails after that. Fetching each year individually works.

Steps to reproduce the behaviour

➜  Vuls time for i in {2002..2016}; do \
    docker run --rm -it \
    -v $PWD:/vuls \
    -v $PWD/go-cve-dictionary-log:/var/log/vuls \
    vuls/go-cve-dictionary fetchnvd -years $i; \
  done
 0 / 1    0.00%[Nov 15 14:18:49]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2002.xml.gz
[Nov 15 14:18:52]  INFO Fetched 6721 CVEs
[Nov 15 14:18:52]  INFO Opening DB (sqlite3).
[Nov 15 14:18:52]  INFO Migrating DB
[Nov 15 14:18:52]  INFO Inserting CVEs...
 6721 / 6721 [=====================================================================================================================================================================================================================================================] 100.00% 39s
[Nov 15 14:19:32]  INFO Refreshed 6721 Nvds.
 0 / 1    0.00%[Nov 15 14:19:34]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2003.xml.gz
[Nov 15 14:19:44] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2003.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:43620->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2003.xml.gz]
 0 / 1    0.00%[Nov 15 14:19:45]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2004.xml.gz
[Nov 15 14:19:55] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2004.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:48016->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2004.xml.gz]
 0 / 1    0.00%[Nov 15 14:19:57]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2005.xml.gz
[Nov 15 14:20:07] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2005.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:35175->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2005.xml.gz]
 0 / 1    0.00%[Nov 15 14:20:08]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2006.xml.gz
[Nov 15 14:20:18] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2006.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:40533->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2006.xml.gz]
 0 / 1    0.00%[Nov 15 14:20:20]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2007.xml.gz
[Nov 15 14:20:30] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2007.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:35287->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2007.xml.gz]
 0 / 1    0.00%[Nov 15 14:20:32]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2008.xml.gz
[Nov 15 14:20:42] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2008.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:49168->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2008.xml.gz]
 0 / 1    0.00%[Nov 15 14:20:44]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2009.xml.gz
[Nov 15 14:20:54] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2009.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:57437->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2009.xml.gz]
 0 / 1    0.00%[Nov 15 14:20:56]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2010.xml.gz
[Nov 15 14:21:06] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2010.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:49489->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2010.xml.gz]
 0 / 1    0.00%[Nov 15 14:21:07]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2011.xml.gz
[Nov 15 14:21:17] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2011.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:39689->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2011.xml.gz]
 0 / 1    0.00%[Nov 15 14:21:19]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2012.xml.gz
[Nov 15 14:21:29] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2012.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:35709->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2012.xml.gz]
 0 / 1    0.00%[Nov 15 14:21:32]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2013.xml.gz
[Nov 15 14:21:42] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2013.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:42221->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2013.xml.gz]
 0 / 1    0.00%[Nov 15 14:21:44]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2014.xml.gz
[Nov 15 14:21:54] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2014.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:43332->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2014.xml.gz]
 0 / 1    0.00%[Nov 15 14:21:55]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2015.xml.gz
[Nov 15 14:22:05] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2015.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:43753->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2015.xml.gz]
 0 / 1    0.00%[Nov 15 14:22:06]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2016.xml.gz
[Nov 15 14:22:16] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2016.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:49990->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2016.xml.gz]
➜  Vuls time for i in {2003..2016}; do \
    docker run --rm -it \
    -v $PWD:/vuls \
    -v $PWD/go-cve-dictionary-log:/var/log/vuls \
    vuls/go-cve-dictionary fetchnvd -years $i; \
  done
 0 / 1    0.00%[Nov 15 15:19:38]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2003.xml.gz
[Nov 15 15:19:40]  INFO Fetched 1519 CVEs
[Nov 15 15:19:40]  INFO Opening DB (sqlite3).
[Nov 15 15:19:40]  INFO Migrating DB
[Nov 15 15:19:40]  INFO Inserting CVEs...
 1519 / 1519 [======================================================================================================================================================================================================================================================] 100.00% 9s
[Nov 15 15:19:49]  INFO Refreshed 1519 Nvds.
 0 / 1    0.00%[Nov 15 15:19:50]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2004.xml.gz
[Nov 15 15:20:00] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2004.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:52807->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2004.xml.gz]
 0 / 1    0.00%[Nov 15 15:20:02]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2005.xml.gz
[Nov 15 15:20:12] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2005.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:40495->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2005.xml.gz]

[kotakanbe]

I did not reproduce it in my environment.
Does anyone have the same problem?

[kitplummer]

Probably should’ve mentioned it in my original post that I’m trying to run this in a Docker instance on a the native-Mac runtime. Urgh.

On Nov 15, 2016, at 7:56 PM, Kota Kanbe notifications@github.com wrote:

I did not reproduce it in my environment.
Does anyone have the same problem?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub #267 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/AAAQmQ8F_DyHTBbKSCx6hLnBpBkt3JM7ks5q-nDVgaJpZM4KyrrA.

[Code0x58]

I'm wondering if this is an issue with the shell in the container either not being Bash, or it having some settings which are causing the issue, as the first iteration of the loop works each time.

I will have more of a look tonight, but you can pass -years 2002 2003 2004 … to fetchnvd, which is probably what should be documented.

[kitplummer]

I'm also seeing it sporadically:

➜  Vuls time docker run --rm -it vuls/go-cve-dictionary fetchnvd -years 2010
 0 / 1    0.00%[Nov 16 16:31:11]  INFO Fetching... https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2010.xml.gz
[Nov 16 16:31:21] ERROR Failed to fetch cve data from NVD. err: [HTTP error. errs: [Get https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2010.xml.gz: dial tcp: lookup static.nvd.nist.gov on 192.168.65.1:53: read udp 172.17.0.2:60340->192.168.65.1:53: i/o timeout], url: https://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2010.xml.gz]
docker run --rm -it vuls/go-cve-dictionary fetchnvd -years 2010  0.01s user 0.02s system 0% cpu 11.774 total

It'll work, then not work.

[Code0x58]

Looking at the log messages, they show UDP to port 53 and lookup static.nvd.nist.gov which suggest that this is a DNS problem on your side.

This may be of some help to you, as it looks like it may be a Mac issue (this image is based on Denian Jessie rather than Alpine).