Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v0.5.0 (no backwards compatibility) #478

Merged
merged 132 commits into from
Aug 27, 2018
Merged

v0.5.0 (no backwards compatibility) #478

merged 132 commits into from
Aug 27, 2018

Conversation

kotakanbe
Copy link
Member

@kotakanbe kotakanbe commented Aug 31, 2017

Clever vulnerability detection of non-OS-packages

https://vuls.io/docs/en/usage-scan-non-os-packages.html

Before v0.5.0, Vuls can detect only by exact match of CPE.
After v0.5.0, Vuls supports NVD JSON Feed
NVD JSON Feed has detailed affected version information.

        "cpe" : [ {
          "vulnerable" : true,
          "cpe22Uri" : "cpe:/a:ibm:security_guardium",
          "cpe23Uri" : "cpe:2.3:a:ibm:security_guardium:*:*:*:*:*:*:*:*",
          "versionEndIncluding" : "8.2"
        }, {

Using this information, Vuls can detect vulnerabilities of non-OS-packages more cleverly than before, because vuls compares versions of specified CPEs in config.toml and the version in the feed.

related ( #599 )


Significant improvement in scanning

Support knqyf263/gost (go-security-tracker)

New version Vuls 0.5.0 now possible to detect vulnerabilities that patches have not been published from distributors using new datasource named gost.

RedHat API

before

One Line Summary
================
c74     Total: 38 (High:12 Medium:18 Low:7 ?:1) 36/36 Fixed     708 installed, 288 updatable

after (with gost)

One Line Summary
================
c74     Total: 459 (High:48 Medium:326 Low:84 ?:1)      36/457 Fixed    708 installed, 288 updatable

Debian Security Tracker

before

One Line Summary
================
deb8    Total: 105 (High:22 Medium:39 Low:9 ?:35)       55/103 Fixed    513 installed, 43 updatable

after (with gost)

One Line Summary
================
deb8    Total: 524 (High:56 Medium:148 Low:21 ?:299)    53/522 Fixed    512 installed, 43 updatable


HTTP


Enable to define scan mode for each servers in config.tom

#510 #669

fast scan with internet access

  • config.toml
[servers]

[servers.localhost]
host         = "192.168.100.111" # or "127.0.0.1"
port         = "22"
scanMode     = ["fast"]

fast scan without internet access

  • config.toml
[servers]

[servers.localhost]
host         = "192.168.100.111" # or "127.0.0.1"
port         = "22"
scanMode     = ["fast", "offline"]

fast-root scan

fast-root scan with internet access

  • config.toml
[servers]

[servers.localhost]
host         = "192.168.100.111" # or "127.0.0.1"
port         = "22"
scanMode     = ["fast-root"]

fast-root scan without internet access

  • config.toml
[servers]

[servers.localhost]
host         = "192.168.100.111" # or "127.0.0.1"
port         = "22"
scanMode     = ["fast-root", "offline"]

deep scan

deep scan with internet access

  • config.toml
[servers]

[servers.localhost]
host         = "192.168.100.111" # or "127.0.0.1"
port         = "22"
scanMode     = ["deep"]

Reports

The following items were added

  • Display CWE name
  • OWASP TOP 10
  • Affected Packages, Processes
  • Mitigation

TUI

screen shot 2018-07-24 at 16 59 28


report -format-full-text

screen shot 2018-07-25 at 22 26 50


report -format-list

screen shot 2018-07-24 at 23 21 43


Slack Notification

screen shot 2018-07-18 at 21 24 40


Stride Notification

https://vuls.io/docs/en/usage-report.html#example-send-scan-results-to-stride
#624
2018-03-31 14 52 57


Charwork Notification

#634
https://vuls.io/docs/en/usage-report.html#example-send-scan-results-to-chatwork

2018-04-25 17 35 42


go-cve-dictionary list subcommand

vulsio/go-cve-dictionary#80

screen shot 2018-08-23 at 14 43 27


Change the format of ScanResult

Model has changed.
https://github.com/future-architect/vuls/tree/master/models


Change the format of config.toml

$ ./vuls discover 127.0.0.1/32
# Create config.toml using below and then ./vuls -config=/path/to/config.toml

[cveDict]
type = "sqlite3"
path = "/path/to/cve.sqlite3"
#url = ""

[ovalDict]
type = "sqlite3"
path = "/path/to/oval.sqlite3"
#url = ""

[gost]
type = "sqlite3"
path = "/path/to/gost.sqlite3"
#url = ""

# https://vuls.io/docs/en/usage-settings.html#slack-section
#[slack]
#hookURL      = "https://hooks.slack.com/services/abc123/defghijklmnopqrstuvwxyz"
##legacyToken = "xoxp-11111111111-222222222222-3333333333"
#channel      = "#channel-name"
##channel     = "${servername}"
#iconEmoji    = ":ghost:"
#authUser     = "username"
#notifyUsers  = ["@username"]

# https://vuls.io/docs/en/usage-settings.html#email-section
#[email]
#smtpAddr      = "smtp.example.com"
#smtpPort      = "587"
#user          = "username"
#password      = "password"
#from          = "from@example.com"
#to            = ["to@example.com"]
#cc            = ["cc@example.com"]
#subjectPrefix = "[vuls]"

# https://vuls.io/docs/en/usage-settings.html#http-section
#[http]
#url = "http://localhost:11234"

# https://vuls.io/docs/en/usage-settings.html#syslog-section
#[syslog]
#protocol    = "tcp"
#host        = "localhost"
#port        = "514"
#tag         = "vuls"
#facility    = "local0"
#severity    = "alert"
#verbose     = false

# https://vuls.io/docs/en/usage-report.html#example-put-results-in-s3-bucket
#[aws]
#profile                = "default"
#region                 = "ap-northeast-1"
#s3Bucket               = "vuls"
#s3ResultsDir           = "/path/to/result"
#s3ServerSideEncryption = "AES256"

# https://vuls.io/docs/en/usage-report.html#example-put-results-in-azure-blob-storage<Paste>
#[azure]
#accountName   = "default"
#accountKey    = "xxxxxxxxxxxxxx"
#containerName = "vuls"

# https://vuls.io/docs/en/usage-settings.html#stride-section
#[stride]
#hookURL   = "xxxxxxxxxxxxxxx"
#authToken = "xxxxxxxxxxxxxx"

# https://vuls.io/docs/en/usage-settings.html#hipchat-section
#[hipchat]
#room      = "vuls"
#authToken = "xxxxxxxxxxxxxx"

# https://vuls.io/docs/en/usage-settings.html#chatwork-section
#[chatwork]
#room     = "xxxxxxxxxxx"
#apiToken = "xxxxxxxxxxxxxxxxxx"

# https://vuls.io/docs/en/usage-settings.html#default-section
[default]
#port               = "22"
#user               = "username"
#keyPath            = "/home/username/.ssh/id_rsa"
#scanMode           = ["fast", "fast-root", "deep", "offline"]
#cpeNames = [
#  "cpe:/a:rubyonrails:ruby_on_rails:4.2.1",
#]
#owaspDCXMLPath     = "/tmp/dependency-check-report.xml"
#ignoreCves         = ["CVE-2014-6271"]
#containerType      = "docker" #or "lxd" or "lxc" default: docker
#containersIncluded = ["${running}"]
#containersExcluded = ["container_name_a"]

# https://vuls.io/docs/en/usage-settings.html#servers-section
[servers]

[servers.127-0-0-1]
host                = "127.0.0.1"
#port               = "22"
#user               = "root"
#keyPath            = "/home/username/.ssh/id_rsa"
#scanMode           = ["fast", "fast-root", "deep", "offline"]
#type               = "pseudo"
#memo               = "DB Server"
#cpeNames            = [ "cpe:/a:rubyonrails:ruby_on_rails:4.2.1" ]
#owaspDCXMLPath     = "/path/to/dependency-check-report.xml"
#ignoreCves         = ["CVE-2014-0160"]
#containerType      = "docker" #or "lxd" or "lxc" default: docker
#containersIncluded = ["${running}"]
#containersExcluded = ["container_name_a"]

#[servers.127-0-0-1.containers.container_name_a]
#cpeNames        = [ "cpe:/a:rubyonrails:ruby_on_rails:4.2.1" ]
#owaspDCXMLPath = "/path/to/dependency-check-report.xml"
#ignoreCves     = ["CVE-2014-0160"]

#[servers.127-0-0-1.optional]
#key = "value1"

Enable to define CpeNames for each container.

#668

To detect the vulnerability of Ruby on Rails v4.2.1 on specific container, cpeNames needs to be set in the servers>containers section.
The following is an example of running Ruby on Rails v4.2.1 on dockerA.

[servers]

[servers.172-31-4-82]
host         = "172.31.4.82"
user        = "ec2-user"
keyPath     = "/home/username/.ssh/id_rsa"
containerType = "docker"
containersIncluded = ["${running}"]

[servers.172-31-4-82.containers.dockerA]
cpeNames = [
    "cpe:/a:rubyonrails:ruby_on_rails:4.2.1",
]

Enable to define a path of OWASP Dependency Check XML for each container.

#667

The following is an example of how to specify a XML of OWASP DC to the specific container.

[servers]

[servers.172-31-4-82]
host         = "172.31.4.82"
user        = "ec2-user"
keyPath     = "/home/username/.ssh/id_rsa"
containerType = "docker"
containersIncluded = ["${running}"]

[servers.172-31-4-82.containers.dockerA]
owaspDCXMLPath = "/tmp/dependency-check-report.xml"

Enable to define ignoreCVEs for each container.

#666

[servers]

[servers.172-31-4-82]
host         = "172.31.4.82"
user        = "ec2-user"
keyPath     = "/home/username/.ssh/id_rsa"
containerType = "docker"
containersIncluded = ["${running}"]

[servers.172-31-4-82.containers.dockerA]
ignoreCves = ["CVE-2016-6314"]

Add ignorePkgsRegexp for each host, container in config.toml

#665

Format

[servers.192-168-11-6]
host         = "192.168.11.6"
ignorePkgsRegexp = ["^kernel.*"]
containersIncluded = ["${running}"]
ContainersExcluded= ["container_name_a"]

[servers.192-168-11-6.containers.container_name_a]
ignorePkgsRegexp = ["^vim.*"]

  • Ignore vulnerabilities in packages matching to patterns defined by ignorePkgs in config.toml
  • Can be defined for both host and container
  • Can be described with regular expressions
  • Match with pkgname

optional field

before

[servers.172-31-4-82]
host         = "172.31.4.82"
optional = [
    ["key", "value"],
    ["key2", "value2"],
]

after

[servers.172-31-4-82]
host         = "172.31.4.82"
[servers.172-31-4-82.Optional]
key = "value"
key2 = "value2"

(fast-root and deep) Detect processes affected by next update using yum-ps #482

Issues yum ps to detect processed affected by next software update on RedHat, CentOS, Amazon , Oracle.
AffectedProcs key under Packages will be added by this P/R.

  • result.json
{
  "Packages" : {
    "wpa_supplicant": {
      ...,
      "AffectedProcs": [
        {
          "PID": "638",
          "ProcName": "wpa_supplicant",
        }
      ]
    },
  },
}

(fast-root and deep) Detect need-restarting-processes for Redhat, CentOS, Ubuntu and Debian


Add -uuid option to report subcommand

When Servername was changed, Vulsrepo etc could not look back in the past. So, I added a unique UUID that will not be changed. If you report with this flag on, Vuls generates UUIDs automatically.
This Auto-generated UUID will be added to the config.toml automatically and result JSON.
The previous config.toml will be renamed to config.toml.bak.

Add memo field to Server section

When reporting with the -uuid flag on, config.toml is automatically recreated as described above. So I added the memo field because the TOML comment disappears. For memo about the server, please use this field, not TOML comment.


How to install

How to upgrade

@kotakanbe kotakanbe force-pushed the v0.5.0 branch 10 times, most recently from 391c86a to a235c4f Compare September 5, 2017 05:53
Detect processes affected by update using yum-ps
@kotakanbe kotakanbe force-pushed the v0.5.0 branch 2 times, most recently from 097f5be to 2bda3a7 Compare September 14, 2017 19:44
kotakanbe and others added 2 commits October 16, 2017 19:31
* master:
  Fix OVAL detection on Debian and Ubuntu (#509)
  Add filter options to tui subcommand (#508)
  Add filter options to tui subcommand (#508)
  Add -ignore-unfixed option to report subcommand #485 (#507)
  Add goreleaser to distribute binaries (#460)
  Add goreportcard to README
  Support SUSE Enterprise Linux (#487)
  Debian: Use --showformat flag to get status of packages and ignore n(not-inst… (#484)
  fix vulsrepo dockerfile (#496)
  Update README.ja.md (#498)
  Fix typos (#499)
  Check repoquery with sudo nopasswd in deep scan mode on RedHat (#492)
sadayuki-matsuno and others added 5 commits October 26, 2017 12:52
* master:
  Fix yum changelog option (#543)
  Update README.md (#542)
  Fix false positive: ignore oval info when kernel major version is different. (#541)
  Fix: Misdetection of OvalMatch for CentOS and Scientific in oval/util.go (#536)
  Fix: "Reboot Required" detection process in scan/redhat.go (#534)
  Remove README.fr.md because unable to maintenance..
  README Typo Update (#538)
  Insert `sudo` only at the beginning of command in deep scan #495 (#539)
  Fix config.toml validation (#537)
  Add pseudo server type for non-ssh scanning (only cpe scan) #512 (#531)
  README Update (#530)
  Update README
  send slack msg by api (#525)
  fix typo Privious -> Previous (#523)
  Fix package query fails on debian based container (#519) (#522)
  fix typo from "enviroment" to "environment" (#518)
  Fix typo in config/jsonloader.go (#517)
  Modified Spell Miss of "README.md". (#516)
  Fix .goreleaser.yml
  Fix a bug of making channels when fill oval information via HTTP (#514)
* master:
  Update README (#547)
  Support Alpine Linux #194 (#545)
  Replace strings.HasPrefix with strings.Index for SuSE scanner (#546)
* master:
  Remove empty CveContent output to JSON with Alpine Linux scan (#550)
* master:
  Fixed panic occurred when blank line continued in changelog (#569)
  Fixed Typo (#574)
  fix typo(BackSpace) in README.ja.md (#576)
  Fix a bug of sending to closed socket while oval access via HTTP #578 (#579)
  fix typo in reports/s3.go (#573)
  LXC container support without LXD (#552)
  Support Amazon Linux2 (#562)
  Fix #548 and #557 - RHEL's Fast Scan no longer required internet connection and root privilege (#559)
  Change error handling of "Reboot Required" detection (#556)
  Fix some RPM related commands failed in the container (#554)
  travis: update go version (#555)
  Add more kernel related packages (Fix #541) (#551)
@kotakanbe kotakanbe mentioned this pull request Jan 19, 2018
kotakanbe and others added 7 commits August 1, 2018 12:38
…md opt (#690)

* feat(config): enable to set options in config.toml instead of cmd opt

* fix(config): change Conf.Report.Slack to Conf.Slack

* fix(discover): change tempalte

* fix(report): fix config.toml auto-generate with -uuid
kotakanbe and others added 2 commits August 8, 2018 12:59
* add(report) add -to-saas option

* ignore other writer if -to-saas

* fix(saas) fix bug
@kotakanbe kotakanbe changed the title [WIP]v0.5.0 (no backwards compatibility) v0.5.0 (no backwards compatibility) Aug 27, 2018
@kotakanbe kotakanbe merged commit 44fa2c5 into master Aug 27, 2018
@kotakanbe kotakanbe deleted the v0.5.0 branch August 27, 2018 05:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants