feat(filesystem): scan in client/server mode #1829

afdesk · 2022-03-14T18:28:08Z

Description

Added a new option --server to filesystem command for support local filesystem scan in client/server mode.

Also there was moved client package to artifact/client.go.

Related issues

Close trivy scan local filesystem not supported in client mode #634

Checklist

I've read the guidelines for contributing to this repository.
I've followed the conventions in the PR title.
I've added tests that prove my fix is effective or that my feature works.
I've updated the documentation with the relevant information (if needed).
I've added usage information (if the PR introduces new options)
I've included a "before" and "after" example to the description (if the PR is a user interface change).

…ion to `fs` command

knqyf263 · 2022-03-15T14:13:02Z

pkg/commands/app.go

@@ -313,6 +312,19 @@ var (
 		EnvVars: []string{"TRIVY_INSECURE"},
 	}

+	remoteServer = cli.StringFlag{
+		Name:    "remote",


I want to rename it with --server, but --remote should be also used as an alias.

knqyf263 · 2022-03-15T14:15:52Z

pkg/commands/artifact/option.go

@@ -18,9 +21,19 @@ type Option struct {
 	option.CacheOption
 	option.ConfigOption

+	// For policy downloading
+	NoProgress bool


We already have it from DBOption.

trivy/pkg/commands/option/db.go

Line 16 in 78b2b89

NoProgress bool

Removed. the comment got confused me.

knqyf263 · 2022-03-15T14:16:10Z

pkg/commands/artifact/option.go

+	RemoteAddr    string
+	token         string
+	tokenHeader   string
+	customHeaders []string
+	// this field is populated in Init()
+	CustomHeaders http.Header


We should define a new struct for these options.

I've created a new struct: RemoteOption

knqyf263 · 2022-03-15T15:38:37Z

pkg/scanner/scan.go

@@ -46,6 +46,12 @@ var StandaloneFilesystemSet = wire.NewSet(
 	StandaloneSuperSet,
 )

+// StandaloneFilesystemRemoteSet binds filesystem dependencies for client/server mode
+var StandaloneFilesystemRemoteSet = wire.NewSet(


This is not standalone, but client/server.

knqyf263 · 2022-03-15T15:58:40Z

pkg/commands/artifact/run.go

+	return remoteCache.NewRemoteCache(c.RemoteAddr, c.CustomHeaders, c.Insecure), nil, nil
+}
+
+func initFSCache(c Option) (cache.ArtifactCache, cache.LocalArtifactCache, error) {


We may want to rename it with initCache and define NopCache like NopCloser.
https://pkg.go.dev/io#NopCloser

The signature could be:

func NopCache(ArtifactCache) Cache

so that RemoteCache can implement cache.Cache. Then, update operation.go as below.

// client/server mode if c.ServerAddr != "" { remoteCache, err = cache. NewRemoteCache(c.ServerAddr, c.CustomHeaders, c.Insecure) if err != nil {...} // Wrap RemoteCache so that it can implement cache.Cache return Cache{Cache: cache.NopCache(remoteCache)}, nil } // standalone mode with local fs cache fsCache, err := cache.NewFSCache(utils.CacheDir()) if err != nil {...} return Cache{Cache: fsCache}, nil

trivy/pkg/commands/operation/operation.go

Lines 59 to 63 in cabd18d

fsCache, err := cache.NewFSCache(utils.CacheDir())

if err != nil {

return Cache{}, xerrors.Errorf("unable to initialize fs cache: %w", err)

}

return Cache{Cache: fsCache}, nil

What do you think of it?

I think it's an interesting idea. I've done it

…r mode

knqyf263 · 2022-03-20T15:43:47Z

pkg/commands/artifact/option_test.go

@@ -177,11 +253,17 @@ func TestOption_Init(t *testing.T) {
 			set.Bool("reset", false, "")
 			set.Bool("skip-db-update", false, "")
 			set.Bool("download-db-only", false, "")
+			set.Bool("clear-cache", false, "")


Is this needed?

no, removed.

pkg/commands/artifact/option_test.go

pkg/rpc/client/client_test.go

knqyf263 · 2022-03-20T15:53:22Z

pkg/commands/option/remote.go

+}
+
+// Init initialize the options for client/server mode
+func (c *RemoteOption) Init(logger *zap.SugaredLogger) (err error) {


Looks like we don't need to return an error.

good catch! removed.

knqyf263 · 2022-03-20T15:58:07Z

integration/client_server_test.go

+				RemoteAddrOption: "--server",
+				Target:           "testdata/fixtures/fs/pom/",
+			},
+			golden: "testdata/fs-pom.json.golder",


We have to use the same golden file with standalone mode. Maybe we have some differences now?

Great idea! done.

knqyf263 · 2022-03-21T13:51:29Z

Thanks!

Co-authored-by: knqyf263 <knqyf263@gmail.com>

afdesk added 7 commits March 15, 2022 00:18

Replace client package to artifact/client.go and add remote opt…

fae7c89

…ion to `fs` command

fix using remote cache and skip downloading db

b18a608

Skip FillVulnerabilityInfo for remore scans

4297d66

Fix goimports-ed

273226f

formatting arguments

81ae669

Fix unit tests

5bcfd42

Fix remote image scan

4d54f51

afdesk mentioned this pull request Mar 15, 2022

feat(client): add local filesystem scan in client/server mode #1742

Closed

6 tasks

afdesk added 2 commits March 15, 2022 18:54

clean a default server address

cf15602

split Cache into ArtifactCache and LocalArtifactCache

e8d5204

knqyf263 reviewed Mar 15, 2022

View reviewed changes

afdesk added 7 commits March 16, 2022 19:50

Merge branch 'main' of github.com:aquasecurity/trivy into fs_remote

f5663f6

Rename the option to --server and retain --remote as an alias.

d1c703a

Remove unnecessary field

9d6c20c

Replace remote options into the separate struct

241223f

Set up a correct name

3796dcf

Fix unit tests for remote options

4537a72

Migrate to common initCache

88c7b4f

afdesk requested a review from knqyf263 March 16, 2022 15:54

afdesk and others added 11 commits March 16, 2022 22:15

Fix goimports-ed

f2dbf89

refactor

f854e5c

Merge branch 'main' of github.com:aquasecurity/trivy into fs_remote

eb90969

(tests fail) Merge unit tests for options

f936701

(tests fail) update unit tests for options

8d486ba

Clean private remote options for testability

7f25b20

Fix unit tests for remote options

26436e3

goimportsfiles

f8d11a3

(tests fail) migrate to httptest

e4e0fb7

Fix marshalling in httptest server

1b31f41

Add usecase when httptest server returns an error

f31e6cc

afdesk added 2 commits March 20, 2022 20:09

Add integration test for scan pox.xml with fs command in client/serve…

3b78bf9

…r mode

Update goldens with a newline at end of file

04d86ec

afdesk marked this pull request as ready for review March 20, 2022 14:18

knqyf263 reviewed Mar 20, 2022

View reviewed changes

afdesk and others added 7 commits March 20, 2022 22:04

Remove clear-cache flag from unit tests for options

71a8978

Remove unusable error

afc3c2d

Using the same golder file for standalone and client/server modes

16df147

change struct to interface{} for error

de2449d

Merge branch 'main' of github.com:aquasecurity/trivy into fs_remote

a32fab6

update documentation for fs scan in client/server mode

d0b51ed

docs: update filesystem scanning

47e66a7

knqyf263 approved these changes Mar 21, 2022

View reviewed changes

knqyf263 merged commit d6418cf into main Mar 21, 2022

knqyf263 deleted the fs_remote branch March 21, 2022 13:51

ankk13 pushed a commit to ankk13/trivy that referenced this pull request Mar 28, 2022

feat(filesystem): scan in client/server mode (aquasecurity#1829)

5b75e41

Co-authored-by: knqyf263 <knqyf263@gmail.com>

liamg pushed a commit that referenced this pull request Jun 7, 2022

feat(filesystem): scan in client/server mode (#1829)

9aca256

Co-authored-by: knqyf263 <knqyf263@gmail.com>

o1oo11oo mentioned this pull request Jun 9, 2023

Enable client/server mode for trivy by default to cache the vulnerability DB (closes #911) secureCodeBox/secureCodeBox#1760

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(filesystem): scan in client/server mode #1829

feat(filesystem): scan in client/server mode #1829

afdesk commented Mar 14, 2022 •

edited

Loading

knqyf263 Mar 15, 2022

afdesk Mar 16, 2022

knqyf263 Mar 15, 2022

afdesk Mar 16, 2022

knqyf263 Mar 15, 2022

afdesk Mar 16, 2022

knqyf263 Mar 15, 2022

afdesk Mar 16, 2022

knqyf263 Mar 15, 2022

afdesk Mar 17, 2022

knqyf263 Mar 20, 2022

afdesk Mar 20, 2022

knqyf263 Mar 20, 2022

afdesk Mar 20, 2022

knqyf263 Mar 20, 2022

afdesk Mar 20, 2022

knqyf263 commented Mar 21, 2022

	fsCache, err := cache.NewFSCache(utils.CacheDir())
	if err != nil {
	return Cache{}, xerrors.Errorf("unable to initialize fs cache: %w", err)
	}
	return Cache{Cache: fsCache}, nil

feat(filesystem): scan in client/server mode #1829

feat(filesystem): scan in client/server mode #1829

Conversation

afdesk commented Mar 14, 2022 • edited Loading

Description

Related issues

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knqyf263 commented Mar 21, 2022

afdesk commented Mar 14, 2022 •

edited

Loading