Proposal - Container security scanning with Clair

[dtrudg]

Clair is the CoreOS project for security static analysis of containers, scanning them for security issues (from databases of known CVEs). I'd like to propose adding support to sregistry for scanning containers using Clair.

Though Clair is centered around docker or appc images, it has been used to scan openvz templates, which are .tar archives - see FastVPSEestiOu/check_openvz_mirror_with_clair. I'm pretty sure something similar could be done for singularity images.

This is something I'm planning to work on, and thought I'd add a ticket here in case it's of interest to others / there are any thoughts? I'm thinking I will be working to:

• Create a stand alone python tool to scan a singularity image via Clair API
• Investigating contributing this to singularity-python (if it's of interest there)
• Integrate with sregistry - celery jobs for Clair scanning periodically / on push?

Would welcome any input on if this is of interest for sregistry, or more generally.

[vsoch]

I've tried out Clair (I made a Singularity version that was going to be for a check) and I stopped because it was requiring a server. This would definitely be great, but a few thoughts:

• I'm not sure that the registry server itself is the place to have this, although it could be deployed alongside a registry. We would arguably want a check to happen before it is uploaded to the registry, and the entire purpose of clair is to provide a server so the (user) could do this.
• If it is most logical to integrate with the registry, I think it should be optional, but not default. Eg, there is another folder with a Docker-compose file to deploy it, and the docker-compose / settings should have an environment variable that turns checks on/off.

My general thinking is that I like the idea, but I think it should be a separate module from the registry itself. Eg, it really should be running on its own server, and then have different (independent) commands run:

sregistry check container.img clair
sregistry push container.img --name library/collection:tag

The command line client (sregistry) would have two different servers / applications it is pinging to get that functionality. This also makes sense so we don't develop a hairball of a mess of different applications under one roof. The registry should accept and serve images, and be optimized for that. The security bits (+1 for important!) should be done separate to that.

Your approach sounds good! I can definitely add the endpoint for checks to the sregistry client. To be clear on your last point, I don't think the right place is with the singularity registry application itself, but for the client (sregistry).

I anticipate you will run into challenges that clair wants .tar.gz layer things, and we have entire images! Let's use this issue as a board to discuss things as they come up :)

[remyd1]

Refers to:
singularityhub/singularityhub.github.io#33

Or, as a local tool:
apptainer/singularity#818

[dtrudg]

Container scanning with Clair turns out to be nice and straightforward, tested against Clair 2.0.0. A singularity image, exported to tar.gz, masquerades as a single layer docker image for Clair's scanning purposes. I was able to scan an oldish, fat ubuntu singularity image straightforwardly:

Gist - Scan a Singularity image with Clair

I'll go ahead and create a python tool to do this in a bit more friendly manner. I do like the idea of having something like an sregistry check command where a user can get this done.

My personal bias is that container scanning should be an optional feature of the registry. Having scan results collected and stored separate from the container storage location (registry) is a bit awkward. Also, it would be nice for scheduled scans to be able to show people what issues their containers are picking up as they age.

[vsoch]

+1 - if you want to write this into python (or I'd be happy to actually, it doesn't look very long!) I'll add to sregistry! The harder bit is creating and running the server - is there a condensed document we could use to explain how to run a clair server alongside a registry?

[dtrudg]

@vsoch - yeah, getting a scan result is surprisingly easy. There is a bunch of filtering that is probably required though. Clair spits out a lot of CVEs for an up-to-date Ubuntu. They are genuine security vulns, but not of interest to most as the are low/minimal severity. A filter on severity for output, tabulation / html report option would be good. That's something I'd be happy to work up given a bit of time.

How to run a Clair server depends on what you want to do with it. It's feasible people would want something just for sregistry, but maybe they want to use it to check docker images etc. too. The easiest thing is to use the 2 docker containers from arminc with instructions from https://github.com/arminc/clair-local-scan as I did in the Gist - since they have the CVE info already loaded into the DB. Starting a Clair from scratch it takes a long time to pull that info. Possible we could maintain a simple docker-compose.yml somewhere that gives you a Clair server? Production deployments would be expected to be more complex with use of SSL client certs to ensure you aren't sending your image (with maybe confidential stuff) anywhere but to the trusted Clair server you expect.

[vsoch]

@dctrud that would make sense! We can have a plugins folder for the registry with clair as a submodule, or just include the file natively, depending on your preference.

[remyd1]

That is awesome !!!!

If you need some help I created a simple PHP/JS code that parses Json datas to display it as an html tab which can be sort easily. But I think @vsoch prefers the python version.

Another step forward would be to pull and then convert automatically the docker Clair images as singularity images (cron ?) already included in the sregistry. We just need to wait the full integration of singularity services by @bauerm97.

Thanks,
Rémy

[dtrudg]

I'm beginning my step 1 (a stand alone tool that I need) at: https://github.com/dctrud/clair-singularity. Hope to have something usable (with some docs and a docker compose file that'll get clair up easily) by the end of the weekend. After that can chat more about integrating the functionality to the sregistry CLI etc?

@remyd1 - thanks for the code offer, but I'm keeping to Python also.

[vsoch]

+1

[dtrudg]

@vsoch @remyd1 - The initial version at https://github.com/dctrud/clair-singularity is now a working thing with some docs, in case you'd like to try it out. I had a very uninterrupted morning, so got much further than I thought I would.

Now I am starting to think about integration with sregistry there are a few things:

• The standalone tool I've made runs a temporary http server so it can serve the image .tar.gz for Clair to pick up (Clair pulls the image, you don't POST it to Clair). If we want the sregistry CLI client to be able to submit images to Clair that are not yet pushed to sregisty then it would need to do something similar. It's not really a particularly nice thing to do to run a local http server in a client security-wise, nor firewall friendly. If it's done then it definitely needs to support SSL cert verification etc. which would be essential for any production deployment with anything sensitive inside images.

• If we only want to scan things that have already been pushed to sregistry it's easier - There is a centralized sregistry http server for Clair to pull them from

What does @remyd1 need in terms of interacting with Clair/sregistry, and how would the reports be used? It'd be good to know how others might use this, so I don't propose something far too specific to what I have in mind.

[vsoch]

Oh hmm. I think we would arguably want scanning / assurance of security before uploading to sregistry. It It's important that the registry is optimized for just getting and providing images, and doesn't need to do things like wait on scans. I completely agree that "another web server" is not idea, and in fact this is why I stopped pursuing using Clair for singularity checks.

@dctrud - I think if we want to do this right we need some kind of middle ground. I'm guessing Clair prefers being a webby - based thing so it can update itself? Ideally, we could have a (local) client to run clair, with some output to the user every so often to run an update.

[dtrudg]

The trouble with scanning only before upload is that a scan result is a point-in-time thing. You don't just care about whether a container is secure only at the point it is pushed - vulnerabilities come up continuously, and affect an increasing proportion of your existing containers in the registry.

Thinking more about this, I tend to see scans as principally useful when people pull an image. Can imagine scenarios for sensitive data workflows where might e.g. need to blacklist and disallow a pull of a container that's had a critical vuln outstanding for 30 days - or other such things. That type of stuff needs scans of registry contents either internal, or external, to the registry. If we have a user who has generated a bunch of containers, an easy central overview of their current vulnerability status could be a useful thing.

@vsoch - Clair is a persistent database backed web app, with interaction only via the JSON API. The update is a scheduled task that its worker process carries out auto-magically. I don't think it's amenable to being run locally by a client - unless the client can spawn docker/singularity services on demand, which probably isn't realistic.

The embedding into a registry approach is the preferred Clair integration, hence the web service design. A key feature is once a layer is added to Clair then there can be notifications sent when new CVEs come up.

https://github.com/coreos/clair/blob/master/Documentation/running-clair.md

Clair can be integrated directly into a container registry such that the registry is responsible for interacting with Clair on behalf of the user. This type of setup avoids the manual scanning of images and creates a sensible location to which Clair's vulnerability notifications can be propagated. The registry can also be used for authorization to avoid sharing vulnerability information about images to which one might not have access.

And the alternative CI scenario they have is designed around post-push scans:

Clair can be integrated into a CI/CD pipeline such that when a container image is produced, the step after pushing the image to a registry is to compose a request for Clair to scan that particular image. This type of integration is more flexible, but relies on additional components to be setup in order to secure.

No matter what ends up in sregistry I'll definitely be implementing either an external tool to do scheduled scanning of things in the registry - or some kind of fork/plugin to add into sregistry itself. That's something we see a need for here, but understand completely your wish to keep sregistry simple and focused on image storage/serving.

[vsoch]

The trouble with scanning only before upload is that a scan result is a point-in-time thing. You don't just care about whether a container is secure only at the point it is pushed - vulnerabilities come up continuously, and affect an increasing proportion of your existing containers in the registry.

This makes sense, but then I would still argue it would still be reasonable to scan an image on push, and again on pull. The image sitting statically as a file won't be much issue, and it doesn't seem efficient with resources to be constantly scanning it. A huge burden for the registry to be constantly scanning upwards of thousands of containers, plus being available for pushing and pulling from gosh knows how many sources.

Thinking more about this, I tend to see scans as principally useful when people pull an image. Can imagine scenarios for sensitive data workflows where might e.g. need to blacklist and disallow a pull of a container that's had a critical vuln outstanding for 30 days - or other such things. That type of stuff needs scans of registry contents either internal, or external, to the registry. If we have a user who has generated a bunch of containers, an easy central overview of their current vulnerability status could be a useful thing.

This would require still constant scanning of images, which isn't reasonable for a single application. It's better to do one simple function really well than try to do everything and be kind of slow, etc.

@vsoch - Clair is a persistent database backed web app, with interaction only via the JSON API. The update is a scheduled task that its worker process carries out auto-magically. I don't think it's amenable to being run locally by a client - unless the client can spawn docker/singularity services on demand, which probably isn't realistic.

They had a command line client which is still used, but not hugely worked on (see my original issue I posted to them). Could it be brought back to life for Singularity? Another idea - what if we just had clair pointed at the same image base (a folder with subdirectories of images) to fire off scans at some point? And then we would have clair send the registry a ping to flag a container if something bad came up? The main complication here, then, would be needing to access the same image base from both applications, which probably means using the same server (poorer performance) or a more advanced setup (that most don't have).

The embedding into a registry approach is the preferred Clair integration, hence the web service design. A key feature is once a layer is added to Clair then there can be notifications sent when new CVEs come up.

We couldn't use some kind of other triggers?

https://github.com/coreos/clair/blob/master/Documentation/running-clair.md

Clair can be integrated directly into a container registry such that the registry is responsible for interacting with Clair on behalf of the user. This type of setup avoids the manual scanning of images and creates a sensible location to which Clair's vulnerability notifications can be propagated. The registry can also be used for authorization to avoid sharing vulnerability information about images to which one might not have access.

Having managed many web applications that work with different APis, services, I just don't see the average (smaller) institution being able to deploy an infrastructure to handle doing both. Ideally yes, everyone would have kubernetes clusters with a ton of images, but realistically the registry would be deployed on a single left over server, and probably already burdened with just pushing and pulling.

And the alternative CI scenario they have is designed around post-push scans:

I think this is what would be desired, with a signal that is based on the image being saved / changed, but not reliant on the registry itself. Then the registry is notified if/when there is an issue.

Clair can be integrated into a CI/CD pipeline such that when a container image is produced, the step after pushing the image to a registry is to compose a request for Clair to scan that particular image. This type of integration is more flexible, but relies on additional components to be setup in order to secure.

The building / continuous integration is a separate thing entirely, more akin to singularity hub. The registry is agnostic to the build process, that is up to the user/admins. Some might use slurm / other cluster technology, some might use a cloud service, some might use their own computer, Github with CI, or a private server.

No matter what ends up in sregistry I'll definitely be implementing either an external tool to do scheduled scanning of things in the registry - or some kind of fork/plugin to add into sregistry itself. That's something we see a need for here, but understand completely your wish to keep sregistry simple and focused on image storage/serving.

I think this is a good plan. Don't think much about sregistry to start - build something solid that you think is good for its purpose. Then we can start to figure out the integration. I think there are many options and best to do that when the time comes :)

[dtrudg]

For reference, here's what you get on quay.io when looking at a container repo:

And you can go to detailed scan results, e.g.:

https://quay.io/repository/biocontainers/ariba/image/dd247e688e2d5392a899fd38881a281ea51e8011ae0db73eb9f8ebe8758d73a4?tab=vulnerabilities