SIG: Automated Vulnerability Fixing

[JLLeitschuh]

I'd like to propose the creation of a SIG group under this working group focused on automated vulnerability fixing, and potentially also automated vulnerability disclosure with fixes.

There are several individuals engaged in automated vulnerability fixing at-scale, including:

• Myself: @JLLeitschuh
• Moderne (working with myself on the Dan Kaminsky Fellowship)
• OpenRefactory (various Python vulnerabilitys)
• Trellix (Python TAR Slip)
• Google

A working group would allow us to establish a set of proposed norms for the industry, and propose & discuss auto fix campaigns.

[pombredanne]

Un-sollicited bot PRs like what the one Trellix has been engaged in are essentially mass spam and/or spamvertizing. And are likely in direct violation of the GitHub terms of service.

I think there is no socially acceptable and ethically acceptable norms for auto fixing that will pass muster. Do not do auto fix in the wild. Use real people that submit PR and that reply and follow through.

[JasonKeirstead]

I second this request. It makes a lot of sense to align with other groups performing the same activities, to coordinate and avoid duplication. @pombredanne I understand your objection, but the ship has already sailed on this activity so a working group makes a lot of sense to figure out best practices.

[JLLeitschuh]

And are likely in direct violation of the GitHub terms of service.

These campaigns have been undertaken with direct collaboration with the GitHub Security Lab. AFAIK, all campaigns to-date have collaborated with GitHub directly to avoid TOS violations

[JLLeitschuh]

I think there is no socially acceptable and ethically acceptable norms for auto fixing that will pass muster.

What ethical framework are you arguing this under?

Let's take, as an example, that you become aware of a widespread common security vulnerability impacting tens of thousands of OSS projects. You have the following options, IMHO:

1. Automate disclosing the vulnerability at-scale
2. Create a team of people to fix the vulnerability, by-hand at-scale
3. Write a blog post, and then do nothing
4. Automate fixing the vulnerability at-scale

Option 1, automated disclosure, overwhelms a maintainer, and isn't directly actionable. They have to spend cycles developing a fix manually. And if the automation isn't exact, it's spammy.

Option 2, manual disclosure, is quite expensive to do for every vulnerability. Software developers aren't cheap, and software security researchers, even more so.

Option 3, blogging about it, puts the vulnerability into the public, but doesn't actually try to fix the vulnerability where it exists. Doesn't this have ethical ramifications of its own? A critical vulnerability is disclosed, and everyone impacted is still at-risk. Many will never read the disclosure, but the attackers will.

Option 4, automating the fix and contributing it, has the lowest cognitive impact on the maintainer. It's highly actionable. In the worst case, a vulnerability fix should be a valid security hardening. In the best case, it should fix a real security vulnerability.

[xcorail]

These campaigns have been undertaken with direct collaboration with the GitHub Security Lab. AFAIK, all campaigns to-date have collaborated with GitHub directly to avoid TOS violations

@JLLeitschuh the Trellix campaign was not done in direct collaboration with the GitHub Security Lab.

[xcorail]

Un-sollicited bot PRs like what the one Trellix has been engaged in are essentially mass spam and/or spamvertizing. And are likely in direct violation of the GitHub terms of service.
I think there is no socially acceptable and ethically acceptable norms for auto fixing that will pass muster. Do not do auto fix in the wild. Use real people that submit PR and that reply and follow through.

I think what @pombredanne is asking is reasonable: No bot PR, but a human that replies and follow through. What @JLLeitschuh was doing is close to that. He automated the generation of the PRs, but they were submitted under his account, and he was following through.

[JasonKeirstead]

It feels like this is going off topic.

This GHE issue is not requesting permission to do automated vuln fixing. That process is already happening, OpenSSF is already funding it under Alpha/Omega, debates & conversations about that probably need to be taken there. Trellix and Google are already doing it on their own as well, completely outside OpenSSF.

The request is to create a WG in order for people who decide to do this, to coordinate and figure out best practices. It is not "should this be done, at all" - it is already happening, and this WG has no control over that.

[JLLeitschuh]

The request is to create a WG in order for people who decide to do this, to coordinate and figure out best practices. It is not "should this be done, at all" - it is already happening, and this WG has no control over that.

I think that we should be careful about this mentality as it will be an immediate turn off for maintainers engaging with us. There is clearly some pain these maintainers are experiencing. I want to also give them a place to discuss with us their own concerns and pain as well so we can learn and improve these norms in response to that.

[zmanion]

Of interest to me, and I think the WG should certainly be respectful of "forcing unwanted help on maintainers," but I'd be OK deciding that the WG assumes automated fixing should be performed and focusing on how to do so effectively, efficiently, and respectfully (of project/maintainer effort and self-determination).

I'd expect a "how" discussion to lead to some anti paterns/practices, i.e., "do not automate in this way."

[laurie-tyz]

One way to to be proactive during installation, prompt the installer to select one of the following:

1. Notify me of every patch
2. Notify me and automatically install
3. Don't notify me (they may have software controls and do their own testing and patching.)

[JLLeitschuh]

If you'd like to participate in this sub-working group, please provide your availability in this doodle poll:
https://doodle.com/meeting/participate/id/boZ964Ya

There is also a slack channel dedicated to this topic now: https://openssf.slack.com/archives/C04MW17FK8X

[johnandersen777]

Just as a related FYI. In ietf-scitt/use-cases#14 we're playing with enabling reporting and fixing leveraging federation of transparency service claims and receipts to secure comms and facilitate eventing. Ideally this work ties in eventually with #94 (comment) for automated vuln submission by entities who would then ideally want to submit fixes for those vulns: #124

If someone could please invite me to that slack channel that would be awesome. Thank you! My email is johnandersenpdx (at) gmail.com

[SecurityCRob]

At this time, due to level of activity, this work has been archived.