GitHub: Require PR authors to disclose AI usage

[F0bes]

Description of Changes

Add a "Did you use AI blah blah blah" disclosure question for PR authors to answer.

I want this to be a little bit of an RFC. Does anyone else agree / disagree?

Rationale behind Changes

Unfortunately open source seems to be experiencing a increasingly common pattern of AI slop in many forms. Such as bogus CVE reports and pull requests. Avoiding the politics involved with its usage, it's just incredibly time consuming for us when it's used improperly. It would be nice to know when to put on my BS glasses.

[Copilot]

Pull Request Overview

This pull request adds a disclosure question to the PR template so that authors can indicate whether they used AI for finding, testing, or implementing changes.

• Introduced a new question prompting authors to disclose AI usage.
• Updated the PR template file to include checkbox options for both AI usage and non-usage.

[RedDevilus]

The problem is if people lie when they use it, how do you catch them. Also giving besides the benefit of doubt, other people might also use this as an attack for newbie/overzealous coders. I understand the idea ,but will it bring a better environment to the GitHub?

Even well intentioned I won't be surprised if you don't see difference between AI and people who have various degrees of coding. It's also not uncommon to receive help from multiple people to guide with code and is that also in contention by then?

Will it be used as a gotcha like they checked the box did you use it and then they said no on the checkbox but said yes in the comments or got caught in a credible manner. On the other end some legit coders might feel pressured or feel offended on just the implication alone.

Though it does remind me of that time when a Hacktober (October month where people could contribute to multiple repos to win a T-shirt or something) was a mess of doing low-effort or troll commits. So in that case is the essence getting rid of bad PRs in general?

I do get the frustration, and I'm not totally sure if including this or not will truely help alleviate the workflow bottlenecks so to speak.

[Mrlinkwii]

i think its a very good idea , but as red has mentioned , we probably have to vigilant if/when people lie

[F0bes]

The problem is if people lie when they use it, how do you catch them.

People are always going to lie, this is not going to fix that. The largest issue I'm trying to prevent is the possibility of wasting my own or other reviewers time trying to figure out if someone knows something I don't while I review their hallucinated code.

Other people might also use this as an attack for newbie/overzealous coders.

I don't understand this point. If a newbie uses AI and discloses it and it looks completely wrong to the reviewer, the solution isn't to attack them. If a reviewer is attacking PR authors, that isn't the fault of this change. If the reviewer holds authority, that is the fault on us (the PCSX2 team) for allowing someone who represents us to attack others. If the reviewer is just a random person, then we would follow the normal moderation procedure against individuals who exhibit toxic behaviour.

Even well intentioned I won't be surprised if you don't see difference between AI and people who have various degrees of coding.

This is essentially the original point in different words. People are always going to lie, this is not going to fix that.

Will it be used as a gotcha like they checked the box did you use it and then they said no on the checkbox but said yes in the comments or got caught in a credible manner.

Yes. If you check no and then it is found with absolute confidence through interpretation of your code changes, or by your own admission after the fact, that you did use AI, you will be at worst blocked permanently from our GitHub organization. I don't want your (probably wrong) code if you're going to lie to my face.

On the other end some legit coders might feel pressured or feel offended on just the implication alone.

I'm confident that an intermediate programmer can understand why we have have them check that box. If there happens to be a developer out there who gets offended at it, I don't know what to tell them. They must be one of those "Digital Nomads" who happens live under a rock.

So in that case is the essence getting rid of bad PRs in general?

If someone ran the entire codebase through an LLM, asked it to find a bug and propose a PR (this happens), and they see that we require AI disclosure and decide to not PR it out of lack of confidence, that is a win in my eyes.

[AmandaRoseChaqueta]

I 100% agree with this, specially with how a lot of the code used to train those LLMs are probably not very legal. OpenAI and ChatGPT already got on trouble multiple times due to copyright issues.

[chaoticgd]

This is a required.

should probably be

This is required.

or

This is a required field.

Also, I'd probably put it at the bottom instead.

[TheTechnician27]

• Discourages people from using LLM slop, knowing that we don't want it. (The honest ones realize the problem is detrimental enough that we had to add a disclosure; the dishonest ones are more afraid of increased vigilance since the problem is pervasive enough that we hadd to add a disclosure.)
• For the honest people who do still want to forge ahead, it gives them an easy, standardized way to say "Hey, please be extra careful checking my work."
• People who are caught lying on the disclosure form and raise a fit about it can be banned from contributing indefinitely because "hey, we asked". There's no real case for "But I didn't know!"
• Takes two seconds for a PR that at minimum took like 300x as long to write, compile, and test.

sgtm

[TheTechnician27]

@F0bes If we do have a checkbox about AI stuff, would it be reasonable to have a second entry to the effect of "If you did use AI, what did you use it for? Select all that apply." And we can have stuff like linting, boilerplate, identifying a bug, identifying the solution to the bug, writing the PR summary, etc.

[F0bes]

That was the point of the (please explain briefly) when you check yes.
I'd rather just hear it from the author's own words. It'll be easier for them, and it might let us gauge their perspective of their AI usage.

[RedDevilus]

The most important thing if they use it is that they tested they work and get the logic, but it's the same with electrical bikes (still need to be able to drive well with mechanical). A ton of people use it as they don't feel confident or don't have enough experience. It is that last part that just begets annoyance for seemingly random or even broken changes to code. If it's a nitpicky PR it can still be discussed if it's a bonus for the project of course.

I hope that people understand it's just getting rid of well let's just say bad actors and not people who need a guiding hand. I care more about how de facto outcome will improve the situation then just a click and forget thing. But that's something we have to test over the upcoming months and what-ifs don't really help being against or for it.

In the end it's a tool ,but with a lot of things you don't use a hammer where you need a screwdriver. If it's just getting an idea or inspiration or pointing to correct location is totally different than saying give me all the code changes and give me the result which won't make you grow either or become a better coder.

[F0bes]

This is a required.

should probably be

This is required.

or

This is a required field.

Also, I'd probably put it at the bottom instead.

Thanks for pointing that out.

[turtleDev]

This is an interesting discussion.

Companies are actively pushing devs to use AI code editors to increase throughput. Although I have my own reservations against it, it's hard to articulate the reason why it adds to our problems rather than subtracting from it.