AI-generated descriptions in issues and PRs

[chenlica]

I see more issues and PRs have their descriptions generated by AI tools. Such descriptions are very verbose and hard to understand. High-level ideas are lost in the long text. I understand a benefit of such long descriptions is that they can make other AI tools understand it better. But I am concerned about the low understandability for humans.

I would like to start this discussion about a good practice of these descriptions in this project.

[sshiv012]

I looked at a few approaches. Team NewPipe prohibits using generative AI to fill issue or PR templates, while FastAPI permits AI assistance for reports but requires human verification and disclosure. Apache guidance likewise keeps responsibility for generated material with the contributor. More generally, GitHub recommends descriptions that let reviewers understand a change quickly.

A balanced practice for Texera could be: require a short, human-edited summary first (problem/motivation, proposed change, and how it was verified, I believe we have a template like this for pull requests); put optional implementation detail below; disclose substantial AI assistance; and require the author to understand the submission and answer questions about it. The same compact template could apply whether or not AI was used, since verbosity rather than the tool itself is the immediate readability problem.

Alternatively we can also look at CodeRabbit. I believe they have a feature specifically aimed at AI slop, but I wouldn't trust any for profit company in terms of sustainable long term future. We'd also have to check if Apache permits this.

[Yicong-Huang]

I do not think we can, or should, prevent agents from drafting contribution materials. Agentic tools can increase contribution velocity severalfold, and requiring a human-written or substantially human-edited summary for every contribution could discourage their use and reduce throughput.

I lean toward solving agent-related problems agentically. For example, we can build conciseness checks directly into the tools that generate issues, PR descriptions, and reviews. I am interested in exploring more solutions along these lines while still holding contributors responsible for the clarity and accuracy of what they submit. I like the CodeRabbit tool you mentioned but I doubt ASF can allow us to use it. maybe we can try it?

[sshiv012]

Yes I think we should try it, there is no policy I could find that strictly prohibits the use of such tooling or I couldn't find any strong incompatibility in terms of license. As long as you guys (the pmc) are okay we can give it a shot! I can look into the integration @chenlica @Yicong-Huang

[Yicong-Huang]

feel free to give it a try and report back! we might need some evaluations before we discuss and decide whether onboard it or not.

[chenlica]

@sshiv012 Please feel free to explore this CodeRabbit tool and report your findings.

[parshimers]

I've started to see this too in AsterixDB and I don't really care much for it either. In code it's easy to mark what was generated or not. In a discussion or other forum where it's supposed to be humans talking to humans, I think it leaves a bad impression on the reader if the text is obviously not written by a human and isn't clearly marked as such.
I like Suryaa's idea of at least enforcing that an abstract be written if it is a long issue. This way you can be certain the author actually read the generated output and didn't pass the buck onto you to determine whether or not it even makes sense.

[chenlica]

@sshiv012 @parshimers Thanks for the input. I like the idea of requiring the author to write a brief summary at the beginning of the description.

Let's wait for other members to chime in.

[aglinxinyuan]

I also like the idea of summary.

[zyratlo]

Here's my opinion. Issues are meant to be more abstract, describing a problem or suggesting a fix, so I don't think exact implementation details belong there. Human readability matters more, and diagrams or pictures can add a lot. Some technical detail is fine for developers who can read and follow it, but there's such a thing as being so verbose that only agents can get through it.

PRs are more technical by nature since the code changes are right there. But if the description is too verbose, reading it is basically the same as reading the source, which defeats the point of having a description at all. I agree the right balance is a section at the top written for humans, and optionally a longer technical explanation below that's mainly for agents.

If the real problem is agents writing descriptions that are too verbose and hard to read, we can require the poster to review the agent's output before publishing. The goal would be to improve human readability and cut down on wordiness, instead of assuming the generated content is good enough to post as-is.

[Neilk1021]

I agree with this, many current git issues are verbose to the point they discourage engagement and often necessitate the use of an agent even if the underlying behavior isn't relatively complex. A human generated summary potentially acts as a safe-guard against this.

[Yicong-Huang]

yes! issues should describe issues/problems/bugs/tasks/todos, not implementation details. PR description should not talk about line/file level changes. it should stay high level as well.

but I also find a hard time to prevent agents from doing so. I tried it with the plugin tool I mentioned in #7377 (comment) and it works most of time but still sometimes slips away. Any suggestions or feedbacks?

[aglinxinyuan]

The PR descriptions used to be simple until a certain point, when everyone started writing much more.
I don't think the problem is AI-generated descriptions. AI can also generate simple and clear descriptions.

Since @Yicong-Huang designed the current PR template, can you provide what you expect in each section?

[Yicong-Huang]

The template I designed is for human use, before AI writing or descriptions. There is no limit on the template. Let me comment separately about how to solve AI verbosity

[Yicong-Huang]

While we welcome AI-assisted contributions, agents do tend to produce overly verbose issues, pull request descriptions, and comments. Here are my thinkings from a few perspectives:

• From a project-management perspective: A general hard limit on the length of descriptions or comments would not be ideal. These interfaces are used by both agents and humans, and humans sometimes have legitimate reasons to provide longer explanations.

• For agents: Contributors should instruct their agents to write concisely. We can reinforce this in two ways:

  1. State our expectations more firmly in AGENTS.md.
  2. Build and use agent tooling with built-in safeguards. I created a Texera/
     texera-agent-plugins (currently in a private repository) with open-issue and open-pr skills that include a verbosity gate to prevent unnecessarily long content. I am testing it with several recent issues:
     • Cache the license-eye build in the Check License Headers job #7136,
     • Document PR title conventions for feat vs fix, tests, dependency bumps, and backports #7393,
     • Remove two orphaned GUI config flags that no longer have consumers #7135
       and PRs:
     • chore(config): remove two orphaned GUI config flags with no consumers #7137
     • docs: define how to pick a PR title's type and scope #7427
     • ci: route merge-queue builds to self-hosted runner tiers with hosted fallback #7172

Please let me know what you think. The same verbosity gate will also apply to AI-assisted PR reviews performed using the Texera agent plugin.

• For humans: Although we should not impose strict length limits, encouraging concise writing would improve communication. We could add guidance to CONTRIBUTING.md recommending that issues, pull requests, and comments remain focused. Related guidance could discourage including unnecessary implementation details in issue descriptions. These should remain recommendations rather than hard requirements, which would also be difficult to enforce technically.

• For reviewers: When a description or comment is excessively long, reviewers should feel comfortable asking the author to revise it for clarity and conciseness.

[chenlica]

While I support the idea of using agents to improve the development efficiency, I am very concerned about a situation where "agents talk to agents, without caring about how humans think." I believe that is a dangerous trend, and we as humans may lose the control of the system when we lack a good understanding of the codebase.

Therefore, I prefer a practice where each author (of an issue and a PR) must write a clear, succinct, and human-readable summary of the description, possibly together with an agent-created text. Even if an agent can be instructed to produce such a summary, I doubt in the near future an agent can meet those requirements. So I prefer to let the author write the summary by themselves.

[Tomosu-AI]

AI makes it easy to generate more PRs and issues, but that also means there’s more for humans to review and understand.

The interesting part is keeping the useful context without turning every PR into a wall of AI generated text.

We’re working on a related problem around giving agents better production context, so it’s interesting to see how people are approaching this on the development side too. The short human summary + optional agent detail approach makes a lot of sense.