Better lineno in needs JSON - #1692
Open
sebastiansetzer wants to merge 4 commits into
Open
Conversation
This function converts the "virtual" lineno into a real line number as it appears in the source file, even in the presence of code injected with state_machine.insert_input. The injected code has line numbers starting from 1 again, and the returned "source" name is the name of an include file, if that's where the injected code is from. The list2need test is extended by one normal need, following the needs created with list2needs. Previously, this test would have failed because list2needs uses insert_input, messing up line numbers for everything following later.
It is not used anymore, and the next commit will turn the commented out line unusable because you need to iterate over each line to get line numbers.
sebastiansetzer
force-pushed
the
lineno_experiments
branch
5 times, most recently
from
August 7, 2026 21:23
5d7a6b7 to
6b24b65
Compare
The line numbers are known in the list2need directive (A), but they are output into needs in class NeedDirective later (B), when the test inserted with state_machine.insert_input has been parsed. Somehow they must be passed from A to B. This is solved by encoding a mapping of wrong to correct line numbers in the path string which gets passed from A to B already. You can't simply put it into the lineno attribute, because there is only one lineno attribute for all needs in the whole list2needs directive, and that gets mangled by state_machine.insert_input. You could maybe pass the information through a new global variable, but I don't know how well that would play with parallel build. Are global variables written at A available at B in parallel builds? You can't split up the state_machine.insert_input into multiple ones, one for each generated need, with the lineno tweaked to give the right output, because then the parent-child hierarchy between the generated needs gets lost - each state_machine.insert_input starts from the same indentation level. The cleanest solution maybe would be to get rid of state_machine.insert_input alltogether and generate the needs directly. But that would be a bigger change that I can't judge the implications for.
sebastiansetzer
force-pushed
the
lineno_experiments
branch
from
August 7, 2026 21:24
596a48d to
ae0b9a8
Compare
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This should fix #1349.
The fix for the
includeandrst-prologuse cases is simple.I just wonder if anyone relies on having unique line numbers for one "docname" and docname being a full rst document, not an include file. If that's the case, maybe instead of fixing the
linenoattribute, we should add a new attribute (maybe call it "source position"?)On the other hand, the only reason I can think of why you would not want include file names as "source" attribute is if you had one that you included in multiple places. And if you do that, you won't have needs in there, because those would then get duplicated.
The fix for the
list2needuse case is abusing the "source" string to pass line number information from the list2needs directive to the generated needs directives.I considered passing this information through a generated "option", but when I declare this in
needs_extra_options, it shows up for all needs (as "None" when unused) in the JSON, so that's a bit ugly.I guess the best solution would be to refactor the
list2needdirective to create the needs objects directly, instead of generating rst text which is then injected back into the parser. This would avoid the need to pass line numbers, and it would even avoid callingstate_machine.insert_input, so it wouldn't corrupt line numbers for subsequent needs.Of course, corrupted line numbers of subsequent needs are already fixed by using
state_machine.get_source_and_line, and you'd want to keep this for theincludeandrst-prologuse cases, so the important reason for the refactoring is just avoiding to encode line numbers in the source string.Before attempting to do this refactoring, I would like to understand why the current solution with
state_machine.insert_inputwas chosen. Directly creating the needs objects seems more obvious, so I wonder if I'm missing the reason for Chesterton's fence here.My test setup is attached - not integrated into the sphinx-needs test suite so far.
rst-lineno-bug-tests.zip