Skip to content

Commit

Permalink
#378: Added support for underlined headlines in MarkdownImporter.
Browse files Browse the repository at this point in the history
  • Loading branch information
@redcatbear
redcatbear committed Feb 14, 2024
1 parent b0aa38d commit a92ebc7
Show file tree
Hide file tree
Showing 8 changed files with 169 additions and 89 deletions.
7 changes: 4 additions & 3 deletions api/src/main/java/org/itsallcode/openfasttrace/api/importer/SpecificationListBuilder.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -185,13 +185,14 @@ public void endSpecificationItem()
resetState();
}

// [impl->dsn~cleaning-imported-multi-line-text-elements~1]
private SpecificationItem createNewSpecificationItem()
{
return this.itemBuilder //
.id(this.id) //
.description(this.description.toString()) //
.rationale(this.rationale.toString()) //
.comment(this.comment.toString()) //
.description(this.description.toString().trim()) //
.rationale(this.rationale.toString().trim()) //
.comment(this.comment.toString().trim()) //
.location(this.location) //
.build();
}
Expand Down
22 changes: 21 additions & 1 deletion ...src/test/java/org/itsallcode/openfasttrace/api/importer/TestSpecificationListBuilder.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.containsInAnyOrder;
import static org.hamcrest.Matchers.equalTo;
import static org.junit.jupiter.api.Assertions.assertAll;

import java.util.*;
import java.util.stream.Collectors;
Expand All @@ -13,7 +14,6 @@

class TestSpecificationListBuilder
{

private static final String DESCRIPTION = "description";
private static final String TITLE = "title";
private final static SpecificationItemId ID = SpecificationItemId.parseId("feat~id~1");
Expand Down Expand Up @@ -205,4 +205,24 @@ void testFilterSpecificationItemsByTagsIncludingNoTags()
assertThat(items.stream().map(SpecificationItem::getName).collect(Collectors.toList()),
containsInAnyOrder("in-A", "in-B", "in-D"));
}

// [utest->dsn~cleaning-imported-multi-line-text-elements~1]
@Test
void testMultilineTextFieldsGetTrimmed()
{
final SpecificationListBuilder builder = SpecificationListBuilder.create();
builder.beginSpecificationItem();
builder.setId(SpecificationItemId.createId("foo", "bar", 1));
builder.appendComment(" a comment ");
builder.appendDescription(" a description\t \t");
builder.appendRationale("\n\na rationale\n \n");
builder.endSpecificationItem();
final List<SpecificationItem> items = builder.build();
final SpecificationItem item = items.get(0);
assertAll(
() -> assertThat(item.getComment(), equalTo("a comment")),
() -> assertThat(item.getDescription(), equalTo("a description")),
() -> assertThat(item.getRationale(), equalTo("a rationale"))
);
}
}
21 changes: 21 additions & 0 deletions doc/spec/design.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -950,6 +950,27 @@ Exchanging the CLI later takes considerable effort.
* No CLI (plain argument list) - not flexible enough
* External CLI - breaks design goal

## How do we Clean Imported Multi-line Text Elements
`dsn~cleaning-imported-multi-line-text-elements~1`

It is the responsibility of `ImportEventListener`s do the following clean-up steps in multi-line text elements like the description:

* Trimming (removing leading and trailing spaces of the whole text -- no individual lines.)

Rationale:

The reason for this is that we then do it in a central place and don't produce code duplication. Extra clean-up in the importers is still possible, but basics like trimming need to be handled in common code.

Needs: impl, utest

### Why is This Architecture Relevant?

Authors of importers need to be able to relly on these cleanups being done centrally, so that they don't have to implement them themselves.

### Alternatives Considered

Clean-up in every importer individually. That was the case up to and including OFT 3.7.1 which turned out to make the importer more complex than necessary.

# Bibliography

The following documents or are referenced in this specification.
Expand Down
20 changes: 18 additions & 2 deletions doc/spec/system_requirements.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ OFT imports specification items from Markdown files.

Rationale:

Markdown is a clean an simple format that:
Markdown is a clean a simple format that:

* is viewable with any text editor
* focuses on content instead of layout
Expand All @@ -82,6 +82,22 @@ maintained over a long time.

Needs: req

### ReStructured Text (RST) Import
`feat~rst-import~1`

OFT imports specification items from ReStructured Text (RST) files.

Rationale:

RST is a text-based documentation format with non-invasive structure elements. It originated in the Python world and has become the standard for documentation there.

The same benefits as for [Markdown](#markdown-import) apply:

* is viewable with any text editor
* focuses on content instead of layout
* is portable across platforms
* easy to process with text manipulation tools

### ReqM2 Import
`feat~reqm2-import~1`

Expand Down Expand Up @@ -241,7 +257,7 @@ Needs: dsn
##### Markdown Outline Readable
`req~markdown-outline-readable~1`

The Markdown outline -- a table of contents created from the heading structure by various Markdown editors -- must be human readable.
The Markdown outline -- a table of contents created from the heading structure by various Markdown editors -- must be human-readable.

Rationale:

Expand Down
103 changes: 41 additions & 62 deletions ...rkdown/src/main/java/org/itsallcode/openfasttrace/importer/markdown/MarkdownImporter.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@ class MarkdownImporter implements Importer
private static final Logger LOG = Logger.getLogger(MarkdownImporter.class.getName());

// @formatter:off
private final Transition[] transitions = {
private final Transition[] transitions = {
transition(START , SPEC_ITEM , MdPattern.ID , this::beginItem ),
transition(START , TITLE , MdPattern.TITLE , this::rememberTitle ),
transition(START , OUTSIDE , MdPattern.FORWARD , this::forward ),
transition(START , OUTSIDE , MdPattern.EVERYTHING , () -> {} ),

transition(TITLE , SPEC_ITEM , MdPattern.ID , this::beginItem ),
transition(TITLE , TITLE , MdPattern.TITLE , this::rememberTitle ),
transition(TITLE , TITLE , MdPattern.EMPTY , () -> {} ),
Expand All @@ -30,7 +30,8 @@ class MarkdownImporter implements Importer
transition(OUTSIDE , SPEC_ITEM , MdPattern.ID , this::beginItem ),
transition(OUTSIDE , OUTSIDE , MdPattern.FORWARD , this::forward ),
transition(OUTSIDE , TITLE , MdPattern.TITLE , this::rememberTitle ),

transition(OUTSIDE , TITLE , MdPattern.UNDERLINE , this::rememberPreviousLineAsTitle ),

transition(SPEC_ITEM , SPEC_ITEM , MdPattern.ID , this::beginItem ),
transition(SPEC_ITEM , SPEC_ITEM , MdPattern.STATUS , this::setStatus ),
transition(SPEC_ITEM , TITLE , MdPattern.TITLE , () -> {endItem(); rememberTitle();} ),
Expand All @@ -45,37 +46,37 @@ class MarkdownImporter implements Importer
transition(SPEC_ITEM , DESCRIPTION, MdPattern.DESCRIPTION, this::beginDescription ),
transition(SPEC_ITEM , DESCRIPTION, MdPattern.NOT_EMPTY , this::beginDescription ),

transition(DESCRIPTION, SPEC_ITEM , MdPattern.ID , () -> {endDescription(); beginItem();} ),
transition(DESCRIPTION, TITLE , MdPattern.TITLE , () -> {endDescription(); endItem(); rememberTitle(); }),
transition(DESCRIPTION, RATIONALE , MdPattern.RATIONALE , () -> {endDescription(); beginRationale();} ),
transition(DESCRIPTION, COMMENT , MdPattern.COMMENT , () -> {endDescription(); beginComment();} ),
transition(DESCRIPTION, COVERS , MdPattern.COVERS , this::endDescription ),
transition(DESCRIPTION, DEPENDS , MdPattern.DEPENDS , this::endDescription ),
transition(DESCRIPTION, NEEDS , MdPattern.NEEDS_INT , () -> {endDescription(); addNeeds();} ),
transition(DESCRIPTION, NEEDS , MdPattern.NEEDS , this::endDescription ),
transition(DESCRIPTION, SPEC_ITEM , MdPattern.ID , this::beginItem ),
transition(DESCRIPTION, TITLE , MdPattern.TITLE , () -> {endItem(); rememberTitle();} ),
transition(DESCRIPTION, RATIONALE , MdPattern.RATIONALE , this::beginRationale ),
transition(DESCRIPTION, COMMENT , MdPattern.COMMENT , this::beginComment ),
transition(DESCRIPTION, COVERS , MdPattern.COVERS , () -> {} ),
transition(DESCRIPTION, DEPENDS , MdPattern.DEPENDS , () -> {} ),
transition(DESCRIPTION, NEEDS , MdPattern.NEEDS_INT , this::addNeeds ),
transition(DESCRIPTION, NEEDS , MdPattern.NEEDS , () -> {} ),
transition(DESCRIPTION, TAGS , MdPattern.TAGS_INT , this::addTag ),
transition(DESCRIPTION, TAGS , MdPattern.TAGS , () -> {} ),
transition(DESCRIPTION, DESCRIPTION, MdPattern.EVERYTHING , this::appendDescription ),


transition(RATIONALE , SPEC_ITEM , MdPattern.ID , () -> {endRationale(); beginItem();} ),
transition(RATIONALE , TITLE , MdPattern.TITLE , () -> {endRationale(); endItem(); rememberTitle(); } ),
transition(RATIONALE , COMMENT , MdPattern.COMMENT , () -> {endRationale(); beginComment();} ),
transition(RATIONALE , COVERS , MdPattern.COVERS , this::endRationale ),
transition(RATIONALE , DEPENDS , MdPattern.DEPENDS , this::endRationale ),
transition(RATIONALE , NEEDS , MdPattern.NEEDS_INT , () -> {endRationale(); addNeeds();} ),
transition(RATIONALE , NEEDS , MdPattern.NEEDS , this::endRationale ),
transition(RATIONALE , SPEC_ITEM , MdPattern.ID , this::beginItem ),
transition(RATIONALE , TITLE , MdPattern.TITLE , () -> {endItem(); rememberTitle();} ),
transition(RATIONALE , COMMENT , MdPattern.COMMENT , this::beginComment ),
transition(RATIONALE , COVERS , MdPattern.COVERS , () -> {} ),
transition(RATIONALE , DEPENDS , MdPattern.DEPENDS , () -> {} ),
transition(RATIONALE , NEEDS , MdPattern.NEEDS_INT , this::addNeeds ),
transition(RATIONALE , NEEDS , MdPattern.NEEDS , () -> {} ),
transition(RATIONALE , TAGS , MdPattern.TAGS_INT , this::addTag ),
transition(RATIONALE , TAGS , MdPattern.TAGS , () -> {} ),
transition(RATIONALE , RATIONALE , MdPattern.EVERYTHING , this::appendRationale ),

transition(COMMENT , SPEC_ITEM , MdPattern.ID , () -> {endComment(); beginItem();} ),
transition(COMMENT , TITLE , MdPattern.TITLE , () -> {endComment(); endItem(); rememberTitle(); } ),
transition(COMMENT , COVERS , MdPattern.COVERS , this::endComment ),
transition(COMMENT , DEPENDS , MdPattern.DEPENDS , this::endComment ),
transition(COMMENT , NEEDS , MdPattern.NEEDS_INT , () -> {endComment(); addNeeds();} ),
transition(COMMENT , NEEDS , MdPattern.NEEDS , this::endComment ),
transition(COMMENT , RATIONALE , MdPattern.RATIONALE , () -> {endComment(); beginRationale();} ),
transition(COMMENT , SPEC_ITEM , MdPattern.ID , this::beginItem ),
transition(COMMENT , TITLE , MdPattern.TITLE , () -> {endItem(); rememberTitle();} ),
transition(COMMENT , COVERS , MdPattern.COVERS , () -> {} ),
transition(COMMENT , DEPENDS , MdPattern.DEPENDS , () -> {} ),
transition(COMMENT , NEEDS , MdPattern.NEEDS_INT , this::addNeeds ),
transition(COMMENT , NEEDS , MdPattern.NEEDS , () -> {} ),
transition(COMMENT , RATIONALE , MdPattern.RATIONALE , this::beginRationale ),
transition(COMMENT , TAGS , MdPattern.TAGS_INT , this::addTag ),
transition(COMMENT , TAGS , MdPattern.TAGS , () -> {} ),
transition(COMMENT , COMMENT , MdPattern.EVERYTHING , this::appendComment ),
Expand Down Expand Up @@ -141,10 +142,8 @@ class MarkdownImporter implements Importer
private final ImportEventListener listener;
private final MarkdownImporterStateMachine stateMachine;
private String lastTitle = null;
private String lastLine = null;
private boolean inSpecificationItem;
private StringBuilder lastDescription;
private StringBuilder lastRationale;
private StringBuilder lastComment;
private int lineNumber = 0;

MarkdownImporter(final InputFile fileName, final ImportEventListener listener)
Expand All @@ -166,6 +165,7 @@ public void runImport()
{
++this.lineNumber;
this.stateMachine.step(line);
this.lastLine = line;
}
}
catch (final IOException exception)
Expand All @@ -186,7 +186,7 @@ private void finishImport()
}
}

private static final Transition transition(final State from, final State to,
private static Transition transition(final State from, final State to,
final MdPattern pattern, final TransitionAction action)
{
return new Transition(from, to, pattern, action);
Expand Down Expand Up @@ -234,59 +234,35 @@ private void setStatus()

private void beginDescription()
{
this.lastDescription = new StringBuilder(this.stateMachine.getLastToken());
this.listener.appendDescription(this.stateMachine.getLastToken());
}

private void appendDescription()
{
this.lastDescription.append(System.lineSeparator())
.append(this.stateMachine.getLastToken());
}

private void endDescription()
{
this.listener.appendDescription(this.lastDescription.toString().trim());
this.lastDescription = null;
this.listener.appendDescription(System.lineSeparator());
this.listener.appendDescription(this.stateMachine.getLastToken());
}

private void beginRationale()
{
this.lastRationale = new StringBuilder();
this.listener.appendRationale(System.lineSeparator());
}

private void appendRationale()
{
if (this.lastRationale.length() > 0)
{
this.lastRationale.append(System.lineSeparator());
}
this.lastRationale.append(this.stateMachine.getLastToken());
}

private void endRationale()
{
this.listener.appendRationale(this.lastRationale.toString().trim());
this.lastRationale = null;
this.listener.appendRationale(System.lineSeparator());
this.listener.appendRationale(this.stateMachine.getLastToken());
}

private void beginComment()
{
this.lastComment = new StringBuilder();
this.listener.appendComment(this.stateMachine.getLastToken());
}

private void appendComment()
{
if (this.lastComment.length() > 0)
{
this.lastComment.append(System.lineSeparator());
}
this.lastComment.append(this.stateMachine.getLastToken());
}

private void endComment()
{
this.listener.appendComment(this.lastComment.toString().trim());
this.lastComment = null;
this.listener.appendComment(System.lineSeparator());
this.listener.appendComment(this.stateMachine.getLastToken());
}

private void addDependency()
Expand All @@ -311,6 +287,9 @@ private void rememberTitle()
this.lastTitle = this.stateMachine.getLastToken();
}


private void rememberPreviousLineAsTitle() { this.lastTitle = this.lastLine; }

private void resetTitle()
{
this.lastTitle = null;
Expand Down
4 changes: 3 additions & 1 deletion ...rter/markdown/src/main/java/org/itsallcode/openfasttrace/importer/markdown/MdPattern.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,14 +44,16 @@ enum MdPattern
+ "(\\p{Alpha}+)" //
+ "(?:\\W.*)?"),
NOT_EMPTY("([^\n\r]+)"),
POT_TITLE("(\\p{Alnum}.*)"),
RATIONALE("Rationale:\\s*"),
STATUS("Status:\\s*(approved|proposed|draft)\\s*"),
TAGS_INT("Tags:(\\s*\\w+\\s*(?:,\\s*\\w+\\s*)*)"),
TAGS("Tags:\\s*"),
TAG_ENTRY(PatternConstants.UP_TO_3_WHITESPACES + PatternConstants.BULLETS
+ "\\s*" //
+ "(.*)"),
TITLE("#+\\s*(.*)");
TITLE("#+\\s*(.*)"),
UNDERLINE("([=-]{3,})");
// @formatter:on

private final Pattern pattern;
Expand Down
Loading

0 comments on commit a92ebc7

Please sign in to comment.