`State`ful artifact rendering: Verify it makes sense to implement (and implications), design, and implement

[balacij]

(TODO: @balacij will return to this ticket on Mon. July 22nd Tues. July 23rd, 2024 and write an actual description).

Unblocks (related 'problem' symptoms):

• Sort References by Order of Appearance in Document #806
• Determine Document Outline from Actual Document Contents #946
• Table of Acronyms includes unused acronyms #1235
• SRS boilerplate text should respect information 'filled in' #1415 (related to Determine Document Outline from Actual Document Contents #946)
• Printing mdBook SRS to different files / Generating multi-file software artifacts #3796

[balacij]

NOTE: I'm going to write a few different comments discussing different issues/symptoms.

I'll start with a code snippet:

Drasil/code/drasil-docLang/lib/Drasil/DocumentLanguage.hs

Lines 193 to 211 in 5705da6

	-- | Helper for creating the different document sections.
	mkSections :: SystemInformation -> DocDesc -> [Section]
	mkSections si dd = map doit dd
	where
	doit :: DocSection -> Section
	doit TableOfContents = mkToC dd
	doit (RefSec rs) = mkRefSec si dd rs
	doit (IntroSec is) = mkIntroSec si is
	doit (StkhldrSec sts) = mkStkhldrSec sts
	doit (SSDSec ss) = mkSSDSec si ss
	doit (AuxConstntSec acs) = mkAuxConsSec acs
	doit Bibliography = mkBib (citeDB si)
	doit (GSDSec gs') = mkGSDSec gs'
	doit (ReqrmntSec r) = mkReqrmntSec r
	doit (LCsSec lc) = mkLCsSec lc
	doit (UCsSec ulcs) = mkUCsSec ulcs
	doit (TraceabilitySec t) = mkTraceabilitySec t si
	doit (AppndxSec a) = mkAppndxSec a
	doit (OffShelfSolnsSec o) = mkOffShelfSolnSec o

It seems harmless, right? The answer should be "yes" because that's one of the key "transformation/render" points where we generate the SRS. The issue is that we abuse it:

λ ~/Programming/Drasil/code/ noInterMapDupe rg "mkSections" -ths
drasil-docLang/lib/Drasil/DocumentLanguage.hs
71:  mkSections fullSI l where
96:    allSections = concatMap findAllSec $ mkSections si $ mkDocDesc si dd -- FIXME: `mkSections` on something particularly large that is immediately discarded is a sign that we're doing something wrong. That's in addition to `mkDocDesc`...
128:    allSections = mkSections si $ mkDocDesc si dd

We have 3 references to it!

3 References to mkSection

(1) Generating the Abstract SRS Documents

Drasil/code/drasil-docLang/lib/Drasil/DocumentLanguage.hs

Lines 66 to 73 in 5705da6

	-- * Main Function
	-- | Creates a document from a document description, a title combinator function, and system information.
	mkDoc :: SRSDecl -> (IdeaDict -> IdeaDict -> Sentence) -> SystemInformation -> Document
	mkDoc dd comb si@SI {_sys = sys, _kind = kind, _authors = docauthors} =
	Document (nw kind `comb` nw sys) (foldlList Comma List $ map (S . name) docauthors) (findToC l) $
	mkSections fullSI l where
	fullSI = fillcdbSRS dd si
	l = mkDocDesc fullSI dd

(2) Finding all "Sections and LabelledContent" for inputting into the ChunkDB

Drasil/code/drasil-docLang/lib/Drasil/DocumentLanguage.hs

Lines 89 to 118 in 5705da6

	-- | Fill in the 'Section's and 'LabelledContent' maps of the 'ChunkDB' from the 'SRSDecl'.
	fillSecAndLC :: SRSDecl -> SystemInformation -> SystemInformation
	fillSecAndLC dd si = si2
	where
	-- Get current contents of si
	chkdb = si ^. sysinfodb
	-- extract sections and labelledcontent
	allSections = concatMap findAllSec $ mkSections si $ mkDocDesc si dd -- FIXME: `mkSections` on something particularly large that is immediately discarded is a sign that we're doing something wrong. That's in addition to `mkDocDesc`...
	allLC = concatMap findAllLC allSections
	existingLC = map (fst.snd) $ Map.assocs $ chkdb ^. labelledcontentTable
	-- fill in the appropriate chunkdb fields
	chkdb2 = set labelledcontentTable (idMap $ nub $ existingLC ++ allLC) chkdb
	-- return the filled in system information
	si2 = set sysinfodb chkdb2 si
	-- Helper and finder functions
	findAllSec :: Section -> [Section]
	findAllSec s@(Section _ cs _) = s : concatMap findAllSubSec cs
	findAllSubSec :: SecCons -> [Section]
	findAllSubSec (Sub s) = findAllSec s
	findAllSubSec _ = []
	findAllLC :: Section -> [LabelledContent]
	findAllLC (Section _ cs _) = concatMap findLCSecCons cs
	findLCSecCons :: SecCons -> [LabelledContent]
	findLCSecCons (Sub s) = findAllLC s
	findLCSecCons (Con (LlC lblcons@(LblC {_ctype = Figure {}}))) = [lblcons]
	findLCSecCons _ = []

(3) Finding all "References" for inputting into the ChunkDB

Drasil/code/drasil-docLang/lib/Drasil/DocumentLanguage.hs

Lines 120 to 149 in 5705da6

	-- | Takes in existing information from the Chunk database to construct a database of references.
	fillReferences :: SRSDecl -> SystemInformation -> SystemInformation
	fillReferences dd si@SI{_sys = sys} = si2
	where
	-- get old chunk database + ref database
	chkdb = si ^. sysinfodb
	cites = citeDB si
	-- convert SRSDecl into a list of sections (to easily get at all the references used in the SRS)
	allSections = mkSections si $ mkDocDesc si dd
	-- get refs from SRSDecl. Should include all section labels and labelled content.
	refsFromSRS = concatMap findAllRefs allSections
	-- get refs from the stuff already inside the chunk database
	ddefs = map fst $ Map.elems $ chkdb ^. dataDefnTable
	gdefs = map fst $ Map.elems $ chkdb ^. gendefTable
	imods = map fst $ Map.elems $ chkdb ^. insmodelTable
	tmods = map fst $ Map.elems $ chkdb ^. theoryModelTable
	concIns = map fst $ Map.elems $ chkdb ^. conceptinsTable
	lblCon = map fst $ Map.elems $ chkdb ^. labelledcontentTable
	-- search the old reference table just in case the user wants to manually add in some references
	refs = map fst $ Map.elems $ chkdb ^. refTable
	-- set new reference table in the chunk database
	chkdb2 = set refTable (idMap $ nub $ refsFromSRS
	++ map (ref . makeTabRef' . getTraceConfigUID) (traceMatStandard si)
	++ secRefs -- secRefs can be removed once #946 is complete
	++ traceyGraphGetRefs (programName sys) ++ map ref cites
	++ map ref ddefs ++ map ref gdefs ++ map ref imods
	++ map ref tmods ++ map ref concIns ++ map ref lblCon
	++ refs) chkdb
	-- set new chunk database into system information
	si2 = set sysinfodb chkdb2 si

Symptom A

Immediately relevant issues/PRs:

• Demote references table to 'not a chunk table' in ChunkDB #4022
• Demote 'LabelledContent' to 'not a chunk' until they're dealt with properly #4023

As I wrote in the second code snippet:

FIXME: `mkSections` on something particularly large that is immediately discarded is a sign that we're doing something wrong. That's in addition to `mkDocDesc`...

We are effectively generating an abstract copy of the entire SRS document 3 times in the code, with all 3 used in code generation, in different ways. (2) and (3) are 'assistive' for (1) -- they search for references to all Sections, LabelledContent, and References and insert them into the ChunkDB. When (1) is finally rendered into a concrete SRS artifact (i.e., HTML, Jupyter, mdBook, $\LaTeX$), UID references are all resolved, which is why (2) and (3) are helpful -- they automate insertion of the References and LabelledContent, and whatever else, into the ChunkDB for (1) to resolve content for.

[balacij]

Symptom (B)

Now let's look at: #1235 and #1661. This issue is also related to why we have usedinfodb (also discussed in #3260 and more in #1661 (comment)) -- a hack solution that requires manual collection of the terms with acronyms presented in the generated SRS documents.

How could we fix this today? Similar to the issues related to Symptom (A), we could generate the entire abstract document with mkSections and figure out which acronyms are actually referenced.

The issue with this is that it would exacerbate Symptom (A) and potentially computational waste if we do it in the same way as the earlier referenced code.

[balacij]

Symptom (C)

Now let's look at: #806. (Actually, in this ticket, I jump to the solution presented in the title, but I forgot about that until now....)

We want our generated bibliography to be ordered according to order of presentation. Again, if we wanted to fix this today, we could follow through with the same way with mkSections.

#1415 and #946 are very similar. The way to fix them today is to scan our mkSections and use that information as necessary.

[balacij]

Problem

The real problem is that the content of our documents has scopes, and said content wants to be aware of its scope. For example, the bibliography is "top-level/meta"-content/information that wants to be aware of everything else. The document outline and global tables of $X$ are similar.

Local tables of $X$ are also similar, but within smaller scopes. For example, when rendering our IM/TM/GD/DD boxes, when we present the list of variables, they want to be aware of the variables in scope of the equations presented within the box and to present only the ones immediately used in said box.

Our current strategy to dealing with this relies on some hacks:

1. Scanning mkSections and discarding results.
2. Conflating display knowledge with math knowledge and assuming that it will be displayed ->

   Drasil/code/drasil-theory/lib/Theory/Drasil/DataDefinition.hs

   Lines 101 to 104 in 5705da6

   	-- | Finds the reference address of a 'DataDefinition where'.
   	instance Referable DataDefinition where
   	refAdd = (^. ddPkt pktS)
   	renderRef l = RP (prepend $ abrv l) (refAdd l)

[balacij]

My head's starting to spin with this issue because of its sheer scale. There are a bunch of issues that are coming down to the same problem, and now I'm having a hard time analyzing everything together, coming up with a solution, and verbalizing it coherently.

There's also #3796 (reply in thread), which is something of a "Symptom (D)" and where I built a prototype solution that we could extend.

I'm spending too much time on this, so I'm going to stop myself. @smiths @JacquesCarette if either of you have any thoughts on this, that would be appreciated, otherwise, I will put this on the backburner for a little bit and gather my thoughts slowly.

[JacquesCarette]

The real problem is that the content of our documents has scopes, and said content wants to be aware of its scope.

I think this is a valuable insight. Luckily, phrasing it that way also helps, because it also leads to standard solutions from PL.

We need to rethink

1. What the 'assembly' process for a document should be
2. What we think 'document pieces' should be

Maybe we should strongly leverage programming language ideas: hierarchical name spaces, context polymorphism, static and dynamic scopes (and scope resolution), etc.

We could also seek inspiration from XML/XSLT (and HTML/CSS). And maybe from LaTeX and/or typst and/or org-roam.

In other words: what is a good programming language for

1. declaring re-usable parts of documents
2. assembling whole documents from parts
   Note that I'm not assuming that this is the same language. I am assuming that both as DSLs.