Types declared in non-top-level scope gensym their symbols to workaround bad mangling

[metagn]

Description

macro foo(x: typed) =
  result = x
  echo result.repr

block: # or `proc main =`
  foo:
    type Foo = object
  echo Foo()

Nim Version

2.0.0

Current Output

type
  Foo_520093702 = object
(7, 8) Error: undeclared identifier: 'Foo'

Expected Output

type
  Foo = object
()

Possible Solution

Read comments below (#22571 (comment) and #22571 (comment)) for description of why this happens. The issue is caused by the compiler intentionally gensymming these types to workaround a codegen issue that should have been solved in codegen.

Additional Information

Works if the macro is called at top level, or with declarations that are not types like proc or const

[juancarlospaco]

!nim c

macro foo(x: typed) =
  result = x
  echo result.repr

block:
  foo:
    type Foo = object
  echo Foo()

[github-actions]

@juancarlospaco (collaborator)

devel 👎 FAIL

Output

Error: Command failed: nim c --run  -d:nimDebugDlOpen -d:ssl -d:nimDisableCertificateValidation --forceBuild:on --colors:off --verbosity:0 --hints:off --warnings:off --lineTrace:off --nimcache:/home/runner/work/Nim/Nim --out:/home/runner/work/Nim/Nim/temp /home/runner/work/Nim/Nim/temp.nim
/home/runner/work/Nim/Nim/temp.nim(7, 8) Error: undeclared identifier: 'Foo'

Stats

• Started 2023-08-28T11:53:42
• Finished 2023-08-28T11:53:43
• Duration

IR

Compiled filesize 0 bytes (0 bytes)

AST

nnkStmtList.newTree(
  nnkMacroDef.newTree(
    newIdentNode("foo"),
    newEmptyNode(),
    newEmptyNode(),
    nnkFormalParams.newTree(
      newEmptyNode(),
      nnkIdentDefs.newTree(
        newIdentNode("x"),
        newIdentNode("typed"),
        newEmptyNode()
      )
    ),
    newEmptyNode(),
    newEmptyNode(),
    nnkStmtList.newTree(
      nnkAsgn.newTree(
        newIdentNode("result"),
        newIdentNode("x")
      ),
      nnkCommand.newTree(
        newIdentNode("echo"),
        nnkDotExpr.newTree(
          newIdentNode("result"),
          newIdentNode("repr")
        )
      )
    )
  ),
  nnkBlockStmt.newTree(
    newEmptyNode(),
    nnkStmtList.newTree(
      nnkCall.newTree(
        newIdentNode("foo"),
        nnkStmtList.newTree(
          nnkTypeSection.newTree(
            nnkTypeDef.newTree(
              newIdentNode("Foo"),
              newEmptyNode(),
              nnkObjectTy.newTree(
                newEmptyNode(),
                newEmptyNode(),
                newEmptyNode()
              )
            )
          )
        )
      ),
      nnkCommand.newTree(
        newIdentNode("echo"),
        nnkCall.newTree(
          newIdentNode("Foo")
        )
      )
    )
  )
)

stable 👎 FAIL

Output

Error: Command failed: nim c --run  -d:nimDebugDlOpen -d:ssl -d:nimDisableCertificateValidation --forceBuild:on --colors:off --verbosity:0 --hints:off --warnings:off --lineTrace:off --nimcache:/home/runner/work/Nim/Nim --out:/home/runner/work/Nim/Nim/temp /home/runner/work/Nim/Nim/temp.nim
/home/runner/work/Nim/Nim/temp.nim(7, 8) Error: undeclared identifier: 'Foo'

Stats

• Started 2023-08-28T11:53:44
• Finished 2023-08-28T11:53:44
• Duration

IR

Compiled filesize 0 bytes (0 bytes)

AST

nnkStmtList.newTree(
  nnkMacroDef.newTree(
    newIdentNode("foo"),
    newEmptyNode(),
    newEmptyNode(),
    nnkFormalParams.newTree(
      newEmptyNode(),
      nnkIdentDefs.newTree(
        newIdentNode("x"),
        newIdentNode("typed"),
        newEmptyNode()
      )
    ),
    newEmptyNode(),
    newEmptyNode(),
    nnkStmtList.newTree(
      nnkAsgn.newTree(
        newIdentNode("result"),
        newIdentNode("x")
      ),
      nnkCommand.newTree(
        newIdentNode("echo"),
        nnkDotExpr.newTree(
          newIdentNode("result"),
          newIdentNode("repr")
        )
      )
    )
  ),
  nnkBlockStmt.newTree(
    newEmptyNode(),
    nnkStmtList.newTree(
      nnkCall.newTree(
        newIdentNode("foo"),
        nnkStmtList.newTree(
          nnkTypeSection.newTree(
            nnkTypeDef.newTree(
              newIdentNode("Foo"),
              newEmptyNode(),
              nnkObjectTy.newTree(
                newEmptyNode(),
                newEmptyNode(),
                newEmptyNode()
              )
            )
          )
        )
      ),
      nnkCommand.newTree(
        newIdentNode("echo"),
        nnkCall.newTree(
          newIdentNode("Foo")
        )
      )
    )
  )
)

2.0.0 👎 FAIL

Output

Error: Command failed: nim c --run  -d:nimDebugDlOpen -d:ssl -d:nimDisableCertificateValidation --forceBuild:on --colors:off --verbosity:0 --hints:off --warnings:off --lineTrace:off --nimcache:/home/runner/work/Nim/Nim --out:/home/runner/work/Nim/Nim/temp /home/runner/work/Nim/Nim/temp.nim
/home/runner/work/Nim/Nim/temp.nim(7, 8) Error: undeclared identifier: 'Foo'

Stats

• Started 2023-08-28T11:53:44
• Finished 2023-08-28T11:53:45
• Duration

IR

Compiled filesize 0 bytes (0 bytes)

AST

nnkStmtList.newTree(
  nnkMacroDef.newTree(
    newIdentNode("foo"),
    newEmptyNode(),
    newEmptyNode(),
    nnkFormalParams.newTree(
      newEmptyNode(),
      nnkIdentDefs.newTree(
        newIdentNode("x"),
        newIdentNode("typed"),
        newEmptyNode()
      )
    ),
    newEmptyNode(),
    newEmptyNode(),
    nnkStmtList.newTree(
      nnkAsgn.newTree(
        newIdentNode("result"),
        newIdentNode("x")
      ),
      nnkCommand.newTree(
        newIdentNode("echo"),
        nnkDotExpr.newTree(
          newIdentNode("result"),
          newIdentNode("repr")
        )
      )
    )
  ),
  nnkBlockStmt.newTree(
    newEmptyNode(),
    nnkStmtList.newTree(
      nnkCall.newTree(
        newIdentNode("foo"),
        nnkStmtList.newTree(
          nnkTypeSection.newTree(
            nnkTypeDef.newTree(
              newIdentNode("Foo"),
              newEmptyNode(),
              nnkObjectTy.newTree(
                newEmptyNode(),
                newEmptyNode(),
                newEmptyNode()
              )
            )
          )
        )
      ),
      nnkCommand.newTree(
        newIdentNode("echo"),
        nnkCall.newTree(
          newIdentNode("Foo")
        )
      )
    )
  )
)

1.6.0 👎 FAIL

Output

Error: Command failed: nim c --run  -d:nimDebugDlOpen -d:ssl -d:nimDisableCertificateValidation --forceBuild:on --colors:off --verbosity:0 --hints:off --warnings:off --lineTrace:off --nimcache:/home/runner/work/Nim/Nim --out:/home/runner/work/Nim/Nim/temp /home/runner/work/Nim/Nim/temp.nim
/home/runner/work/Nim/Nim/temp.nim(7, 8) Error: undeclared identifier: 'Foo'
candidates (edit distance, scope distance); see '--spellSuggest': 
 (1, 2): 'foo' [macro declared in /home/runner/work/Nim/Nim/temp.nim(1, 7)]

Stats

• Started 2023-08-28T11:53:48
• Finished 2023-08-28T11:53:48
• Duration

IR

Compiled filesize 0 bytes (0 bytes)

AST

nnkStmtList.newTree(
  nnkMacroDef.newTree(
    newIdentNode("foo"),
    newEmptyNode(),
    newEmptyNode(),
    nnkFormalParams.newTree(
      newEmptyNode(),
      nnkIdentDefs.newTree(
        newIdentNode("x"),
        newIdentNode("typed"),
        newEmptyNode()
      )
    ),
    newEmptyNode(),
    newEmptyNode(),
    nnkStmtList.newTree(
      nnkAsgn.newTree(
        newIdentNode("result"),
        newIdentNode("x")
      ),
      nnkCommand.newTree(
        newIdentNode("echo"),
        nnkDotExpr.newTree(
          newIdentNode("result"),
          newIdentNode("repr")
        )
      )
    )
  ),
  nnkBlockStmt.newTree(
    newEmptyNode(),
    nnkStmtList.newTree(
      nnkCall.newTree(
        newIdentNode("foo"),
        nnkStmtList.newTree(
          nnkTypeSection.newTree(
            nnkTypeDef.newTree(
              newIdentNode("Foo"),
              newEmptyNode(),
              nnkObjectTy.newTree(
                newEmptyNode(),
                newEmptyNode(),
                newEmptyNode()
              )
            )
          )
        )
      ),
      nnkCommand.newTree(
        newIdentNode("echo"),
        nnkCall.newTree(
          newIdentNode("Foo")
        )
      )
    )
  )
)

1.4.0 👎 FAIL

Output

Error: Command failed: nim c --run  -d:nimDebugDlOpen -d:ssl -d:nimDisableCertificateValidation --forceBuild:on --colors:off --verbosity:0 --hints:off --warnings:off --lineTrace:off --nimcache:/home/runner/work/Nim/Nim --out:/home/runner/work/Nim/Nim/temp /home/runner/work/Nim/Nim/temp.nim
/home/runner/work/Nim/Nim/temp.nim(7, 8) Error: undeclared identifier: 'Foo'

Stats

• Started 2023-08-28T11:53:52
• Finished 2023-08-28T11:53:52
• Duration

IR

Compiled filesize 0 bytes (0 bytes)

AST

nnkStmtList.newTree(
  nnkMacroDef.newTree(
    newIdentNode("foo"),
    newEmptyNode(),
    newEmptyNode(),
    nnkFormalParams.newTree(
      newEmptyNode(),
      nnkIdentDefs.newTree(
        newIdentNode("x"),
        newIdentNode("typed"),
        newEmptyNode()
      )
    ),
    newEmptyNode(),
    newEmptyNode(),
    nnkStmtList.newTree(
      nnkAsgn.newTree(
        newIdentNode("result"),
        newIdentNode("x")
      ),
      nnkCommand.newTree(
        newIdentNode("echo"),
        nnkDotExpr.newTree(
          newIdentNode("result"),
          newIdentNode("repr")
        )
      )
    )
  ),
  nnkBlockStmt.newTree(
    newEmptyNode(),
    nnkStmtList.newTree(
      nnkCall.newTree(
        newIdentNode("foo"),
        nnkStmtList.newTree(
          nnkTypeSection.newTree(
            nnkTypeDef.newTree(
              newIdentNode("Foo"),
              newEmptyNode(),
              nnkObjectTy.newTree(
                newEmptyNode(),
                newEmptyNode(),
                newEmptyNode()
              )
            )
          )
        )
      ),
      nnkCommand.newTree(
        newIdentNode("echo"),
        nnkCall.newTree(
          newIdentNode("Foo")
        )
      )
    )
  )
)

1.2.0 👍 $\color{green}\textbf{\large OK}$

Output

type
  Foo_3387030 = object
()

Stats

• Started 2023-08-28T11:54:08
• Finished 2023-08-28T11:54:09
• Duration

1.0.0 👍 $\color{green}\textbf{\large OK}$

Output

type
  Foo = object
()

Stats

• Started 2023-08-28T11:54:22
• Finished 2023-08-28T11:54:22
• Duration

0.20.2 👍 $\color{green}\textbf{\large OK}$

Output

type
  Foo = object
()

Stats

• Started 2023-08-28T11:54:32
• Finished 2023-08-28T11:54:33
• Duration

#fb58066b6 ➡️ 🐛

Diagnostics

clyybber introduced a bug at 2020-08-27 15:50:59 +0200 on commit #fb58066b6 with message:

Fix #5691 (#15158)

* Fix #5691
* Cleanup and thoughts
* Use scope approach
* Seperate defined/declared/declaredInScope magics
* Fix declaredInScope
* Update spec accordingly

The bug is in the files:

compiler/ast.nim
compiler/condsyms.nim
compiler/lookups.nim
compiler/sem.nim
compiler/semcall.nim
compiler/semexprs.nim
compiler/semgnrc.nim
compiler/semstmts.nim
compiler/sigmatch.nim
doc/manual.rst
lib/system.nim
tests/macros/tmacros_various.nim
tests/macros/tmacrotypes.nim
tests/template/tparams_gensymed.nim
tests/template/tparamscope.nim

The bug can be in the commits:

• #13e659cfe
• #f8c48fc18
• #fb58066b6
• #d11933ad9
• #ccccd30cf
• #a9a9860b0
• #d85015e92
• #40969296e
• #71025b8a8

(Diagnostics sometimes off-by-one).

Stats

• GCC 11.4.0
• LibC 2.35
• Valgrind 3.18.1
• NodeJS 17.1
• Linux 5.15.0
• Created 2023-08-28T11:52:47Z
• Comments 1
• Commands nim c --run -d:nimDebugDlOpen -d:ssl -d:nimDisableCertificateValidation --forceBuild:on --colors:off --verbosity:0 --hints:off --warnings:off --lineTrace:off --nimcache:/home/runner/work/Nim/Nim --out:/home/runner/work/Nim/Nim/temp /home/runner/work/Nim/Nim/temp.nim

🤖 Bug found in 12 minutes bisecting 4539 commits at 363 commits per second.

[metagn]

Culprit (840e13d and 83e715c):

Nim/compiler/semstmts.nim

Lines 1589 to 1591 in 2e4e2f8

	# fix bug #5170, bug #17162, bug #15526: ensure locally scoped types get a unique name:
	if s.typ.kind in {tyEnum, tyRef, tyObject} and not isTopLevel(c):
	incl(s.flags, sfGenSym)

Not sure if this code is sound

[metagn]

So while the code above in semstmts is the issue, when that code is removed, codegen fails because the codegen just uses the sighash to mangle the type name, and the sighash doesn't receive block information (relevant test is https://github.com/nim-lang/Nim/blob/devel/tests/ccgbugs/tsamename3.nim). Type mangling here:

Nim/compiler/ccgtypes.nim

Lines 143 to 144 in 2e4e2f8

	typ.typeName(typ.loc.r)
	typ.loc.r.add $sig

I couldn't find where this was done at first so I attempted a more sweeping refactor where skLabel syms of the blocks are set as sym owners (and unnamed blocks gensym a name), but this still caused the issue for some types defined in templates.

So the simpler fix for now is to handle sighash conflicts in the type name mangling. I don't know how to do this properly and I would appreciate someone else looking into it because the "workaround" in semstmts is really bad.