Naming convention for runtime-compiled modules?

[gukoff]

Context

Debugpy uses the frame's co_filename in many ways. It uses it for debuginfo caching, to determine whether it's user code or not (which affects behavior of the setting justMyCode), and whether to interpret the source code as raw Python code or Python bytecode.

For example, the filename is handled differently if it begins with <. It is considered non-user code not sourced from a file:

• debugpy/src/debugpy/_vendored/pydevd/pydevd_file_utils.py

  Line 481 in 1aff9aa

  if filename.startswith("<"):

• debugpy/src/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_constants.py

  Line 62 in 1aff9aa

  LIBRARY_CODE_BASENAMES_STARTING_WITH = ("<",)

• debugpy/src/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_api.py

  Line 535 in 1aff9aa

  if adjust_line and not translated_absolute_filename.startswith("<"):

Some more special names include:

debugpy/src/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_constants.py

Lines 47 to 59 in 1aff9aa

	# Any filename that starts with these strings is not traced nor shown to the user.
	# In Python 3.7 "<frozen ..." appears multiple times during import and should be ignored for the user.
	# In PyPy "<builtin> ..." can appear and should be ignored for the user.
	# <attrs is used internally by attrs
	# <__array_function__ is used by numpy
	IGNORE_BASENAMES_STARTING_WITH = ("<frozen ", "<builtin", "<attrs", "<__array_function__")
	# Note: <string> has special heuristics to know whether it should be traced or not (it's part of
	# user code when it's the <string> used in python -c and part of the library otherwise).
	# Any filename that starts with these strings is considered user (project) code. Note
	# that files for which we have a source mapping are also considered as a part of the project.
	USER_CODE_BASENAMES_STARTING_WITH = ("<ipython",)

It is also important whether the file ends with ".py", as in this case inspecting this file in VSCode enables syntax highlighting.

This works well for the regular Python files. But there is an edge case of runtime-compiled modules for which you can assign the filename dynamically. For example: https://github.com/wbond/pymeta3/blob/master/pymeta/builder.py#L309-L326

Question

How do I pick the name for a runtime-compiled module (not backed up by an actual source file) so that it plays well with debugpy?

• If the filename starts with /, debugpy interprets this runtime-compiled module as user code, even if everything happens in a library. This may lead to cryptic errors.
• If the filename is something like <generated myfile.py>, VSCode renders it as Python bytecode even when Python code is available for the frame.
• If the file doesn't begin with < or /, it's normalized to a non-existent file path under .../site-packages/{filename}.

What works well is a filename such as <generated>myfile.py. It begins with < and ends with .py, which ensures a correct interpretation. I wonder if such a naming makes sense from the point of view of debugpy, and whether there's a better way to achieve my goal that I'm missing.

[rchiodo]

Sorry I transferred this to discussions as it's not really an issue. I believe there are settings you can use to specify which files are just my code or not. Looking them up now.

[rchiodo]

You should be able to mark files as system code by using the rules option in the launch.json (well assuming they're all generated in a specific directory).

See the discussion here:
#1170 (comment)

[gukoff]

Adjusting debugpy configuration like this works for the individual projects, but I don't see how it can scale to widely used libraries.

Take microsoft/semantic-kernel, a framework with 100.000s of users. Its transitive dependency is pymeta3 which runs some generated-and-compiled code during startup.

If you debug a Python application which uses semantic kernel and select to stop on user uncaught exceptions, you'll always stop on an exception with a stacktrace which doesn't make sense. It is very confusing, for instance:

• Runtime Parse Error When Using Semantic Kernel In Python Project PTVS#8022
• Python: Bug: Semantic Kernel breaks FASTAPI Debugging Launch config semantic-kernel#8420

The stacktrace here doesn't make sense, and the exception is not in fact user-uncaught. Debugpy makes a mistake because it interprets the compiled module as user code.

Unless I'm missing something, all 100.000s of users of semantic kernel would have to modify their launch.json in order to ignore certain file names of the generated modules. On the other hand, what would scale is if the library could somehow mark its compiled module specifically for debugpy as "non-user, generated code".

[rchiodo]

I think that would only be a problem if the semantic kernel generated code wasn't in the site-packages folder somewhere. Not sure where it ends up. But yes, if semantic kernel generated code outside of the paths debugpy searches, it will be marked as user code.

Right now I believe the only options are:

• The naming tricks you mentioned above
• Generate files somewhere that pydevd thinks they're system files
• Use the rules in the launch.json

Maybe we could add another naming hack? like if the name contained _generated_ in it or something.