PYTHONPATH in dap-debug

[lccambiaghi]

How can I set the env variable PYTHONPATH when I run dap-debug?

My debug template looks like:

    (dap-register-debug-template "Debug Python"
                             (list :type "python"
                                   :args "-i"
                                   :cwd (expand-file-name "~/git/projectname")
                                   :env '(("PYTHONPATH" . "/Users/luca/git/projectname/src/"))
                                   :request "launch"
                                   :name "Python :: Run Configuration"))

And I get an error in the first lines because of a failed import.

I use direnv-mode and in my .envrc file I have exported my PYTHONPATH. It seems the env variable is not picked up by dap-mode. This works for pytest, the env variables are set like this: https://github.com/wbolster/emacs-python-pytest/blob/master/python-pytest.el#L529-L531 . I tried to replicate the same behaviour with:

(after! (direnv dap-python)
  (defvar direnv-non-file-modes)
  (add-to-list 'direnv-non-file-modes 'dap-mode))

but it does not work.

Thank you for your help.

[yyoncho]

Use this:

(dap-debug
 (list :type "python"
       :args ""
       :cwd nil
       :module nil
       :program "/home/kyoncho/.temp/test.py"
       :request "launch"
       :environment-variables '(("FOO" . "BAR"))
       :name "Python :: Run Configuration"
       :hostName "localhost"
       :host "localhost"))

[lccambiaghi]

Using your exact template, only replacing :program and :environment-variables, it does not work. Why did you close the issue?

[yyoncho]

Can you put this in test.py and then edit :program to point to that file and report if it works or not.

import os

print(os.environ['FOO'])

[lccambiaghi]

-*- mode: dap-server-log; default-directory: "~/git/emptiesforecast/" -*-
Debug Adapter started at Mon Jan 27 23:42:05

/Users/luca/git/emptiesforecast/.venv/bin/python -m ptvsd --wait --host localhost --port 49947  /Users/luca/git/emptiesforecast/scratch.py 
Traceback (most recent call last):
  File "/Users/luca/.pyenv/versions/3.7.6/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/luca/.pyenv/versions/3.7.6/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/luca/git/emptiesforecast/.venv/lib/python3.7/site-packages/ptvsd/__main__.py", line 446, in <module>
    main(sys.argv)
  File "/Users/luca/git/emptiesforecast/.venv/lib/python3.7/site-packages/ptvsd/__main__.py", line 432, in main
    run()
  File "/Users/luca/git/emptiesforecast/.venv/lib/python3.7/site-packages/ptvsd/__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "/Users/luca/.pyenv/versions/3.7.6/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/Users/luca/.pyenv/versions/3.7.6/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/Users/luca/.pyenv/versions/3.7.6/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/luca/git/emptiesforecast/scratch.py", line 3, in <module>
    print(os.environ["FOO"])
  File "/Users/luca/.pyenv/versions/3.7.6/lib/python3.7/os.py", line 679, in __getitem__
    raise KeyError(key) from None
KeyError: 'FOO'

A detail: I have overridden this function to return the python executable in my .venv:

(after! dap-mode
  (defun dap-python--pyenv-executable-find (command)
    (concat (getenv "VIRTUAL_ENV") "/bin/python")))

It is weird: I am using python and ptvsd from my virtualenv but somehow the env variables come from the pyenv global environment.

[yyoncho]

Can you do M-: (setenv "FOO" "BAR") RET and then M-x shell-command and execute whatever you try to do. Because if the way you are running the python process discards the env variables there is nothing we can do in dap-mode but you should handle it in your config.

[lccambiaghi]

I setenv.

I run /Users/luca/git/emptiesforecast/.venv/bin/python /Users/luca/git/emptiesforecast/scratch.py in shell, I get the correct env variable.

I run /Users/luca/git/emptiesforecast/.venv/bin/python -m ptvsd --host localhost --port 49947 /Users/luca/git/emptiesforecast/scratch.py and I get the correct env variable.

I try to add the --wait argument (with a breakpoint() in my code) and the process just hangs. The command is /Users/luca/git/emptiesforecast/.venv/bin/python -m ptvsd --wait --host localhost --port 5678 /Users/luca/git/emptiesforecast/scratch.py. Whatever.

Looking at the log I shared with you, I suspect the problem lies in the function dap-start-debugging: the :program-to-start is constructed correctly but when it calls compilation-start I think it calls some other environment than the one where I set the :environment-variables.

[yyoncho]

Can you do M-x trace-function RET setenv and also M-x trace-function RET dap-start-debugging.

And then execute: #202 (comment)

Here it is the output on my side:

1 -> (dap-start-debugging (:type "python" :args "" :cwd nil :module nil :program "/home/kyoncho/.temp/test.py" :request "launch" :environment-variables (("FOO" . "BAR2")) :name "Python :: Run Configuration" :hostName #1="localhost" :host #1# :program-to-start "/usr/bin/python -m ptvsd --wait --host localhost --port 42271  /home/kyoncho/.temp/test.py " :debugServer 42271 :port 42271))
| 2 -> (setenv "FOO" "BAR2")
| 2 <- setenv: "BAR2"
1 <- dap-start-debugging: nil

[lccambiaghi]

If I run that, dap-mode uses my pyenv python executable, instead of my virtualenv python executable and the function fails because ptvsd is only installed in the virtualenv.

When I patch dap-python--pyenv-executable-find and call dap-debug with the template, the env variable is not set as in my previous comment.

For now, I "solved" it by overriding the whole dap-python--populate-start-file-args function by prepending the PYTHONPATH manually:

    (defun my/dap-python--populate-start-file-args (conf)
        "Populate CONF with the required arguments."
        [...]
            (plist-put conf :program-to-start
                    (format "%s %s%s -m ptvsd --wait --host %s --port %s %s %s %s"
                            (concat "PYTHONPATH=" (getenv "PYTHONPATH"))
                            (or dap-python-terminal "")
                            (shell-quote-argument python-executable)
                            host
                            debug-port
                            (if module (concat "-m " (shell-quote-argument module)) "")
                            (shell-quote-argument program)
                            python-args))
[...]

It is a nasty hack but at least I can import my sources..

This issue highlights how:

1. dap-python should support using a python executable in a virtualenv. This is how lsp-python-ms does it: https://github.com/emacs-lsp/lsp-python-ms/blob/master/lsp-python-ms.el#L257-L264
2. dap-mode should integrate with direnv-mode. This is how python-pytest does it: https://github.com/wbolster/emacs-python-pytest/blob/master/python-pytest.el#L529-L531

It would be really cool if you could implement these two features :)

[yyoncho]

@mpanarin willing to take a look? I think that you did some work related to dap-python.el and virtualenvs?

[mpanarin]

@lccambiaghi lsp-python-ms support you linked tries to guess the virtualenv.
With pyls it is a bit different. Pyls should be installed in the virtualenv of your project, for completion and other things to work. So we expect that you activated your virtualenv in emacs.
So, here https://github.com/emacs-lsp/dap-mode/blob/master/dap-python.el#L71 what we do:

1. Check if pyenv installed, if it is -> use its env as it takes precedence (can be changed, probably. I don't use pyenv though).
2. Run executable-find to get python path. If virtualenv is activated in emacs (through pyvenv manually or with-venv-el for example) it will fetch the venv path of python. If not -> system path

IMO, guessing virtualenv is a bad idea. With packages above you can make selection explicit.
Or you can make it automatic with dir-locals, packages like auto-virtualenv or check how spacemacs does it automatically with with-venv-el.

[lccambiaghi]

I think it can be a problem on my side since I can't run an ipython shell with the env variables set either.
I am struggling getting direnv-mode to work. What is a good strategy to manage virtualenv and PYTHONPATH automatically?