Support JDT URIs from textDocument/defintion requests

[dannyfreeman]

I have been looking at an issue over in the JDT LSP repository: eclipse-jdtls/eclipse.jdt.ls#2322

A summary of how I reproduce this issue (with the latest snapshot of jdtls, not using eglot-java at all)

• Add this entry to the eglot-server-programs: (java-mode . ("jdtls" :initializationOptions (:extendedClientCapabilities (:classFileContentsSupport t))))
• clone groot, commit 1f275a41cb63dfb985c797934005bcd7159ed0a7
• cd groot && mvn compile
• visit groot/src/main/java/org/jlab/groot/math/F1D.java
• Execute xref-find-definitions on the Expression symbol on line 23

The relevant eglot logs

(:jsonrpc "2.0" :id 8 :method "textDocument/definition" :params
          (:textDocument
           (:uri "file:///home/user/dev/groot/src/main/java/org/jlab/groot/math/F1D.java")
           :position
           (:line 22 :character 4)))
[server-reply] (id:8) Sun Nov 13 13:37:57 2022:
(:jsonrpc "2.0" :id 8 :result
          [(:uri "jdt://contents/exp4j-0.4.4.jar/net.objecthunter.exp4j/Expression.class?=groot/%5C/home%5C/user%5C/.m2%5C/repository%5C/net%5C/objecthunter%5C/exp4j%5C/0.4.4%5C/exp4j-0.4.4.jar=/maven.pomderived=/true=/=/javadoc_location=/jar:file:%5C/home%5C/user%5C/.m2%5C/repository%5C/net%5C/objecthunter%5C/exp4j%5C/0.4.4%5C/exp4j-0.4.4-javadoc.jar%5C!%5C/=/=/maven.groupId=/net.objecthunter=/=/maven.artifactId=/exp4j=/=/maven.version=/0.4.4=/=/maven.scope=/compile=/=/maven.pomderived=/true=/%3Cnet.objecthunter.exp4j(Expression.class" :range
                 (:start
                  (:line 24 :character 13)
                  :end
                  (:line 24 :character 23)))])

I think this package would be a good place to handle those JDT URIs. There is a reply in the issue over in JDTLS repository that explains the process for handling them. Essentially the client must make an extra request to the java/classFileContents method of the language server, which will serve back the contents of the decompiled class. From there the client is free to dump the contents into a buffer.

I would be happy to take a swing at implementing this here at some point, but wanted to drop an issue here in case someone else wants to try and beat me to it.

[husainaloos]

Hi @dannyfreeman . Just wondering if you have any updates on this.

[dannyfreeman]

Hi @dannyfreeman . Just wondering if you have any updates on this.

None at all. I'm wondering if this repository is dead or obsolete?

[yveszoundi]

The repository is not "dead" per say, I will just say that the best way to contribute is via pull requests.

• I barely do any programming professionally or in my spare time nowadays, and rarely in Java lately
• I haven't look at the LSP specification itself for a very long time

@dannyfreeman, I was hoping that when ready you would create a pull request addressing the issue.

[dannyfreeman]

Thanks for the response! I also don't program in Java, but I'd still like to work on this. I'll see about getting a pull request against this repo.

[husainaloos]

@dannyfreeman I was wondering if you were able to look into this issue?

[husainaloos]

I was able to hack something (not lisp proficient). This is based on what I saw here: https://git.sr.ht/~dannyfreeman/jarchive/tree/main/item/jarchive.el

(require 'arc-mode)
(require 'cl-lib)
(require 'seq)
(require 'url-parse)

(defconst jdthandler--uri-regex
  (rx
   line-start
   "jdt://contents"
   "/" (* not-newline) ".jar"
   (group "/" (* (not "/")))
   "/"
   ;; match group 1, the jar file location
   (group (* not-newline) ".class")
   "?"
  (* not-newline)
   line-end))

(defvar jdthandler--eglot-server-process  nil)

(defun jdthandler--match! (uri)
  "Perform a regex match on the URI.
Expected by `jdthandler--match-jar' and `jdthandler--match-file'"
  (string-match jdthandler--uri-regex uri))

(defun jdthandler--match-jar (uri)
  "Extract the jar path from a URI.
`jdthandler--match!' must be called first"
  (substring uri (match-beginning 1) (match-end 1)))

(defun jdthandler--match-file (uri)
  "Extract the inter-jar file path from a URI.
`jdthandler--match!' must be called first"
  (substring uri (match-beginning 2) (match-end 2)))

(defmacro jdthandler--inhibit (op handler &rest body)
  "Run BODY with HANDLER inhibited for OP ."
  `(let ((inhibit-file-name-handlers (cons ,handler
                                           (and (eq inhibit-file-name-operation ,op)
                                                inhibit-file-name-handlers)))
         (inhibit-file-name-operation ,op))
     ,@body))
(defun jdthandler--find-file-not-found ()
  "Return t if the file not found was a file extracted by jdthandler.
TODO: this might be unnecessary, try to remove"
  (and (string-match-p jdthandler--uri-regex buffer-file-name)
       t))

(defmacro jdthandler--inhibit (op handler &rest body)
  "Run BODY with HANDLER inhibited for OP ."
  `(let ((inhibit-file-name-handlers (cons ,handler
                                           (and (eq inhibit-file-name-operation ,op)
                                                inhibit-file-name-handlers)))
         (inhibit-file-name-operation ,op))
     ,@body))

(defun jdthandler--file-name-handler (op &rest args)
  "A `file-name-handler-alist' function for files matching jar URIs.
Jar URIs are identified by `jdthandler--url-regex'.
OP is an I/O primitive and ARGS are the remaining arguments passed to that
primitive. See `(elisp)Magic File Names'."
  (if-let ((uri (car args)))  ;; Sometimes this is invoked with nil args
      (let* ((_   (jdthandler--match! uri))
             (jar-path (jdthandler--match-jar uri))
             (file-path (jdthandler--match-file uri)))
        (jdthandler--inhibit op 'jdthandler--file-name-handler
                           (cond
                            ((eq op 'expand-file-name) uri)
                            ((eq op 'file-truename) uri)
                            ((eq op 'file-name-directory) jar-path)
                            ((eq op 'file-name-sans-versions) file-path)
                            ((eq op 'file-name-as-directory) jar-path)
                            ((eq op 'file-name-nondirectory) file-path)
                            ((eq op 'directory-file-name) jar-path)
                            ((eq op 'file-name-case-insensitive-p) t)
                            ((eq op 'file-attributes) nil)
                            ((eq op 'make-auto-save-file-name) uri)
                            ((eq op 'abbreviate-file-name) uri)

                            ;; Predicates
                            ((eq op 'file-directory-p) nil)
                            ((eq op 'file-readable-p) t)
                            ((eq op 'file-writable-p) nil)
                            ((eq op 'file-exists-p) t)
                            ((eq op 'file-remote-p) nil)
                            ((eq op 'file-symlink-p) nil)
                            ((eq op 'file-accessible-directory-p) nil)
                            ((eq op 'file-executable-p) nil)
                            ((eq op 'vc-registered) nil)

                            ;; Custom implementations
                            ((eq op 'get-file-buffer)
                             (seq-find (lambda (buf)
                                         (string= uri (buffer-local-value 'buffer-file-name buf)))
                                       (buffer-list)))
                            ((eq op 'insert-file-contents) ;; This is executed in the context of a new buffer.
                             (cl-destructuring-bind (_filename visit beg end replace) args
                               (setq buffer-file-name uri)
                               ()
                               (when replace
                                 (erase-buffer))
                               (setq content (jsonrpc-request jdthandler--eglot-server-process
                                                              :java/classFileContents
                                                              `(:uri ,(browse-url-encode-url uri))))
                               (insert (format "%s" content))
                               (goto-char (point-min))
                               (unless visit
                                 (set-buffer-modified-p nil)
                                 (when (or beg end)
                                   (display-warning
                                    'jdthandler
                                    "The beg and end options are not respected by the jdthandler `insert-file-contents' handler."
                                    :warning)))
                               (setq buffer-offer-save nil)
                               (rename-buffer (format "%s/%s"
                                                      jar-path
                                                      file-path)
                                              t)
                               (list uri (string-width (buffer-string))))))))
    (jdthandler--inhibit op 'jdthandler--file-name-handler
                       (apply op args))))
(defun jdthandler-setup ()
  "Setup jdthandler, enabling Emacs to open files inside jar archives.
the files can be identified with the `jar' uri scheme."
  (interactive)
  (add-to-list 'file-name-handler-alist (cons jdthandler--uri-regex #'jdthandler--file-name-handler))
  (add-to-list 'find-file-not-found-functions #'jdthandler--find-file-not-found))

(defun jdthandler--wrap-legacy-eglot--path-to-uri (original-fn &rest args)
  "Hack until eglot is updated.
ARGS is a list with one element, a file path or potentially a URI.
If path is a jar URI, don't parse. If it is not a jar call ORIGINAL-FN."
  (setq jdthandler--eglot-server-process (eglot-current-server))
  (let ((path (file-truename (car args))))
    (if (equal "jdt" (url-type (url-generic-parse-url path)))
        path
      (apply original-fn args))))

(defun jdthandler--wrap-legacy-eglot--uri-to-path (original-fn &rest args)
  "Hack until eglot is updated.
ARGS is a list with one element, a URI.
If URI is a jar URI, don't parse and let the `jdthandler--file-name-handler'
handle it. If it is not a jar call ORIGINAL-FN."
  (setq jdthandler--eglot-server-process (eglot-current-server))  
  (let ((uri (car args)))
    (if (string= "file" (url-type (url-generic-parse-url uri)))
        (apply original-fn args)
      uri)))

(defun jdthandler-patch-eglot ()
  "Patch old versions of Eglot to work with Jdthandler."
  (interactive) ;; TODO, remove when eglot is updated in melpa
  (unless (or (and (advice-member-p #'jdthandler--wrap-legacy-eglot--path-to-uri 'eglot--path-to-uri)
                   (advice-member-p #'jdthandler--wrap-legacy-eglot--uri-to-path 'eglot--uri-to-path))
              (<= 29 emacs-major-version))
    (advice-add 'eglot--path-to-uri :around #'jdthandler--wrap-legacy-eglot--path-to-uri)
    (advice-add 'eglot--uri-to-path :around #'jdthandler--wrap-legacy-eglot--uri-to-path)
    (message "[jdthandler] Eglot successfully patched.")))

now you just need to run (jdthandler-setup) and (depending on whether you run old eglot or not) (jdthandler-patch-eglot).

[dannyfreeman]

I have been too busy to work on this, my apologies. Your solution based on jarchive I wouldn't consider a long term or permanent one. JDT urls are not meant to be parsed by the client. They are basically tokens that the jdt lsp server can use to serve back up the contents of the file using a special extension method:

eclipse-jdtls/eclipse.jdt.ls#2322 (comment)

This comment from @theothornhill seems to be a lot more on point, though I haven't tried it out (again, I really don't do any java development).

[yveszoundi]

I like what is suggested by in that jdt.ls comment, that also seems aligned with what lsp-java does (via a uri handler, except that the URI handler feature would be part of core emacs "in the future").

If I'm not mistaken @dannyfreeman , in comments from the eglot project issues (joaotavora/eglot#661), it feels like the maintainer strongly believed that any kind of URI handler belongs to core emacs from day one.

• That keeps the code clean in eglot and a clear separation of concerns, but also doesn't provide "any assisted incremental transitional path". Hopefully some level of support ends up in core emacs relatively soon.
• In helper libraries such as eglot-java, any elisp advice is also a bit fragile, as code around eglot--*--uri manipulations can change without notice

Still, kudos @husainaloos it's great that you were able to come up with a workaround!

[theothornhill]

Fwiw I already started a discussion over at emacs-devel, so we'll see if that example could go in core. I'm a little undecided because it uses eglot/java/jsonrpc-specific code, so it's hard to find a definitive spot for it. Maybe when @joaotavora sees it we'll know some more :)

[joaotavora]

Heads up, I'm not reading the emacs-devel daily lately. So if you want me to comment or weigh in on an issue, make sure to CC me in those emails (and even then I don't guarantee a timely response, but I do try).

[dannyfreeman]

The issue you linked @yveszoundi isn't quite the same as this one. The eglot discussion was related to a standardized jar scheme URL, which is well defined and used by more than just one language server. I would argue that it is useful outside of the language server context. With that understanding, I created the jarchive package that doesn't know anything about language servers, it only teaches Emacs how to open jar scheme URLs. I think there is a strong argument for it living in Emacs core instead of a separate package.

The issue here is how to open jdt style URLs, which are not standardized at all. They are bespoke URLs that are really only understood by jdtls and act more like tokens whose actual contents could change depending on the version of jdtls being used. That being said, I think the code for opening them belongs in a package specifically for dealing with eglot and jdtls (this one). Nothing outside of jdtls uses this URL scheme, and I don't think anything else should use this URL scheme. That makes it a bad candidate for inclusion in Emacs core IMO.

[yveszoundi]

Thanks for clarifying, I didn't read the entire thread for that other ticket.

I was mostly concerned about whether it's already possible to attach special "URI handlers" into emacs for arbitrary schemes and how well it plays along with URI related methods" in eglot. To be more specific,

• I was looking at the code sample from theothornhill (may or may not already work, I don't know). I should have linked directly that comment, instead of referencing older conversations not directly related, my bad.
• Then tie it all together with the existing eglot URI handling (eglot--path-to-uri and eglot--uri-to-path with or without an AOP advice)
• And then a subsequent JSON RPC calls to get class contents once the URI scheme is intercepted

tl;dr I do agree about the actual URI handling logic being in a specialized package such as this one. I hope that my position is "clear" enough.

[theothornhill]

(it works, you just have to supply the extended capabilities thingy - i use it every day)

[yveszoundi]

@dannyfreeman @husainaloos , can you confirm that the code below works for you for now?
If yes, I'll look into figuring out issue #9 and that should be most of what is needed (I think, on top of basic cleanup/configurability for cache directories, etc.).

1. Initialize eglot with jdtls

Please replace the jdtls path with a valid location on your machine.

(require 'eglot)
(setcdr (assoc 'java-mode eglot-server-programs) 
'("/Users/yveszoundi/.emacs.d/share/eclipse.jdt.ls/bin/jdtls" :initializationOptions 
(:extendedClientCapabilities (:classFileContentsSupport t))))

2. Add the URI handling logic

This is based on the sample code from @theothornhill and the AOP advice logic from @husainaloos .

(defun jdt-file-name-handler (operation &rest args)
  "Support Eclipse jdtls `jdt://' uri scheme."
  (let* ((uri (car args))
         (cache-dir "/tmp/.eglot")
         (source-file
          (expand-file-name
           (file-name-concat
            cache-dir
            (save-match-data
              (when (string-match "jdt://contents/\\(.*?\\)/\\(.*\\)\.class\\?" uri)
                (message "URI:%s" uri)
                (format "%s.java" (replace-regexp-in-string "/" "." (match-string 2 uri) t t))))))))
    (unless (file-readable-p source-file)
      (let ((content (jsonrpc-request (eglot-current-server) :java/classFileContents (list :uri uri)))
            (metadata-file (format "%s.%s.metadata"
                                   (file-name-directory source-file)
                                   (file-name-base source-file))))
        (message "content:%s" content)
        (unless (file-directory-p cache-dir) (make-directory cache-dir t))
        (with-temp-file source-file (insert content))
        (with-temp-file metadata-file (insert uri))))
    source-file))

(add-to-list 'file-name-handler-alist '("\\`jdt://" . jdt-file-name-handler))

(defun jdthandler--wrap-legacy-eglot--path-to-uri (original-fn &rest args)
  "Hack until eglot is updated.
ARGS is a list with one element, a file path or potentially a URI.
If path is a jar URI, don't parse. If it is not a jar call ORIGINAL-FN."
  (let ((path (file-truename (car args))))
    (if (equal "jdt" (url-type (url-generic-parse-url path)))
        path
      (apply original-fn args))))

(defun jdthandler--wrap-legacy-eglot--uri-to-path (original-fn &rest args)
  "Hack until eglot is updated.
ARGS is a list with one element, a URI.
If URI is a jar URI, don't parse and let the `jdthandler--file-name-handler'
handle it. If it is not a jar call ORIGINAL-FN."
  (let ((uri (car args)))
    (if (string= "file" (url-type (url-generic-parse-url uri)))
        (apply original-fn args)
      uri)))

(defun jdthandler-patch-eglot ()
  "Patch old versions of Eglot to work with Jdthandler."
  (interactive) ;; TODO, remove when eglot is updated in melpa
  (unless (or (and (advice-member-p #'jdthandler--wrap-legacy-eglot--path-to-uri 'eglot--path-to-uri)
                   (advice-member-p #'jdthandler--wrap-legacy-eglot--uri-to-path 'eglot--uri-to-path))
              (<= 29 emacs-major-version))
    (advice-add 'eglot--path-to-uri :around #'jdthandler--wrap-legacy-eglot--path-to-uri)
    (advice-add 'eglot--uri-to-path :around #'jdthandler--wrap-legacy-eglot--uri-to-path)
    (message "[jdthandler] Eglot successfully patched.")))

3. Patch eglot

(jdthandler-patch-eglot )

4. Test some code

I used the groot project as an example per original issue comments (specific git commit)

[theothornhill]

That should work just fine, @yveszoundi ☺️

[yveszoundi]

After visiting an initial "class contents buffer", further type navigation is not support.

This can be mitigated by the following workflow:

• Go back to the previous Emacs project buffer
• Call M-x xref-find-apropos with the name of the class to search for (fully qualified name or simple class name)
  • Sometimes the fully qualified class name gives you good results
  • However, if you don't see the class name in question, please use the simple class name instead in your search

[husainaloos]

Thank you all. The only change that I will make is from

(defun jdthandler--wrap-legacy-eglot--uri-to-path (original-fn &rest args)
  "Hack until eglot is updated.
ARGS is a list with one element, a URI.
If URI is a jar URI, don't parse and let the `jdthandler--file-name-handler'
handle it. If it is not a jar call ORIGINAL-FN."
  (let ((uri (car args)))
    (if (string= "file" (url-type (url-generic-parse-url uri)))
        (apply original-fn args)
      uri)))

to something like

(defun jdthandler--wrap-legacy-eglot--uri-to-path (original-fn &rest args)
  "Hack until eglot is updated.
ARGS is a list with one element, a URI.
If URI is a jar URI, don't parse and let the `jdthandler--file-name-handler'
handle it. If it is not a jar call ORIGINAL-FN."
  (let ((uri (car args)))
    (if (and (stringp uri)
                  (string= "jdt" (url-type (url-generic-parse-url uri))))
        uri
      (apply original-fn args))))

This is because the uri is not always string. I have noticed that when I run eglot-code-actions to import a missing import for a class, the uri is something like :file://<etc>. Other than that, this works.

Sorry, I don't really use eglot-java so I cannot comment on whether the code integrates well in the package.

[yveszoundi]

Thanks, that's great to know for the URI details, I'll perform a quick test and apply updates soon.

Your code does integrate seamlessly with this package and saved me time (as I "tinker" significantly less with elisp nowadays).