Fix /learn in 2.14.0 #747

michaelchia · 2024-04-26T11:12:52Z

Fixed /learn not working properly for directories in 2.14.0. For directories, it only kept the files from the final iteration of os.walk(path) instead of accumulating the filepaths across iterations.

for more information, see https://pre-commit.ci

srdas · 2024-04-26T14:33:57Z

@michaelchia Thanks for the PR, the key missing line is 80 below, and thanks for adjusting the rest of the code accordingly.

I fixed the failing test, and @dlqqq @3coins -- please take a look to confirm, the fix looks good, I have tested it as well.

3coins · 2024-04-26T15:32:03Z

@michaelchia
Thanks for submitting the fix. This is a critical piece of logic that should have some unit tests to make sure there are no regressions. Let us know if you need help with this.

michaelchia · 2024-04-26T15:58:27Z

I did look into the tests but didn't find any similar tests for other handlers or their supporting functions. I wasn't sure what the convention was for this project. It is probably best that the core team create the tests for this.

3coins · 2024-04-26T17:42:46Z

@michaelchia
Looks good, here are the results of my testing. @srdas will take care of the unit tests in a separate PR.

(jupyter-ai-lab4) ➜  playground git:(fix-learn) ✗ tree misc
misc
├── Chemical Equations and Stoichiometry.ipynb
├── Python Decorators: A Comprehensive Guide.ipynb
├── README.md
├── Testing, Training, and Deploying a Model with Amazon SageMaker.ipynb
├── Understanding Python Decorators.ipynb
├── Untitled.ipynb
├── Using the GitHub API: A Jupyter Notebook.ipynb
├── config.example.py
├── config.py
├── magics-demo.ipynb
├── magics.ipynb
├── neptune_utils.ipynb
├── notebooks
│   ├── An Overview of Traitlets.ipynb
│   ├── Introduction to Differential Equations.ipynb
│   └── Matplotlib Usage and Customization.ipynb
├── state-of-union
│   └── state_of_the_union.txt
├── test
│   ├── PDF generation test.ipynb
│   └── empty.ipynb
├── test.ipynb
└── test_generate_components.ipynb

Before

>>> from jupyter_ai.document_loaders.directory import split
>>> split('README.md', False, None)
Processing file: README.md
>>> split('misc', False, None)
dir is misc
subdirs before filtering: ['test', 'state-of-union', '.ipynb_checkpoints', 'notebooks']
subdirs after filtering: ['test', 'state-of-union', 'notebooks']
dir is misc/test
subdirs before filtering: ['.ipynb_checkpoints']
subdirs after filtering: []
dir is misc/state-of-union
subdirs before filtering: ['.ipynb_checkpoints']
subdirs after filtering: []
dir is misc/notebooks
subdirs before filtering: ['.ipynb_checkpoints']
subdirs after filtering: []
Processing file: misc/notebooks/Introduction to Differential Equations.ipynb
Processing file: misc/notebooks/An Overview of Traitlets.ipynb
Processing file: misc/notebooks/Matplotlib Usage and Customization.ipynb

After

>>> from jupyter_ai.document_loaders.directory import split
>>> split('README.md', False, None)
Processing file: README.md

>>> split('misc', False, None)
dir is misc
subdirs before filtering: ['test', 'state-of-union', '.ipynb_checkpoints', 'notebooks']
subdirs after filtering: ['test', 'state-of-union', 'notebooks']
dir is misc/test
subdirs before filtering: ['.ipynb_checkpoints']
subdirs after filtering: []
dir is misc/state-of-union
subdirs before filtering: ['.ipynb_checkpoints']
subdirs after filtering: []
dir is misc/notebooks
subdirs before filtering: ['.ipynb_checkpoints']
subdirs after filtering: []
Processing file: misc/test_generate_components.ipynb
Processing file: misc/magics-demo.ipynb
Processing file: misc/config.py
Processing file: misc/Untitled.ipynb
Processing file: misc/Testing, Training, and Deploying a Model with Amazon SageMaker.ipynb
Processing file: misc/magics.ipynb
Processing file: misc/README.md
Processing file: misc/config.example.py
Processing file: misc/Python Decorators: A Comprehensive Guide.ipynb
Processing file: misc/Chemical Equations and Stoichiometry.ipynb
Processing file: misc/neptune_utils.ipynb
Processing file: misc/Understanding Python Decorators.ipynb
Processing file: misc/Using the GitHub API: A Jupyter Notebook.ipynb
Processing file: misc/test.ipynb
Processing file: misc/test/PDF generation test.ipynb
Processing file: misc/test/empty.ipynb
Processing file: misc/state-of-union/state_of_the_union.txt
Processing file: misc/notebooks/Introduction to Differential Equations.ipynb
Processing file: misc/notebooks/An Overview of Traitlets.ipynb
Processing file: misc/notebooks/Matplotlib Usage and Customization.ipynb
Processing file: misc/test_generate_components.ipynb
Processing file: misc/magics-demo.ipynb
Processing file: misc/config.py
Processing file: misc/Untitled.ipynb
Processing file: misc/Testing, Training, and Deploying a Model with Amazon SageMaker.ipynb
Processing file: misc/magics.ipynb
Processing file: misc/README.md
Processing file: misc/config.example.py
Processing file: misc/Python Decorators: A Comprehensive Guide.ipynb
Processing file: misc/Chemical Equations and Stoichiometry.ipynb
Processing file: misc/neptune_utils.ipynb
Processing file: misc/Understanding Python Decorators.ipynb
Processing file: misc/Using the GitHub API: A Jupyter Notebook.ipynb
Processing file: misc/test.ipynb
Processing file: misc/test/PDF generation test.ipynb
Processing file: misc/test/empty.ipynb
Processing file: misc/state-of-union/state_of_the_union.txt
Processing file: misc/notebooks/Introduction to Differential Equations.ipynb
Processing file: misc/notebooks/An Overview of Traitlets.ipynb
Processing file: misc/notebooks/Matplotlib Usage and Customization.ipynb

* accumulate filepaths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

dlqqq · 2024-04-26T21:23:10Z

@meeseeksdev please backport to 1.x

Co-authored-by: michaelchia <michael_chia7@msn.com>

* learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * learn_arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * learn_arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Streamlined code for learning arxiv files (1) removed temp dir handling {2) extracted only tex files (3) Moved imports into the `arxiv_to_text` function {4) improved tar file processing * update learn for arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removed extra imports * Fix /learn in 2.14.0 (#747) * accumulate filepaths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update learn for arxiv files Redoing code after the PR 747 made changes to the same file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Improved code for arxiv files Improvements to PR 742: (i) removed extra `arxiv.Client` call (ii) removed unnecessary `try/catch` (iii) moved `datetime` import outside `arxiv_to_text` function * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Saves arxiv to root, better exception handling. * Added arxiv feature to docs. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelchia <michael_chia7@msn.com> Co-authored-by: Piyush Jain <piyushjain@duck.com>

* learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * learn_arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * learn_arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Streamlined code for learning arxiv files (1) removed temp dir handling {2) extracted only tex files (3) Moved imports into the `arxiv_to_text` function {4) improved tar file processing * update learn for arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removed extra imports * Fix /learn in 2.14.0 (jupyterlab#747) * accumulate filepaths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update learn for arxiv files Redoing code after the PR 747 made changes to the same file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Improved code for arxiv files Improvements to PR 742: (i) removed extra `arxiv.Client` call (ii) removed unnecessary `try/catch` (iii) moved `datetime` import outside `arxiv_to_text` function * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Saves arxiv to root, better exception handling. * Added arxiv feature to docs. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelchia <michael_chia7@msn.com> Co-authored-by: Piyush Jain <piyushjain@duck.com>

* save chat history to jupyter lab root dir The /export command was saving chat history in the pwd, i.e., the directory from where JupyterLab is launched rather than the JupyterLab root. For JupyterLab sessions, where the root directory is different from the launch directory, users will not be able to see the exported chat history file in JupyterLab. This is now rectified. * learn arxiv tex files (#742) * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * learn_arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * learn_arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Streamlined code for learning arxiv files (1) removed temp dir handling {2) extracted only tex files (3) Moved imports into the `arxiv_to_text` function {4) improved tar file processing * update learn for arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removed extra imports * Fix /learn in 2.14.0 (#747) * accumulate filepaths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update learn for arxiv files Redoing code after the PR 747 made changes to the same file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Improved code for arxiv files Improvements to PR 742: (i) removed extra `arxiv.Client` call (ii) removed unnecessary `try/catch` (iii) moved `datetime` import outside `arxiv_to_text` function * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Saves arxiv to root, better exception handling. * Added arxiv feature to docs. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelchia <michael_chia7@msn.com> Co-authored-by: Piyush Jain <piyushjain@duck.com> * Distinguish between completion and chat models (#711) * Distinguish between completion and chat models * Fix tests * Shorten the tab name, move settings button Lint * Implement the completion model selection in chat UI * Improve docstring * Call `_validate_lm_em_id` only once, add typing annotations * Remove embeddings provider for completions as the team has no plans to support it :( * Use type alias to reduce changeset/make review easier Without this change prettier reformats the plugin with an extra indentation, which leads to bad changeset display on GitHub. * Rename `_validate_lm_em_id` to `_validate_model_ids` * Rename `LLMHandlerMixin` to `CompletionsModelMixin` and rename the file from `llm_mixin` to `model_mixin` fro consistency. Of note, the file name does not need `completions_` prefix as the file is in `completions/` subdirectory. * Rename "Chat LM" to "LM"; add title attribute; note using the title attribute because getting the icon to show up nicely (getting they nice grey color and positioning as it gets in buttons, compared to just plain black) was not trivial; I think the icon might be the way to go in the future but I would postpone it to another PR. That said, I still think it should say "Chat LM" because it has no effect on magics nor completions. * Rename heading "Completer model" → "Inline completions model" * Move `UseSignal` down to `CompleterSettingsButton` implementation * Rename the label in the select to "Inline completion model" * Disable selection when completer is not enabled * Remove use of `UseSignal`, tweak naming of `useState` from `completerIsEnabled` to `isCompleterEnabled` * Use mui tooltips * Fix use of `jai_config_manager` * Fix tests * Fix `unsupported_slash_commands` default (#768) * Updates to /export command (1) Save chat history file to Jupyter root directory, not cwd, in markdown format (2) Add time stamps to the saved files (3) Enable the `/export <filename>` option, else use default file name. (4) Remove code to increment file numbers for multiple chat histories, given that the time stamps are now being used. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update export function (1) Use argparse to extract filename if the option is used (2) Remove the old function to get the chat file name and streamline the code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add Titan embedding model v2 (#778) * Add Titan embeddning model v2 Included the new embedding model, released recently on Amazon Bedrock. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * refactor /export code Remove if then and replace with a single inline expression * Refactor export code reprise Updated some variable names --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelchia <michael_chia7@msn.com> Co-authored-by: Piyush Jain <piyushjain@duck.com> Co-authored-by: Michał Krassowski <5832902+krassowski@users.noreply.github.com>

* save chat history to jupyter lab root dir The /export command was saving chat history in the pwd, i.e., the directory from where JupyterLab is launched rather than the JupyterLab root. For JupyterLab sessions, where the root directory is different from the launch directory, users will not be able to see the exported chat history file in JupyterLab. This is now rectified. * learn arxiv tex files (jupyterlab#742) * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * learn_arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * learn_arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Streamlined code for learning arxiv files (1) removed temp dir handling {2) extracted only tex files (3) Moved imports into the `arxiv_to_text` function {4) improved tar file processing * update learn for arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removed extra imports * Fix /learn in 2.14.0 (jupyterlab#747) * accumulate filepaths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update learn for arxiv files Redoing code after the PR 747 made changes to the same file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Improved code for arxiv files Improvements to PR 742: (i) removed extra `arxiv.Client` call (ii) removed unnecessary `try/catch` (iii) moved `datetime` import outside `arxiv_to_text` function * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Saves arxiv to root, better exception handling. * Added arxiv feature to docs. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelchia <michael_chia7@msn.com> Co-authored-by: Piyush Jain <piyushjain@duck.com> * Distinguish between completion and chat models (jupyterlab#711) * Distinguish between completion and chat models * Fix tests * Shorten the tab name, move settings button Lint * Implement the completion model selection in chat UI * Improve docstring * Call `_validate_lm_em_id` only once, add typing annotations * Remove embeddings provider for completions as the team has no plans to support it :( * Use type alias to reduce changeset/make review easier Without this change prettier reformats the plugin with an extra indentation, which leads to bad changeset display on GitHub. * Rename `_validate_lm_em_id` to `_validate_model_ids` * Rename `LLMHandlerMixin` to `CompletionsModelMixin` and rename the file from `llm_mixin` to `model_mixin` fro consistency. Of note, the file name does not need `completions_` prefix as the file is in `completions/` subdirectory. * Rename "Chat LM" to "LM"; add title attribute; note using the title attribute because getting the icon to show up nicely (getting they nice grey color and positioning as it gets in buttons, compared to just plain black) was not trivial; I think the icon might be the way to go in the future but I would postpone it to another PR. That said, I still think it should say "Chat LM" because it has no effect on magics nor completions. * Rename heading "Completer model" → "Inline completions model" * Move `UseSignal` down to `CompleterSettingsButton` implementation * Rename the label in the select to "Inline completion model" * Disable selection when completer is not enabled * Remove use of `UseSignal`, tweak naming of `useState` from `completerIsEnabled` to `isCompleterEnabled` * Use mui tooltips * Fix use of `jai_config_manager` * Fix tests * Fix `unsupported_slash_commands` default (jupyterlab#768) * Updates to /export command (1) Save chat history file to Jupyter root directory, not cwd, in markdown format (2) Add time stamps to the saved files (3) Enable the `/export <filename>` option, else use default file name. (4) Remove code to increment file numbers for multiple chat histories, given that the time stamps are now being used. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update export function (1) Use argparse to extract filename if the option is used (2) Remove the old function to get the chat file name and streamline the code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add Titan embedding model v2 (jupyterlab#778) * Add Titan embeddning model v2 Included the new embedding model, released recently on Amazon Bedrock. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * refactor /export code Remove if then and replace with a single inline expression * Refactor export code reprise Updated some variable names --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelchia <michael_chia7@msn.com> Co-authored-by: Piyush Jain <piyushjain@duck.com> Co-authored-by: Michał Krassowski <5832902+krassowski@users.noreply.github.com>

* save chat history to jupyter lab root dir The /export command was saving chat history in the pwd, i.e., the directory from where JupyterLab is launched rather than the JupyterLab root. For JupyterLab sessions, where the root directory is different from the launch directory, users will not be able to see the exported chat history file in JupyterLab. This is now rectified. * learn arxiv tex files (#742) * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * learn_arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * learn_arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Streamlined code for learning arxiv files (1) removed temp dir handling {2) extracted only tex files (3) Moved imports into the `arxiv_to_text` function {4) improved tar file processing * update learn for arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removed extra imports * Fix /learn in 2.14.0 (#747) * accumulate filepaths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update learn for arxiv files Redoing code after the PR 747 made changes to the same file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Improved code for arxiv files Improvements to PR 742: (i) removed extra `arxiv.Client` call (ii) removed unnecessary `try/catch` (iii) moved `datetime` import outside `arxiv_to_text` function * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Saves arxiv to root, better exception handling. * Added arxiv feature to docs. --------- * Distinguish between completion and chat models (#711) * Distinguish between completion and chat models * Fix tests * Shorten the tab name, move settings button Lint * Implement the completion model selection in chat UI * Improve docstring * Call `_validate_lm_em_id` only once, add typing annotations * Remove embeddings provider for completions as the team has no plans to support it :( * Use type alias to reduce changeset/make review easier Without this change prettier reformats the plugin with an extra indentation, which leads to bad changeset display on GitHub. * Rename `_validate_lm_em_id` to `_validate_model_ids` * Rename `LLMHandlerMixin` to `CompletionsModelMixin` and rename the file from `llm_mixin` to `model_mixin` fro consistency. Of note, the file name does not need `completions_` prefix as the file is in `completions/` subdirectory. * Rename "Chat LM" to "LM"; add title attribute; note using the title attribute because getting the icon to show up nicely (getting they nice grey color and positioning as it gets in buttons, compared to just plain black) was not trivial; I think the icon might be the way to go in the future but I would postpone it to another PR. That said, I still think it should say "Chat LM" because it has no effect on magics nor completions. * Rename heading "Completer model" → "Inline completions model" * Move `UseSignal` down to `CompleterSettingsButton` implementation * Rename the label in the select to "Inline completion model" * Disable selection when completer is not enabled * Remove use of `UseSignal`, tweak naming of `useState` from `completerIsEnabled` to `isCompleterEnabled` * Use mui tooltips * Fix use of `jai_config_manager` * Fix tests * Fix `unsupported_slash_commands` default (#768) * Updates to /export command (1) Save chat history file to Jupyter root directory, not cwd, in markdown format (2) Add time stamps to the saved files (3) Enable the `/export <filename>` option, else use default file name. (4) Remove code to increment file numbers for multiple chat histories, given that the time stamps are now being used. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update export function (1) Use argparse to extract filename if the option is used (2) Remove the old function to get the chat file name and streamline the code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add Titan embedding model v2 (#778) * Add Titan embeddning model v2 Included the new embedding model, released recently on Amazon Bedrock. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- * refactor /export code Remove if then and replace with a single inline expression * Refactor export code reprise Updated some variable names --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelchia <michael_chia7@msn.com> Co-authored-by: Piyush Jain <piyushjain@duck.com> Co-authored-by: Michał Krassowski <5832902+krassowski@users.noreply.github.com>

* accumulate filepaths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * learn_arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * learn_arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Streamlined code for learning arxiv files (1) removed temp dir handling {2) extracted only tex files (3) Moved imports into the `arxiv_to_text` function {4) improved tar file processing * update learn for arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removed extra imports * Fix /learn in 2.14.0 (jupyterlab#747) * accumulate filepaths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update learn for arxiv files Redoing code after the PR 747 made changes to the same file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Improved code for arxiv files Improvements to PR 742: (i) removed extra `arxiv.Client` call (ii) removed unnecessary `try/catch` (iii) moved `datetime` import outside `arxiv_to_text` function * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Saves arxiv to root, better exception handling. * Added arxiv feature to docs. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelchia <michael_chia7@msn.com> Co-authored-by: Piyush Jain <piyushjain@duck.com>

* save chat history to jupyter lab root dir The /export command was saving chat history in the pwd, i.e., the directory from where JupyterLab is launched rather than the JupyterLab root. For JupyterLab sessions, where the root directory is different from the launch directory, users will not be able to see the exported chat history file in JupyterLab. This is now rectified. * learn arxiv tex files (jupyterlab#742) * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * learn_arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * learn_arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Streamlined code for learning arxiv files (1) removed temp dir handling {2) extracted only tex files (3) Moved imports into the `arxiv_to_text` function {4) improved tar file processing * update learn for arxiv * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * removed extra imports * Fix /learn in 2.14.0 (jupyterlab#747) * accumulate filepaths * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * learn arxiv tex files * Created a new option remote or -r. Example: /learn -r arxiv <arxiv-id> * Approach: downloads the tar file for the entire paper into downloads_temp. Then, unzips and collects all .tex files in the tar file and concatenates them. Different authors use various approaches. Some have the entire paper in one tex file, whereas others may have separate tex files for each section, so we need to collect all the tex file into a single file and then hand off to the splitter, embedder. After completion, remove the temp directory. Return a properly formatted error if package arxiv needs to be installed. * Handle two types of errors: (i) package arxiv not installed. (ii) User enters a wrong paper id. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update learn for arxiv files Redoing code after the PR 747 made changes to the same file. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Improved code for arxiv files Improvements to PR 742: (i) removed extra `arxiv.Client` call (ii) removed unnecessary `try/catch` (iii) moved `datetime` import outside `arxiv_to_text` function * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Saves arxiv to root, better exception handling. * Added arxiv feature to docs. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelchia <michael_chia7@msn.com> Co-authored-by: Piyush Jain <piyushjain@duck.com> * Distinguish between completion and chat models (jupyterlab#711) * Distinguish between completion and chat models * Fix tests * Shorten the tab name, move settings button Lint * Implement the completion model selection in chat UI * Improve docstring * Call `_validate_lm_em_id` only once, add typing annotations * Remove embeddings provider for completions as the team has no plans to support it :( * Use type alias to reduce changeset/make review easier Without this change prettier reformats the plugin with an extra indentation, which leads to bad changeset display on GitHub. * Rename `_validate_lm_em_id` to `_validate_model_ids` * Rename `LLMHandlerMixin` to `CompletionsModelMixin` and rename the file from `llm_mixin` to `model_mixin` fro consistency. Of note, the file name does not need `completions_` prefix as the file is in `completions/` subdirectory. * Rename "Chat LM" to "LM"; add title attribute; note using the title attribute because getting the icon to show up nicely (getting they nice grey color and positioning as it gets in buttons, compared to just plain black) was not trivial; I think the icon might be the way to go in the future but I would postpone it to another PR. That said, I still think it should say "Chat LM" because it has no effect on magics nor completions. * Rename heading "Completer model" → "Inline completions model" * Move `UseSignal` down to `CompleterSettingsButton` implementation * Rename the label in the select to "Inline completion model" * Disable selection when completer is not enabled * Remove use of `UseSignal`, tweak naming of `useState` from `completerIsEnabled` to `isCompleterEnabled` * Use mui tooltips * Fix use of `jai_config_manager` * Fix tests * Fix `unsupported_slash_commands` default (jupyterlab#768) * Updates to /export command (1) Save chat history file to Jupyter root directory, not cwd, in markdown format (2) Add time stamps to the saved files (3) Enable the `/export <filename>` option, else use default file name. (4) Remove code to increment file numbers for multiple chat histories, given that the time stamps are now being used. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update export function (1) Use argparse to extract filename if the option is used (2) Remove the old function to get the chat file name and streamline the code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add Titan embedding model v2 (jupyterlab#778) * Add Titan embeddning model v2 Included the new embedding model, released recently on Amazon Bedrock. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * refactor /export code Remove if then and replace with a single inline expression * Refactor export code reprise Updated some variable names --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: michaelchia <michael_chia7@msn.com> Co-authored-by: Piyush Jain <piyushjain@duck.com> Co-authored-by: Michał Krassowski <5832902+krassowski@users.noreply.github.com>

michaelchia and others added 2 commits April 26, 2024 19:10

accumulate filepaths

ff55f3d

[pre-commit.ci] auto fixes from pre-commit.com hooks

ba2c364

for more information, see https://pre-commit.ci

michaelchia mentioned this pull request Apr 26, 2024

/learn not working properly for directories in 2.14.0 #748

Closed

srdas added the bug Something isn't working label Apr 26, 2024

3coins approved these changes Apr 26, 2024

View reviewed changes

3coins merged commit f0f1999 into jupyterlab:main Apr 26, 2024
8 of 9 checks passed

dlqqq mentioned this pull request Apr 26, 2024

Remove pypdf from required dependencies #752

Merged

meeseeksmachine pushed a commit to meeseeksmachine/jupyter-ai that referenced this pull request Apr 26, 2024

Backport PR jupyterlab#747: Fix /learn in 2.14.0

1577e40

meeseeksmachine mentioned this pull request Apr 26, 2024

Backport PR #747 on branch 1.x (Fix /learn in 2.14.0) #754

Merged

dlqqq pushed a commit that referenced this pull request Apr 26, 2024

Backport PR #747: Fix /learn in 2.14.0 (#754)

31ed11e

Co-authored-by: michaelchia <michael_chia7@msn.com>

srdas mentioned this pull request May 9, 2024

[1.x] Save chat history to Jupyter Lab's root directory (#770) #780

Merged

michaelchia deleted the fix-learn branch September 4, 2024 05:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix /learn in 2.14.0 #747

Fix /learn in 2.14.0 #747

michaelchia commented Apr 26, 2024

srdas commented Apr 26, 2024

3coins commented Apr 26, 2024 •

edited

Loading

michaelchia commented Apr 26, 2024

3coins commented Apr 26, 2024

dlqqq commented Apr 26, 2024

Fix /learn in 2.14.0 #747

Fix /learn in 2.14.0 #747

Conversation

michaelchia commented Apr 26, 2024

srdas commented Apr 26, 2024

3coins commented Apr 26, 2024 • edited Loading

michaelchia commented Apr 26, 2024

3coins commented Apr 26, 2024

Before

After

dlqqq commented Apr 26, 2024

3coins commented Apr 26, 2024 •

edited

Loading