Search returning entire page in search box [mkdocs-material insiders]

[albertkun]

Contribution guidelines

• I've read the contribution guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the problem doesn't occur with the mkdocs or readthedocs themes
• ... the problem persists when all overrides are removed, i.e. custom_dir, extra_javascript and extra_css
• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Using mkdocs-material insiders search function seems to be returning the entire pages content alongside the paragraph symbol in the search results. This only occurs using deployment on GitHub pages, not locally.

Expected behaviour

I expect the search function to work the same on deployment as it does locally and not display the entire content of the page in the search tool, as follows:

Actual behaviour

The insiders version of mkdocs-material makes the search result return all of the page in its entirety.

Steps to reproduce

1. Add tags to pages using mkdocs-material insiders
2. Commit to GitHub repository
3. Use GitHub actions to deploy:

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: 3.x
        env:
      - run: pip install git+https://${ALBERT_KEY}@github.com/squidfunk/mkdocs-material-insiders.git
        env:
          ALBERT_KEY: ${{secrets.ALBERT_KEY}}      

4. Test out search
5. Search bug appears.

Package versions

• Python: Python 3.9.5
• MkDocs: mkdocs, version 1.2.2
• Material: 7.2.8+insiders.3.0.1

Configuration

site_name: LA Metro Digital Services Team Docs
repo_url: https://github.com/LACMTA/digital-services-team-docs/

theme:
  palette:
    - scheme: default
      primary: black
      toggle:
        icon: material/toggle-switch-off-outline
        name: Switch to dark mode
    - scheme: slate
      primary: black
      accent: white
      toggle:
        icon: material/toggle-switch
        name: Switch to light mode  
  font:
    text: Open Sans
  favicon: assets/media/favicon.png
  logo: assets/media/favicon-reverse.png
  name: material
  features:
      - navigation.instant
      - navigation.tabs
      - content.code.annotate
      - search.highlight
      - toc.integrate
      - navigation.indexes
      - navigation.top
      - search.suggest      

  icon:
    repo: fontawesome/brands/github

markdown_extensions:
    - pymdownx.superfences
    - pymdownx.magiclink
    - smarty
    - meta
    - toc:
        permalink: True
        toc_depth: 2
    - sane_lists
    - admonition
    - footnotes
    - attr_list

edit_uri: edit/dev/docs/
plugins:
  - git-revision-date
  - autorefs
  - table-reader
  - macros
  - tags
  - search:
      lang: en

System information

• Operating system: Windows 10
• Browser: Firefox 92.0

[squidfunk]

Thanks for the reporting. That looks off indeed. It's weird that it only happens when run from GitHub Actions. Could you please include the Markdown of the page and upload the contents of the search/search_index.json file? That could be a great help in finding and isolating the issue.

[squidfunk]

The project in question is here, correct?
https://github.com/LACMTA/digital-services-team-docs/

[squidfunk]

Okay, here's what I think is happening:

I took a look at your repository and realized that the search_index.json has the old format, meaning that it was not generated by the new search plugin. The old format includes exactly what you're seeing – the contents of the entire page with permalink markers etc. in the main entry of the page:

Expand

{
  "config":{
    "indexing":"full",
    "lang":[
      "en"
    ],
    "min_search_length":3,
    "prebuild_index":false,
    "separator":"[\\s\\-]+"
  },
  "docs":[
    {
      "location":"",
      "text":"Welcome to LA Metro Digital Design System Docs \u00b6 This repository is an internal resource for the Digital Services Team at Metro. Documentation either does not exist, is outdated, exists on old platforms, or exists across multiple platforms used by different people. There is a need to centralize and keep documentation up-to-date for the entire team to use and reference. Do not put any account credentials or private keys here. Contributing \u00b6 To provide more information on the various Digital Design Projects at LA Metro, people can contribute to this documentation by following these steps: Pre-requisites \u00b6 Software \u00b6 Make sure you have the following installed: - GitSVN - VS Code (or other text based editor) Accounts \u00b6 -GitHub.com account Clone the Repository \u00b6 Open your editor and run in the terminal: git clone https://github.com/albertkun/digital-design-system-docs.git Categories \u00b6 Web Properties Tools/Services (through P-Card or ITS) Processes Shakeup updates Questions \u00b6 Feel free to reach out if you have any questions!",
      "title":"Welcome to LA Metro Digital Design System Docs"
    },
    {
      "location":"#welcome-to-la-metro-digital-design-system-docs",
      "text":"This repository is an internal resource for the Digital Services Team at Metro. Documentation either does not exist, is outdated, exists on old platforms, or exists across multiple platforms used by different people. There is a need to centralize and keep documentation up-to-date for the entire team to use and reference. Do not put any account credentials or private keys here.",
      "title":"Welcome to LA Metro Digital Design System Docs"
    },
    {
      "location":"#contributing",
      "text":"To provide more information on the various Digital Design Projects at LA Metro, people can contribute to this documentation by following these steps:",
      "title":"Contributing"
    },
    {
      "location":"#pre-requisites",
      "text":"",
      "title":"Pre-requisites"
    },
    {
      "location":"#clone-the-repository",
      "text":"Open your editor and run in the terminal: git clone https://github.com/albertkun/digital-design-system-docs.git",
      "title":"Clone the Repository"
    },
    {
      "location":"#categories",
      "text":"Web Properties Tools/Services (through P-Card or ITS) Processes Shakeup updates",
      "title":"Categories"
    },
    {
      "location":"#questions",
      "text":"Feel free to reach out if you have any questions!",
      "title":"Questions"
    }
  ]
} 

This is also (a subset?) of what you've posted in the screenshot – the main entry lists all contents including headlines and permalink markers. My guess is that when you've visited the page, somehow a stale version of the search_index.json was loaded. It means that your browser probably hasn't requested the new search_index.json due to some cache headers telling it not to, but loaded the new JavaScript and CSS sources, because they include content hashes, which the search index doesn't.

I would kindly ask you to verify two things:

1. Please check that the search_index.json which is loaded in the browser has the correct format. Please see the latest blog post, which explains in detail how the old and new search_index.json differ, and how it should look, and inspect the contents of the search_index.json in the developer tools.

2. Try to deploy your project to another domain, and see whether the issue persists. If it goes away, it's very likely related to cache headers, because the cache requests will now miss due to the new domain.

---

On a side note: a few days ago, somebody also reported this on Gitter, but the problem seems to have just gone away because I didn't hear back from him, which also sounds like a caching problem that has solved itself.

[albertkun]

The project in question is here, correct?
https://github.com/LACMTA/digital-services-team-docs/

yes, that is correct!

[albertkun]

Okay, here's what I think is happening:

I took a look at your repository and realized that the search_index.json has the old format, meaning that it was not generated by the new search plugin. The old format includes exactly what you're seeing – the contents of the entire page with permalink markers etc. in the main entry of the page:
Expand

This is also (a subset?) of what you've posted in the screenshot – the main entry lists all contents including headlines and permalink markers. My guess is that when you've visited the page, somehow a stale version of the search_index.json was loaded. It means that your browser probably hasn't requested the new search_index.json due to some cache headers telling it not to, but loaded the new JavaScript and CSS sources, because they include content hashes, which the search index doesn't.

I would kindly ask you to verify two things:

1. Please check that the `search_index.json` which is loaded in the browser has the correct format. Please see the [latest blog post](https://squidfunk.github.io/mkdocs-material/blog/2021/search-better-faster-smaller/), which explains in detail how the old and new `search_index.json` differ, and how it should look, and inspect the contents of the `search_index.json` in the developer tools.

2. Try to deploy your project to another domain, and see whether the issue persists. If it goes away, it's very likely related to cache headers, because the cache requests will now miss due to the new domain.

On a side note: a few days ago, somebody also reported this on Gitter, but the problem seems to have just gone away because I didn't hear back from him, which also sounds like a caching problem that has solved itself.

the search.json has the incorrect format I believe, here is how it looks in the browser:

In addition to the link:
https://lacmta.github.io/digital-services-team-docs/search/search_index.json

Regarding deploying to another domain, how should I approach this using GitHub pages? Should I just clone into a new repository?

Thank you for taking the time to look into this!!

[squidfunk]

Thanks for investigating. Yes, that looks broken indeed. If you compare it to the search_index.json that you get when you do a local mkdocs build, it will be very different. For this reason, the reported behavior can be clearly attributed to a broken deployment or invalid cache headers. As you're using GitHub, the former is more likely to be true, because they use very short durations (just checked: cache-control: max-age=600 for JSON).

As the site folder is part of master branch, it looks like you're using something different from the default behavior of MkDocs. The simplest fix is probably just to use the recommended approach, which will deploy to a dedicated branch called gh-pages. If that is not what you're aiming for, you should probably ask a team member or somebody else in your organization with the right privileges to fix the configuration. A good start is the GitHub Actions workflow from the docs.

I'm closing this issue since this is not an error that is related to Material for MkDocs, but to a deployment misconfiguration.

---

Regarding deploying to another domain, how should I approach this using GitHub pages? Should I just clone into a new repository?

You could fork the repository and deploy it again, yes.

[albertkun]

@squidfunk hmm, working on this a little more... I'm still unable to resolve this, even by forking or using the default github actions... I was able to resolve this by running mkdocs gh-deploy --force locally...

However, this solution is not ideal since we would like to run github actions to build it.. Here's the current github actions .yml file:

name: Deploy Documentation 
on:
  push:
    branches: 
      - main
jobs:
  deploy:
    runs-on: ubuntu-latest
    if: github.event.repository.fork == false
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: 3.x
      - run: pip install git+https://${GH_TOKEN}@github.com/LACMTA/mkdocs-material-insiders.git
      - run: mkdocs gh-deploy --force

env:
  GH_TOKEN: ${{secrets.ALBERT_KEY}}          
  METRO_DOCS_PASS: ${{secrets.METRO_DOCS_PASS}}

https://github.com/LACMTA/digital-services-team-docs/blob/dev/.github/workflows/main.yml

Any other insights would be appreciated!! Thank you!

[squidfunk]

Okay, so I can finally reproduce this issue and think I've found the cause of the problem.

Insight and observations

1. Insiders provides its own search plugin implementation. It exposes the plugin via entry_points with the same name, because it is a full replacement for MkDocs built-in search plugin. In fact, the built-in search plugin causes the behavior that can be seen (full page content in first result), which means that since Insiders 3.0 it is essentially not supported anymore, as it is superseded by the custom implementation.

2. Exporting a plugin with the same name works in most, but not call cases. I haven't found out why it sometimes works and why it doesn't. The problem manifests in the get_plugins function in MkDocs core, which loads the plugin entry points via importlib_metadata. On my system, importlib_metadata.entry_points() returns:

   [
     EntryPoint(name='search', value='mkdocs.contrib.search:SearchPlugin', group='mkdocs.plugins'),
     EntryPoint(name='search', value='material.plugins.search.plugin:SearchPlugin', group='mkdocs.plugins'),
     EntryPoint(name='social', value='material.plugins.social.plugin:SocialPlugin', group='mkdocs.plugins'),
     EntryPoint(name='tags', value='material.plugins.tags.plugin:TagsPlugin', group='mkdocs.plugins')
   ]
   

   Since the built-in plugin is loaded first, it is overwritten by the custom search plugin. However, I've managed to reproduce the described error in GitHub Actions, where importlib_metadata.entry_points() returns:

   [
     EntryPoint(name='search', value='material.plugins.search.plugin:SearchPlugin', group='mkdocs.plugins'),
     EntryPoint(name='social', value='material.plugins.social.plugin:SocialPlugin', group='mkdocs.plugins'),
     EntryPoint(name='tags', value='material.plugins.tags.plugin:TagsPlugin', group='mkdocs.plugins'),
     EntryPoint(name='search', value='mkdocs.contrib.search:SearchPlugin', group='mkdocs.plugins')
   ]
   

   This means that the built-in search plugin takes precedence. I don't yet understand why, it seems unpredictable.

3. Why renaming the search plugin is not a good option. So, we could just rename the search plugin or prefix it, right? Theoretically, yes. However, this would probably lead to more confusion, because the search plugin is enabled by default, so users now always have to explicitly list the new search plugin. It would also mean that some templates might break, because we're explicitly checking for the search plugin in some places. Several users override the header partial, so we would have to carry out a major release, with the potential for a lot of issues rolling in, since search is so widely used.

Workaround

As a temporary workaround, this installation method seems to work reliably:

git clone --depth 1 https://${GH_TOKEN}@github.com/squidfunk/mkdocs-material-insiders.git
pip install -e mkdocs-material-insiders

Mitigations

The best idea I could think of would be to patch this function in MkDocs to effectively allow for overriding the search plugin. Personally, I believe this is a sound case. After all, the MkDocs team always advertised that the built-in search plugin is only meant for demonstration purposes, and they've always encouraged users to provide alternate implementations. Here we have an alternate implementation that is superior in many ways.

The necessary patch would be:

def get_plugins():
    """ Return a dict of all installed Plugins as {name: EntryPoint}. """

    plugins = importlib_metadata.entry_points(group='mkdocs.plugins')

    pluginmap = {}
    for plugin in plugins:
        if plugin.name in pluginmap and "mkdocs.contrib" in plugin.value:
            continue

        pluginmap[plugin.name] = plugin

    return pluginmap

@oprypin, @facelessuser, @ultrabug – looping you in for a constructive discussion on this topic. If there's any other possibility without patching core I'm happy to discuss. Nonetheless, I'd expect the proposed patch to not break anything, and to be sound in terms of community-contributed plugins.