How can I 'import' markdown pages.

[nicgord]

I have some markdown pages from my old wiki and want to bring them across.

I have copied them under root (and in subfolder under that) but they are not showing in the leafwiki UI?

How do I get them into tree.json so they are visible?

This may be linked to issue #470

I like this app and hope development continues.

[nicgord]

I did a bit of vibe coding and created a python script that recreates tree.json by searching for .md files in root and subfolders. (it also lowercases and removes spaces and _ in file names)

Seems to work well for my purposes, but would be a nice feature to have builtin I think.

Including in case anyone else wants to try/use.

e.g
sudo python3 generate_tree.py --root data/root --output data/tree.json

generate_tree.py

#!/usr/bin/env python3
"""
Script to generate tree.json for wikileaf from markdown files.
Scans data/root directory and creates the appropriate tree structure.
Automatically renames files and folders to follow wikileaf conventions (lowercase, hyphens).
"""

import os
import json
import random
import string
import re


def generate_id(length=9):
    """Generate a random ID similar to the wikileaf format."""
    chars = string.ascii_letters + string.digits + '_-'
    return ''.join(random.choice(chars) for _ in range(length))


def slug_to_title(slug):
    """Convert a slug to a title by capitalizing words."""
    # Replace hyphens and underscores with spaces
    title = slug.replace('-', ' ').replace('_', ' ')
    
    # Handle camelCase by inserting spaces before capitals
    title = re.sub(r'([a-z])([A-Z])', r'\1 \2', title)
    
    # Capitalize each word
    title = ' '.join(word.capitalize() for word in title.split())
    
    return title


def normalize_name(name):
    """
    Normalize a filename/foldername to wikileaf conventions.
    - Lowercase
    - Replace underscores with hyphens
    
    Args:
        name: Original name
    
    Returns:
        str: Normalized name
    """
    return name.lower().replace('_', '-')


def rename_to_wikileaf_convention(path, dry_run=False):
    """
    Rename a file or folder to follow wikileaf conventions if needed.
    
    Args:
        path: Path to the file or folder
        dry_run: If True, only print what would be renamed without doing it
    
    Returns:
        str: The new path (or original if no rename needed)
    """
    directory = os.path.dirname(path)
    basename = os.path.basename(path)
    
    # For files, keep the extension separate
    if os.path.isfile(path) and basename.endswith('.md'):
        name_part = basename[:-3]
        normalized = normalize_name(name_part)
        new_basename = normalized + '.md'
    else:
        new_basename = normalize_name(basename)
    
    # Check if rename is needed
    if basename == new_basename:
        return path
    
    new_path = os.path.join(directory, new_basename)
    
    if dry_run:
        print(f"Would rename: {basename} -> {new_basename}")
        return new_path
    
    # Perform the rename
    try:
        os.rename(path, new_path)
        print(f"Renamed: {basename} -> {new_basename}")
        return new_path
    except OSError as e:
        print(f"Error renaming {path}: {e}")
        return path


def process_directory(path, parent_slug='root', dry_run=False):
    """
    Process a directory and return its node structure.
    Automatically creates blank index.md files for folders that lack them.
    Renames files and folders to follow wikileaf conventions.
    
    Args:
        path: Path to the directory to process
        parent_slug: Slug of the parent (for generating proper slugs)
        dry_run: If True, don't actually rename files or create index.md files
    
    Returns:
        list: List of child nodes
    """
    children = []
    position = 0
    
    # Get all items in the directory
    try:
        items = sorted(os.listdir(path))
    except OSError as e:
        print(f"Error reading directory {path}: {e}")
        return []
    
    # First pass: rename files and directories to follow wikileaf conventions
    renamed_items = []
    for item in items:
        item_path = os.path.join(path, item)
        if item != 'index.md':  # Don't rename index.md
            new_path = rename_to_wikileaf_convention(item_path, dry_run)
            renamed_items.append(os.path.basename(new_path))
        else:
            renamed_items.append(item)
    
    # Re-sort after renaming
    items = sorted(renamed_items)
    
    # Separate files and directories
    md_files = []
    directories = []
    
    for item in items:
        item_path = os.path.join(path, item)
        if os.path.isfile(item_path) and item.endswith('.md') and item != 'index.md':
            md_files.append(item)
        elif os.path.isdir(item_path):
            directories.append(item)
    
    # Get list of directory names for checking duplicates
    dir_names = set(directories)
    
    # Process markdown files (excluding index.md)
    # Skip files that have a corresponding directory with the same name
    for md_file in md_files:
        file_base = md_file[:-3]  # Remove .md extension
        # Slug is already normalized from the rename
        slug = file_base
        
        # Skip this file if a directory with the same name exists
        # (the directory's index.md will represent this page instead)
        if file_base in dir_names:
            print(f"Skipping {md_file} - using folder {file_base}/ instead")
            continue
        
        node = {
            'id': generate_id(),
            'title': slug_to_title(file_base),
            'slug': slug,
            'children': [],
            'position': position
        }
        children.append(node)
        position += 1
    
    # Process subdirectories
    for directory in directories:
        dir_path = os.path.join(path, directory)
        # Slug is already normalized from the rename
        slug = directory
        index_path = os.path.join(dir_path, 'index.md')
        
        # Check if directory has an index.md, create if missing
        if not os.path.exists(index_path):
            if dry_run:
                print(f"Would create index.md for: {dir_path}")
            else:
                # Create a blank index.md file
                try:
                    with open(index_path, 'w', encoding='utf-8') as f:
                        f.write(f"# {slug_to_title(directory)}\n\n")
                    print(f"Created index.md for: {dir_path}")
                except OSError as e:
                    print(f"Error creating index.md in {dir_path}: {e}")
                    continue
        
        # Process the directory's children
        dir_children = process_directory(dir_path, slug, dry_run)
        
        node = {
            'id': generate_id(),
            'title': slug_to_title(directory),
            'slug': slug,
            'children': dir_children,
            'position': position
        }
        children.append(node)
        position += 1
    
    return children


def generate_tree_json(root_path='data/root', output_path='data/tree.json', dry_run=False):
    """
    Generate the tree.json file from markdown files.
    Automatically creates blank index.md files for folders that lack them.
    Automatically renames files and folders to follow wikileaf conventions.
    
    Args:
        root_path: Path to the root directory containing markdown files
        output_path: Path where tree.json should be written
        dry_run: If True, preview changes without actually making them
    """
    if not os.path.exists(root_path):
        print(f"Error: Root path {root_path} does not exist!")
        return False
    
    # Check if root has index.md, create if missing
    root_index_path = os.path.join(root_path, 'index.md')
    if not os.path.exists(root_index_path):
        if dry_run:
            print(f"Would create index.md for root directory")
        else:
            try:
                with open(root_index_path, 'w', encoding='utf-8') as f:
                    f.write("# Home\n\n")
                print(f"Created index.md for root directory")
            except OSError as e:
                print(f"Error creating root index.md: {e}")
    
    # Process the root directory
    children = process_directory(root_path, dry_run=dry_run)
    
    # Create the root node
    tree = {
        'id': 'root',
        'title': 'root',
        'slug': 'root',
        'children': children,
        'position': 0
    }
    
    if dry_run:
        print("\n=== DRY RUN - No files were modified ===")
        print(f"Tree structure preview:")
        print_tree(tree)
        return True
    
    # Write to file
    try:
        with open(output_path, 'w', encoding='utf-8') as f:
            json.dump(tree, f, ensure_ascii=False, separators=(',', ':'))
        print(f"\nSuccessfully generated {output_path}")
        print(f"Total nodes: {count_nodes(tree)}")
        return True
    except OSError as e:
        print(f"Error writing to {output_path}: {e}")
        return False


def count_nodes(node):
    """Count total nodes in the tree."""
    count = 1
    for child in node.get('children', []):
        count += count_nodes(child)
    return count


def print_tree(node, indent=0):
    """Print the tree structure for verification."""
    print('  ' * indent + f"- {node['title']} ({node['slug']})")
    for child in node.get('children', []):
        print_tree(child, indent + 1)


if __name__ == '__main__':
    import argparse
    
    parser = argparse.ArgumentParser(
        description='Generate tree.json for wikileaf from markdown files. Automatically renames files/folders to wikileaf conventions (lowercase, hyphens).'
    )
    parser.add_argument(
        '--root',
        default='data/root',
        help='Path to root directory (default: data/root)'
    )
    parser.add_argument(
        '--output',
        default='data/tree.json',
        help='Output path for tree.json (default: data/tree.json)'
    )
    parser.add_argument(
        '--preview',
        action='store_true',
        help='Preview changes without actually modifying files or generating tree.json'
    )
    
    args = parser.parse_args()
    
    # Generate the tree
    generate_tree_json(args.root, args.output, dry_run=args.preview)

[perber]

Hey @nicgord , thanks a lot for your kind words.
I really appreciate that you’re using LeafWiki!
And thank you for sharing your script, that’s a great contribution.

I’ve also been thinking about import functionality and/or keeping the file system in sync with the UI.
There are still a few conceptual questions I want to solve first, for example:

• What should happen if a folder doesn’t contain an index.md file?
• Is LeafWiki allowed to automatically rename files — or should the file system remain the “source of truth”? (This can become tricky with Git integration.)
• How should we handle existing links that point directly to .md files? Rewrite them or leave them as-is?
• Overall, this isn’t a small feature - there are quite a few edge cases to consider.

In the short term though, I can definitely imagine offering an importer - for example by uploading a ZIP archive that gets extracted and integrated into the wiki structure and tree.json.

Would that approach work for your use case?
What do you think?

And if it’s okay for you:
Could I include your script in the repository as an initial import solution, and as a starting point for a future, built-in importer?

Thanks again for your contribution and feedback — I really appreciate it!

[nicgord]

Of course you can include the script in the repository and change as you would like.

I think you could spend a lot of valuable development time on an importer that people use only once. (or never). Therefore, I think you just do what you have to to get the files imported and visible. (rename files as required, created index.md etc etc).

• What should happen if a folder doesn’t contain an index.md file?
  Create whatever files are missing to enable it to show in the UI. Users can then move around and rename again in the UI if they want.

• Is LeafWiki allowed to automatically rename files — or should the file system remain the “source of truth”? (This can become tricky with Git integration.)
  As part of an 'import' I think renaming is OK as I think would be one off.

• How should we handle existing links that point directly to .md files? Rewrite them or leave them as-is?
  Existing links are difficult... again would be alot of effort that only benefits a few... buyer beware I reckon.
  If I rename a page in leafwiki I assume links to that page break anyway right? (graphics files were another issue... was easier to just fix manually via copy/paste for me.)

• Overall, this isn’t a small feature - there are quite a few edge cases to consider.
  Agree, many wont need this capability at all and from my experimenting/debugging in creating the script there were a few gotchas (maybe more that I missed!) so agree not simple or highest priority.

My use case for this feature was a one off migration from wikijs.
I exported to html using builtin capability, used pandoc to convert to .md then the script to normalise and recreate tree.json. I can now restructure and rename/move in leafwiki directly and is quick and easy.

I am using this for managing my home/personal setup so I don't forget how I did things (am getting older!) and this seems perfect. Is 'simple' with no DB's to deal with, has docker container, and easy to backup restore with just flat files'. I dont need (or want) multi user capability, just a clean interface with a wysiwyg-ish editor.... and this seems just right.

I will be sure to post suggestions/issues as I use it more!
e.g I would love a 'LEAFWIKI_PUBLIC_ACCESS_ISADMIN' option ... so when I connect as anon I am admin and dont require a logons at all..... I will add this as a feature request :-)

[perber]

Yes, you’re right.
At the moment the links will break when renaming or moving pages. But I’m already storing the connections between pages so I can update the references later, whenever the user wants it. At least I’ll be able to warn them, like: “Careful! moving this page will break links on 7 other pages!”

And you’re actually the second person who has asked to disable authentication. To be honest, I expected that to be a pretty rare use case.

[perber]

I added the script as a community contribution to the repository and mentioned you there. I hope that’s okay for you.
I’ll keep it there until I have a built-in importer or sync.

[perber]

HI @nicgord,

just to let you know. The no-auth option will be released in v0.7.0.
I will ping you here as soon as I rolled it out.

[perber]

@nicgord v0.7.0 is out. So the no-auth option is available (--disable-auth)