OKF Parser (okf-parser)

A Pest DSL-powered Lexer & Parser Framework in Rust for the Open Knowledge Format (OKF) standard.

Open Knowledge Format (OKF) is an open, vendor-neutral specification for structuring organizational knowledge bases using plain Markdown files supplemented with structured YAML frontmatter. OKF allows both humans and AI agents to query, parse, and traverse knowledge graphs effortlessly.

---

Features

• Pest Lexer & Parser Framework (src/okf.pest):

  • Declarative DSL Grammar Specification File defining lexical rules, YAML frontmatter, and Markdown body semantics.
  • Distinguishes between YAML Frontmatter and Markdown Body.
  • Generates token streams with line and column position tracking (Position { line, column }).
  • Tokenizes OKF-specific constructs:
    • WikiLinks: Cross-concept links formatted as [[Target Concept]] or [[Target Concept|Alias]].
    • Markdown Links: [Text](URL).
    • Semantic Tags: #tag-name.
    • Blocks: Headings (#), Code Blocks (rust ... ), Lists (ordered and unordered), Blockquotes (>), and Horizontal Rules (---).

• Validated AST Parser (src/parser.rs):

  • Builds a structured OkfDocument Abstract Syntax Tree (AST).
  • Enforces OKF frontmatter validation (e.g., verifying the mandatory type key).
  • Extracts standard reserved metadata (title, description, resource, tags, timestamp) and stores arbitrary custom fields in extra.
  • Supports single-file parsing and batch directory validation (OkfParser::parse_directory).

• CLI & Library API:

  • Usable as a standalone CLI tool or integrated into Rust applications as a library crate.
  • Supports AST JSON export (--json) and raw token inspection (--tokens).

---

Project Structure

okf-parser/
├── Cargo.toml               # Package dependencies (pest, pest_derive, serde, thiserror)
├── README.md                # Documentation
├── src/
│   ├── okf.pest             # Pest DSL Grammar Specification File
│   ├── lib.rs               # Module re-exports
│   ├── lexer.rs             # Pest Lexer macro & token mapping
│   ├── parser.rs            # AST Parser & OKF frontmatter validator
│   └── main.rs              # Command-line application
└── examples/
    ├── machine_learning.md  # Example OKF concept document
    ├── neural_networks.md   # Example OKF document with code & lists
    └── deep_learning.md     # Example OKF guide document

---

Getting Started

Prerequisites

• Rust & Cargo (1.75+ recommended)

Build and Run Tests

# Clone or navigate to repository
cd okf-parser

# Run unit tests for lexer and parser
cargo test

# Build release binary
cargo build --release

---

CLI Usage

The package includes a command-line interface (okf-parser) for parsing OKF markdown files.

# Display help and options
cargo run -- --help

# Run interactive demonstration
cargo run -- --demo

# Parse a single OKF markdown file
cargo run -- examples/machine_learning.md

# Parse an entire directory of OKF markdown files
cargo run -- examples/

# Output directory parse results as JSON
cargo run -- examples/ --json

# Inspect raw token stream from the Lexer
cargo run -- examples/machine_learning.md --tokens

Sample JSON Output

Running cargo run -- examples/machine_learning.md --json outputs:

{
  "frontmatter": {
    "doc_type": "concept",
    "title": "Machine Learning Fundamentals",
    "description": "An introductory overview of machine learning algorithms and paradigms.",
    "resource": "https://example.org/ml-fundamentals",
    "tags": [
      "machine-learning",
      "artificial-intelligence",
      "data-science"
    ],
    "timestamp": "2026-08-01T10:00:00Z",
    "extra": {
      "author": "OKF Team"
    }
  },
  "body": [
    {
      "Heading": {
        "level": 1,
        "text": "Machine Learning Fundamentals"
      }
    },
    {
      "Paragraph": [
        {
          "Text": "Machine learning (ML) is a branch of artificial intelligence..."
        }
      ]
    },
    {
      "Heading": {
        "level": 2,
        "text": "Related Concepts"
      }
    },
    {
      "Paragraph": [
        {
          "Text": "To delve deeper into modern architectures, see "
        },
        {
          "WikiLink": {
            "target": "Neural Networks",
            "alias": null
          }
        },
        {
          "Text": " and "
        },
        {
          "WikiLink": {
            "target": "Deep Learning",
            "alias": "DL"
          }
        }
      ]
    }
  ]
}

Sample Token Stream Output

Running cargo run -- examples/neural_networks.md --tokens outputs:

[000] FrontmatterStart                             (span: line 1, column 1 -> line 12, column 4)
[001] YamlKey("type")                              (span: line 1, column 1 -> line 12, column 4)
[002] YamlValue("concept")                         (span: line 1, column 1 -> line 12, column 4)
[003] YamlKey("title")                             (span: line 1, column 1 -> line 12, column 4)
[004] YamlValue("Neural Networks")                 (span: line 1, column 1 -> line 12, column 4)
[005] FrontmatterEnd                               (span: line 1, column 1 -> line 12, column 4)
[006] Heading { level: 1, text: "Neural Networks" } (span: line 14, column 1 -> line 14, column 18)
[007] Newline                                      (span: line 14, column 1 -> line 14, column 18)
[008] Text("Neural networks form the foundation of modern ") (span: line 18, column 3 -> line 18, column 49)
[009] WikiLink { target: "Deep Learning", alias: None } (span: line 18, column 49 -> line 18, column 66)
[010] Text(" models.")                             (span: line 18, column 66 -> line 18, column 74)
[011] Eof                                          (span: line 1, column 1 -> line 1, column 1)

---

Rust Library API Usage

Add okf-parser to your Cargo.toml or embed the modules into your project:

use okf_parser::{Lexer, OkfParser, OkfDocument, BlockNode, InlineNode};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let input = r#"---
type: concept
title: Rust Lexer & Parser
tags:
  - rust
  - okf
---

# Rust Lexer & Parser

Check out [[Open Knowledge Format]] and #rust.
"#;

    // 1. Tokenize using the Lexer
    let mut lexer = Lexer::new(input);
    let tokens = lexer.tokenize()?;
    println!("Generated {} tokens", tokens.len());

    // 2. Parse into OKF Document AST
    let doc: OkfDocument = OkfParser::parse(input)?;
    println!("Document type: {}", doc.frontmatter.doc_type);
    println!("Title: {:?}", doc.frontmatter.title);
    println!("Body block count: {}", doc.body.len());

    // 3. Parse an entire directory of OKF documents
    let results = OkfParser::parse_directory("examples")?;
    for (rel_path, parse_result) in results {
        match parse_result {
            Ok(doc) => println!("Valid OKF doc: {} ({})", rel_path.display(), doc.frontmatter.doc_type),
            Err(err) => eprintln!("Error in {}: {}", rel_path.display(), err),
        }
    }

    Ok(())
}

---

OKF Formatting Specification Summary

An Open Knowledge Format file is a plain .md file structured as follows:

1. Frontmatter Header: Must start on line 1 with --- and end with ---.
   • type (Required): Defines the concept/document category (e.g. concept, guide, task).
   • Reserved Optional Fields: title, description, resource, tags, timestamp.
   • Custom Fields: Any additional key-value pairs are stored under extra.
2. Body Content: Standard Markdown enriched with:
   • [[Concept Name]] or [[Concept Name|Display Label]] for graph cross-references.
   • #tag-name for inline categorization.

---

License

Licensed under the MIT License.