Replace Cairo PDF device with native PDF device (v0.9.3)

The native PDF device becomes the default -d pdf output, generating PDF
content streams directly from the display list instead of rendering
through Cairo. This preserves original color spaces (CMYK, Gray, RGB)
rather than forcing RGB conversion.

Moved all native_pdf modules into devices/pdf/, removed the Cairo-based
pdf.py and pdf_injector.py, updated imports, control.py finalization,
device config (600 DPI, no ColorModel), and visual_test.py. Updated all
documentation to reflect the new architecture.

Author: AndyCappDev

README.md

@@ -3,7 +3,7 @@
 <p align="center"><strong>A modern, open-source PostScript interpreter written in Python.</strong></p>
 
 <p align="center">
-  <a href="https://github.com/AndyCappDev/postforge/releases"><img src="https://img.shields.io/badge/Version-0.9.2-green.svg" alt="Version 0.9.0"></a>
+  <a href="https://github.com/AndyCappDev/postforge/releases"><img src="https://img.shields.io/badge/Version-0.9.3-green.svg" alt="Version 0.9.0"></a>
   <a href="LICENSE.txt"><img src="https://img.shields.io/badge/License-AGPL--3.0-blue.svg" alt="License: AGPL-3.0"></a>
   <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/Python-3.13%2B-blue.svg" alt="Python 3.13+"></a>
 </p>
@@ -52,10 +52,11 @@ See the [sample gallery](docs/samples.md) for larger rendered examples.
   exploration, debugging, and experimentation
 - **Cython-Accelerated Execution** — Optional Cython-compiled execution loop
   providing 15–40% speedup depending on workload
-- **Multiple Output Formats** — PNG, PDF, SVG, and TIFF output via Cairo
-  graphics backend, plus an interactive Qt display window with a PostScript
-  command prompt; TIFF supports multi-page and CMYK output for prepress
-  workflows; extensible architecture makes it straightforward to add new devices
+- **Multiple Output Formats** — PNG, PDF, SVG, and TIFF output, plus an
+  interactive Qt display window with a PostScript command prompt; the PDF
+  device generates content streams directly, preserving CMYK/Gray/RGB color
+  spaces; TIFF supports multi-page and CMYK output for prepress workflows;
+  extensible architecture makes it straightforward to add new devices
 - **PDF Font Embedding** — Type 1 font reconstruction and subsetting,
   TrueType/CID font extraction with CIDToGIDMap and ToUnicode support
 - **EPS Support** — Automatic page cropping to EPS content dimensions with

docs/design/detailed-gap-analysis.md

@@ -45,7 +45,7 @@
 | Types 2-5 | **Accepted, not processed** | Dictionary validation passes but no halftone-specific rendering behavior. Falls through to device defaults. |
 | Types 6, 10, 16 | **Accepted, not processed** | Same as above |
 
-**Practical impact:** Low. PostForge outputs to Cairo-backed devices (PNG, PDF, SVG) which handle their own halftoning.
+**Practical impact:** Low. PostForge's raster devices (PNG, SVG, TIFF) use Cairo which handles its own halftoning, and the PDF device generates vector output directly.
 
 ### Transfer Functions
 

docs/developer/adding-output-devices.md

@@ -311,8 +311,8 @@ def render_display_list(
 
 - `page_height` is needed for the PostScript-to-Cairo coordinate system flip
   (PostScript origin is bottom-left, Cairo is top-left)
-- `deferred_text_objs` is used by the PDF device to collect text objects for
-  later font embedding. Pass `None` for bitmap devices.
+- `deferred_text_objs` can be used to collect text objects for later
+  processing. Pass `None` for bitmap devices.
 
 ### Option B: Process the Display List Directly
 
@@ -433,24 +433,23 @@ you can store arbitrary Python objects in it (class instances, lists,
 open file handles, etc.) using a bytes key, and retrieve them on subsequent
 `showpage` calls.
 
-The PDF device uses this to maintain a `PDFDocumentState` that holds the Cairo
-PDF surface, font tracker, deferred text objects, and page counter across all
-pages:
+The PDF device uses this to maintain a `PDFDocumentState` that holds the
+font tracker, accumulated page data, Type 3 font state, and page counter
+across all pages:
 
 ```python
 PDF_STATE_KEY = b'_PDFDocumentState'
 
 def showpage(ctxt, pd):
-    pdf_state = pd.get(PDF_STATE_KEY)
-    if pdf_state is None:
+    state = pd.get(PDF_STATE_KEY)
+    if state is None:
         # First page — initialize and store in page device dict
-        pdf_state = PDFDocumentState(file_path, width, height)
-        pd[PDF_STATE_KEY] = pdf_state
+        state = PDFDocumentState(file_path)
+        pd[PDF_STATE_KEY] = state
 
-    # Render current page using persistent state
-    pdf_state.start_new_page(...)
-    render_display_list(ctxt, pdf_state.context, ...)
-    pdf_state.finish_page()
+    # Generate content stream from display list and store page data
+    content_stream, ... = generate_content_stream(ctxt.display_list, ...)
+    state.pages.append(PageData(content_stream, width_pts, height_pts))
 ```
 
 This works because the page device dictionary is just a Python `dict` — you can
@@ -471,15 +470,15 @@ dict entirely are `setpagedevice` (which rebuilds it from scratch) and
 ### Job Finalization (PDF)
 
 Multi-page devices may need a finalization step after the last page. The PDF
-device uses a `finalize_document` function that closes the Cairo surface and
-injects embedded fonts. This is called from the job control code in
+device uses a `finalize` function that assembles all accumulated pages into
+the final PDF with embedded fonts. This is called from the job control code in
 `postforge/operators/control.py`:
 
 ```python
 # In control.py job cleanup:
-from ..devices.pdf.pdf import PDF_STATE_KEY, finalize_document
+from ..devices.pdf.pdf import PDF_STATE_KEY, finalize
 if PDF_STATE_KEY in pd:
-    finalize_document(pd)
+    finalize(pd)
 ```
 
 If your device needs finalization, follow the same pattern: export a
@@ -493,8 +492,8 @@ The PDF device uses `/TextRenderingMode /TextObjs` to receive structured text
 data instead of rendered glyph paths. It then:
 
 1. Tracks font usage across all pages via `FontTracker`
-2. Collects deferred `TextObj` elements that need font embedding
-3. At finalization, reconstructs Type 1 fonts and injects them into the PDF
+2. Generates PDF text operators (BT/ET blocks, TJ arrays with kern values)
+3. At finalization, embeds fonts (Type 1, CID, CFF, Type 42, Type 3) into the PDF
 
 This pattern is only needed for devices that require structured text
 information (e.g., for searchability or font embedding).
@@ -526,13 +525,14 @@ Key features: anti-alias mode support, configurable output path.
 ### PDF (`postforge/devices/pdf/`)
 
 Complex multi-page device. Maintains a `PDFDocumentState` across pages.
-Uses Cairo `PDFSurface` for graphics, then post-processes with pypdf for
-font embedding. Applies a scaling transform to convert from device coordinates
-(at `HWResolution`) to PDF points (72 DPI).
-
-Key features: persistent state, font tracking and embedding (Type 1 and
-CID/TrueType), deferred text rendering, document finalization, stream
-compression.
+Generates PDF content streams directly from the display list (does not use
+Cairo), preserving original color spaces (CMYK, Gray, RGB). Content stream
+generation is split into focused submodules (stroke_ops, text_ops, type3_ops,
+image_ops, shading_ops). The final PDF is assembled at document end via pypdf.
+
+Key features: persistent state, color space preservation, font tracking and
+embedding (Type 1, CID/TrueType, CFF, Type 42, Type 3), text batching with
+TJ arrays, document finalization.
 
 ### SVG (`postforge/devices/svg/`)
 

docs/developer/architecture-overview.md

@@ -39,14 +39,14 @@ PostScript Source ──► Tokenizer ──► Execution ──► │ Display
    like `fill`, `stroke`, and `show` append elements here.
 
 5. **Output Device** — When `showpage` fires, the accumulated display list is
-   handed to a device for rendering. Devices live in `postforge/devices/` and
-   the current implementation typically delegates to a shared Cairo rendering
-   backend, but this is not a requirement — an output device can use whatever
-   rendering method it wants to process the display list. After rendering,
-   `showpage` erases the display list and reinitializes the graphics state for
-   the next page. `copypage` follows the same rendering path but preserves
-   both the display list and graphics state, allowing further drawing on top
-   of the existing page contents.
+   handed to a device for rendering. Devices live in `postforge/devices/` —
+   raster devices (PNG, TIFF, Qt) and SVG use a shared Cairo rendering
+   backend, while the PDF device generates content streams directly from the
+   display list. An output device can use whatever rendering method it wants.
+   After rendering, `showpage` erases the display list and reinitializes the
+   graphics state for the next page. `copypage` follows the same rendering
+   path but preserves both the display list and graphics state, allowing
+   further drawing on top of the existing page contents.
 
 
 ## The Execution Engine
@@ -357,7 +357,7 @@ The display list is a flat Python list containing instances of these classes
 | `Stroke` | `stroke` | Stroked path with line properties and CTM |
 | `PatternFill` | `fill` with pattern color space | Pattern-tiled fill |
 | `ImageElement` | `image`, `imagemask`, `colorimage` | Raster image data |
-| `TextObj` | `show` (in TextObjs mode) | Text for native PDF output |
+| `TextObj` | `show` (in TextObjs mode) | Structured text for PDF output |
 | `ClipElement` | `clip`, `eoclip`, `initclip` | Clipping path update |
 | `GlyphRef` | show (cache hit) | Reference to cached glyph bitmap |
 | `GlyphStart`/`GlyphEnd` | show (cache miss) | Glyph bitmap capture markers |
@@ -421,19 +421,23 @@ conversion formulas (NTSC weighting for gray, etc.).
 ### Color Conversion at Rendering Time
 
 Color conversion is *lazy* — `setcolor` stores the color in the graphics
-state, but the conversion to device color (RGB for the Cairo renderer) happens
-only when a painting operator builds a display list element:
+state, but the conversion to device color happens only when a painting operator
+builds a display list element:
 
 1. A painting operator (`fill`, `stroke`, etc.) calls
    `ColorSpaceEngine.convert_to_device_color()`.
 2. The engine dispatches based on color space family — device spaces pass
    through (with cross-conversion if needed), CIE-based spaces run through
    their decode/matrix/XYZ pipeline, and ICCBased spaces apply an lcms2
    transform.
-3. The resulting RGB values are stored in the display list element (`Fill`,
-   `Stroke`, etc.).
-4. The rendering device receives pre-converted RGB and passes it straight
-   to Cairo.
+3. The resulting color values are stored in the display list element (`Fill`,
+   `Stroke`, etc.). When a `/ColorModel` is set in the page device (e.g.,
+   `/DeviceRGB` for Cairo-based raster devices), colors are converted to that
+   model. When no `/ColorModel` is set (the PDF device), original device color
+   spaces (CMYK, Gray, RGB) are preserved.
+4. The rendering device consumes these colors — Cairo-based devices receive
+   RGB, while the PDF device emits the appropriate PDF color operators for
+   whatever color space was used.
 
 ### ICC Color Management Tiers
 
@@ -472,10 +476,10 @@ Output devices render the display list into a final format. Each device
 consists of two parts that work together: a PostScript configuration file
 in `postforge/resources/OutputDevice/` (e.g., `png.ps`) that defines the page device
 dictionary, and a Python module in `postforge/devices/` that implements a
-`showpage(ctxt, pd)` function to perform the actual rendering. The built-in
-devices use a shared Cairo rendering backend, but this is a convenience, not
-a requirement — a custom device can use any rendering approach it wants
-without involving Cairo at all.
+`showpage(ctxt, pd)` function to perform the actual rendering. The raster
+devices (PNG, TIFF, Qt) use a shared Cairo rendering backend, while the PDF
+device generates PDF content streams directly from the display list. A custom
+device can use any rendering approach it wants.
 
 ### Device Architecture
 
@@ -486,32 +490,43 @@ without involving Cairo at all.
                     │                                                                │
                     │  cairo_renderer.py - dispatch     cairo_patterns.py - patterns │
                     │  cairo_images.py   - images       cairo_shading.py  - shading  │
-                    └─────┬────────────┬────────────┬─────────────┬────────────┬─────┘
-                          │            │            │             │            │
-                          ▼            ▼            ▼             ▼            ▼
-                     ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
-                     │   PNG    │ │   PDF    │ │   SVG    │ │   TIFF   │ │    Qt    │
-                     │  device  │ │  device  │ │  device  │ │  device  │ │  device  │
-                     └──────────┘ └────┬─────┘ └──────────┘ └──────────┘ └──────────┘
-                                       │
-                              ┌────────┴────────┐
-                              │ Font embedding  │
-                              │ (font_embedder, │
-                              │  cid_font_      │
-                              │  embedder,      │
-                              │  pdf_injector)  │
-                              └─────────────────┘
+                    └─────────┬──────────────┬──────────────┬──────────────┬─────────┘
+                              │              │              │              │
+                              ▼              ▼              ▼              ▼
+                         ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐
+                         │   PNG    │   │   SVG    │   │   TIFF   │   │    Qt    │
+                         │  device  │   │  device  │   │  device  │   │  device  │
+                         └──────────┘   └──────────┘   └──────────┘   └──────────┘
+
+                     ┌─────────────────────────────────────────┐
+                     │            PDF device                   │
+                     │         (devices/pdf/)                  │
+                     │                                         │
+                     │  pdf.py ──► content_stream.py           │
+                     │               ├─ stroke_ops.py          │
+                     │               ├─ text_ops.py            │
+                     │               ├─ type3_ops.py           │
+                     │               ├─ image_ops.py           │
+                     │               └─ shading_ops.py         │
+                     │          ──► pdf_builder.py             │
+                     │               ├─ font_embedder.py       │
+                     │               ├─ cid_font_embedder.py   │
+                     │               ├─ cff_font_embedder.py   │
+                     │               └─ font_tracker.py        │
+                     └─────────────────────────────────────────┘
 ```
 
 **PNG** (`postforge/devices/png/png.py`) — Creates a Cairo ImageSurface, calls
 `render_display_list()`, writes a `.png` file. The simplest device and a good
 starting point for understanding the rendering pipeline.
 
-**PDF** (`postforge/devices/pdf/`) — Renders to a Cairo PDFSurface, then
-post-processes the PDF with pypdf to inject embedded fonts. Text in PDF mode
-uses `TextObj` elements that are written as native PDF text operators, producing
+**PDF** (`postforge/devices/pdf/`) — Generates PDF content streams directly
+from the display list (does not use Cairo). Preserves original color spaces
+(CMYK, Gray, RGB) instead of converting everything to RGB. Text uses `TextObj`
+elements written as PDF text operators with TJ arrays and kern values, producing
 searchable/selectable text. Font embedding handles Type 1 reconstruction,
-CID/TrueType extraction, and subsetting.
+CID/TrueType extraction, CFF, Type 42, Type 3, and subsetting. The final PDF
+is assembled at document end via pypdf.
 
 **SVG** (`postforge/devices/svg/svg.py`) — Renders to a Cairo SVGSurface,
 then post-processes the SVG to convert text from outlines to selectable `<text>`
@@ -541,8 +556,10 @@ dictionary is loaded and merged into the graphics state's `page_device`.
 ### Shared Cairo Renderer
 
 `render_display_list()` in `postforge/devices/common/cairo_renderer.py` is the
-main dispatch loop. It iterates over display list elements and delegates to
-type-specific rendering functions:
+main dispatch loop used by the raster and vector-surface devices (PNG, SVG,
+TIFF, Qt). The PDF device does not use this renderer — it generates PDF content
+streams directly. The Cairo renderer iterates over display list elements and
+delegates to type-specific rendering functions:
 
 - Path construction → Cairo `move_to`, `line_to`, `curve_to`, `close_path`
 - Fill/Stroke → Cairo `fill` / `stroke` with color and line properties
@@ -554,10 +571,11 @@ type-specific rendering functions:
 
 **Stroke method**: For bitmap devices (PNG, Qt), strokes are converted to filled
 paths by the interpreter before they reach the display list. This works around
-bugs in Cairo's stroke rasterization, particularly with dashed lines. The PDF
-device uses Cairo's native stroke rendering instead. This behavior is controlled
-per-device by the `/StrokeMethod` entry in the page device dictionary (set in
-each device's `.ps` configuration file).
+bugs in Cairo's stroke rasterization, particularly with dashed lines. Vector
+devices (PDF, SVG) use native stroke rendering instead — PDF emits stroke
+operators directly, while SVG uses Cairo's vector surface. This behavior is
+controlled per-device by the `/StrokeMethod` entry in the page device dictionary
+(set in each device's `.ps` configuration file).
 
 
 ## Resource System

postforge/devices/common/cairo_renderer.py

@@ -8,7 +8,7 @@
 Shared Cairo Rendering Module
 
 This module provides the main display list dispatcher and text/glyph rendering
-logic used by multiple output devices (PNG, PDF, SVG, Qt).
+logic used by the Cairo-based output devices (PNG, SVG, TIFF, Qt).
 
 Architecture:
 - render_display_list() is the main entry point for device implementations