Implement document indexing pipeline

Add core components for document processing pipeline that supports
load → transform → vectorize → store workflow.

Core Components:
- DocumentProcessorInterface and DocumentProcessor for orchestrating the pipeline
- ReplaceTextTransformer for text replacement with validation
- Enhanced TextSplitTransformer with constructor parameters
- withContent() method for TextDocument for immutable content updates

Features:
- Support for multiple transformers in processing chain
- Array casting for flexible source input (string|array)
- Movie fixtures (gladiator, inception, jurassic-park) in Markdown format
- Example script demonstrating OpenAI embeddings with the pipeline

Implementation Details:
- Uses readonly classes for immutability
- Proper validation in transformers
- Comprehensive test coverage for withContent() method
- Clean separation of concerns between loading, transforming, and indexing

Author: OskarStark

examples/indexer/movies.php

@@ -0,0 +1,46 @@
+<?php
+
+/*
+ * This file is part of the Symfony package.
+ *
+ * (c) Fabien Potencier <fabien@symfony.com>
+ *
+ * For the full copyright and license information, please view the LICENSE
+ * file that was distributed with this source code.
+ */
+
+use Symfony\AI\Platform\Bridge\OpenAi\Embeddings;
+use Symfony\AI\Platform\Bridge\OpenAi\PlatformFactory;
+use Symfony\AI\Store\Bridge\Local\InMemoryStore;
+use Symfony\AI\Store\Document\DocumentProcessor;
+use Symfony\AI\Store\Document\Loader\TextFileLoader;
+use Symfony\AI\Store\Document\Transformer\ReplaceTextTransformer;
+use Symfony\AI\Store\Document\Transformer\TextSplitTransformer;
+use Symfony\AI\Store\Document\Vectorizer;
+use Symfony\AI\Store\Indexer;
+
+require_once dirname(__DIR__).'/bootstrap.php';
+
+$platform = PlatformFactory::create(env('OPENAI_API_KEY'), http_client());
+$store = new InMemoryStore();
+$processor = new DocumentProcessor(
+    new TextFileLoader(),
+    [
+        new ReplaceTextTransformer('## Plot', '## Synopsis'),
+        new TextSplitTransformer(500, 100),
+    ],
+    new Indexer(new Vectorizer($platform, new Embeddings('text-embedding-3-small')), $store)
+);
+
+$movies = [
+    dirname(__DIR__, 2).'/fixtures/movies/gladiator.md',
+    dirname(__DIR__, 2).'/fixtures/movies/inception.md',
+    dirname(__DIR__, 2).'/fixtures/movies/jurassic-park.md',
+];
+
+$processor->process($movies);
+
+$results = $store->search($platform->invoke(new Embeddings('text-embedding-3-small'), 'Roman gladiator revenge')->asVectors()[0], 2);
+foreach ($results as $i => $result) {
+    echo sprintf("%d. [%.0f%%] %s\n", $i + 1, $result['similarity'] * 100, substr($result['document']->id, 0, 40).'...');
+}

fixtures/movies/gladiator.md

@@ -0,0 +1,32 @@
+# Gladiator (2000)
+
+**IMDB**: https://www.imdb.com/title/tt0172495/
+
+**Director:** Ridley Scott
+
+## Cast
+
+- **Russell Crowe** as Maximus Decimus Meridius
+- **Joaquin Phoenix** as Emperor Commodus
+- **Connie Nielsen** as Lucilla
+- **Oliver Reed** as Proximo
+- **Derek Jacobi** as Senator Gracchus
+- **Djimon Hounsou** as Juba
+- **Richard Harris** as Marcus Aurelius
+- **Ralf Möller** as Hagen
+- **Tommy Flanagan** as Cicero
+- **David Schofield** as Falco
+
+## Plot
+
+A former Roman General sets out to exact vengeance against the corrupt emperor who murdered his family and sent him into slavery.
+
+**Maximus Decimus Meridius** is a powerful Roman general beloved by the people and the aging Emperor **Marcus Aurelius**. As Marcus Aurelius lies dying, he makes known his wish that Maximus should succeed him and return Rome to the former glory of the Republic rather than the corrupt Empire it has become.
+
+However, Marcus Aurelius's son **Commodus** learns of his father's plan and murders him before he can publicly name Maximus as his successor. Commodus then orders the execution of Maximus and his family. Maximus escapes the execution but arrives at his farm too late to save his wife and son.
+
+Wounded and devastated, Maximus is captured by slave traders and forced to become a gladiator. Under the training of **Proximo**, a former gladiator, Maximus becomes a skilled fighter and eventually makes his way to the **Colosseum** in Rome, where he gains fame and the crowd's favor.
+
+Using his newfound popularity with the people, Maximus seeks to avenge the murder of his family and fulfill his promise to Marcus Aurelius to restore Rome to a republic. The film culminates in a final confrontation between Maximus and Commodus in the arena.
+
+The film explores themes of *honor*, *revenge*, *political corruption*, and the struggle between personal desires and duty to the greater good.

fixtures/movies/inception.md

@@ -0,0 +1,28 @@
+# Inception (2010)
+
+**IMDB**: https://www.imdb.com/title/tt1375666/
+
+**Director:** Christopher Nolan
+
+## Cast
+
+- **Leonardo DiCaprio** as Dom Cobb
+- **Marion Cotillard** as Mal Cobb
+- **Tom Hardy** as Eames
+- **Elliot Page** as Ariadne
+- **Ken Watanabe** as Saito
+- **Dileep Rao** as Yusuf
+- **Cillian Murphy** as Robert Fischer Jr.
+- **Tom Berenger** as Peter Browning
+- **Michael Caine** as Professor Stephen Miles
+- **Lukas Haas** as Nash
+
+## Plot
+
+A skilled thief is given a chance at redemption if he can successfully perform inception, the act of planting an idea in someone's subconscious.
+
+**Dom Cobb** is a skilled thief who specializes in *extraction* - stealing secrets from people's subconscious minds while they dream. This unique skill has made him a valuable player in the world of corporate espionage, but it has also cost him everything he loves. Cobb's rare ability has made him a coveted player in this treacherous new world of corporate espionage, but it has also made him an international fugitive and cost him everything he has ever loved.
+
+Now Cobb is being offered a chance at redemption. One last job could give him his life back but only if he can accomplish the impossible - **inception**. Instead of the perfect heist, Cobb and his team of specialists have to pull off the reverse: their task is not to steal an idea but to plant one. If they succeed, it could be the perfect crime.
+
+The film explores themes of *reality*, *dreams*, *memory*, and the nature of consciousness through multiple layers of dream states, creating a complex narrative structure that challenges both characters and audience to question what is real.

fixtures/movies/jurassic-park.md

@@ -0,0 +1,30 @@
+# Jurassic Park (1993)
+
+**IMDB**: https://www.imdb.com/title/tt0107290/
+
+**Director:** Steven Spielberg
+
+## Cast
+
+- **Sam Neill** as Dr. Alan Grant
+- **Laura Dern** as Dr. Ellie Sattler
+- **Jeff Goldblum** as Dr. Ian Malcolm
+- **Richard Attenborough** as John Hammond
+- **Bob Peck** as Robert Muldoon
+- **Martin Ferrero** as Donald Gennaro
+- **BD Wong** as Dr. Henry Wu
+- **Joseph Mazzello** as Tim Murphy
+- **Ariana Richards** as Lex Murphy
+- **Wayne Knight** as Dennis Nedry
+
+## Plot
+
+During a preview tour, a theme park suffers a major power breakdown that allows its cloned dinosaur exhibits to run amok.
+
+Billionaire **John Hammond** has created a theme park on a remote island where he has successfully cloned dinosaurs from ancient DNA found in prehistoric mosquitoes preserved in amber. Before opening to the public, Hammond invites a select group of people to tour the park, including paleontologist **Dr. Alan Grant**, paleobotanist **Dr. Ellie Sattler**, and mathematician **Dr. Ian Malcolm**.
+
+The tour begins smoothly, but things quickly go wrong when the park's computer systems are sabotaged by the disgruntled programmer **Dennis Nedry**, who is attempting to steal dinosaur embryos. The security systems fail, and the dinosaurs break free from their enclosures.
+
+As the island descends into chaos, the visitors must survive encounters with various dangerous dinosaurs, including the intelligent and deadly **Velociraptors** and the massive **Tyrannosaurus Rex**. Dr. Grant finds himself responsible for Hammond's grandchildren, Tim and Lex, as they attempt to reach safety.
+
+The film explores themes of *scientific ethics*, the *hubris of trying to control nature*, and the *unintended consequences of genetic engineering*. It questions whether humans have the right to resurrect extinct species and whether scientific advancement should be pursued without considering the potential risks and moral implications.

src/store/src/Document/DocumentProcessor.php

@@ -0,0 +1,68 @@
+<?php
+
+/*
+ * This file is part of the Symfony package.
+ *
+ * (c) Fabien Potencier <fabien@symfony.com>
+ *
+ * For the full copyright and license information, please view the LICENSE
+ * file that was distributed with this source code.
+ */
+
+namespace Symfony\AI\Store\Document;
+
+use Psr\Log\LoggerInterface;
+use Psr\Log\NullLogger;
+use Symfony\AI\Store\Indexer;
+
+/**
+ * Default implementation of DocumentProcessorInterface that orchestrates
+ * the complete document processing pipeline: load → transform → vectorize → store.
+ *
+ * @author Oskar Stark <oskarstark@googlemail.com>
+ */
+final readonly class DocumentProcessor implements DocumentProcessorInterface
+{
+    /**
+     * @param TransformerInterface[] $transformers
+     */
+    public function __construct(
+        private LoaderInterface $loader,
+        private array $transformers,
+        private Indexer $indexer,
+        private LoggerInterface $logger = new NullLogger(),
+    ) {
+    }
+
+    public function process(string|array $source, array $options = []): void
+    {
+        $this->logger->debug('Starting document processing', [
+            'source' => $source,
+            'options' => $options,
+        ]);
+
+        $sources = (array) $source;
+        $allDocuments = [];
+
+        // Load documents from all sources
+        foreach ($sources as $singleSource) {
+            $documents = ($this->loader)($singleSource, $options['loader'] ?? []);
+            foreach ($documents as $document) {
+                $allDocuments[] = $document;
+            }
+        }
+
+        // Transform documents through all transformers
+        $transformedDocuments = $allDocuments;
+        foreach ($this->transformers as $transformer) {
+            $transformedDocuments = ($transformer)($transformedDocuments, $options['transformer'] ?? []);
+        }
+
+        // Vectorize and store documents
+        $this->indexer->index($transformedDocuments, $options['chunk_size'] ?? 50);
+
+        $this->logger->debug('Document processing completed', [
+            'source' => $source,
+        ]);
+    }
+}

src/store/src/Document/DocumentProcessorInterface.php

@@ -0,0 +1,31 @@
+<?php
+
+/*
+ * This file is part of the Symfony package.
+ *
+ * (c) Fabien Potencier <fabien@symfony.com>
+ *
+ * For the full copyright and license information, please view the LICENSE
+ * file that was distributed with this source code.
+ */
+
+namespace Symfony\AI\Store\Document;
+
+
+/**
+ * High-level interface for processing documents through the complete pipeline:
+ * load → transform → vectorize → store.
+ *
+ * @author Oskar Stark <oskarstark@googlemail.com>
+ */
+interface DocumentProcessorInterface
+{
+    /**
+     * Process a source through the complete indexing pipeline.
+     *
+     * @param string|array<string> $source  Source identifier (file path, URL, etc.) or array of sources
+     * @param array<string, mixed> $options Processing options
+     */
+    public function process(string|array $source, array $options = []): void;
+
+}

src/store/src/Document/TextDocument.php

@@ -28,4 +28,9 @@ public function __construct(
             throw new InvalidArgumentException('The content shall not be an empty string.');
         }
     }
+
+    public function withContent(string $content): self
+    {
+        return new self($this->id, $content, $this->metadata);
+    }
 }

src/store/src/Document/Transformer/ReplaceTextTransformer.php

@@ -0,0 +1,57 @@
+<?php
+
+/*
+ * This file is part of the Symfony package.
+ *
+ * (c) Fabien Potencier <fabien@symfony.com>
+ *
+ * For the full copyright and license information, please view the LICENSE
+ * file that was distributed with this source code.
+ */
+
+namespace Symfony\AI\Store\Document\Transformer;
+
+use Symfony\AI\Store\Document\Metadata;
+use Symfony\AI\Store\Document\TextDocument;
+use Symfony\AI\Store\Document\TransformerInterface;
+use Symfony\Component\Uid\Uuid;
+
+/**
+ * Replaces specified text within document content.
+ *
+ * @author Oskar Stark <oskarstark@googlemail.com>
+ */
+final readonly class ReplaceTextTransformer implements TransformerInterface
+{
+    public const OPTION_SEARCH = 'search';
+    public const OPTION_REPLACE = 'replace';
+
+    public function __construct(
+        private string $search = '',
+        private string $replace = '',
+    ) {
+        self::validate($search, $replace);
+    }
+
+    /**
+     * @param array{search?: string, replace?: string} $options
+     */
+    public function __invoke(iterable $documents, array $options = []): iterable
+    {
+        $search = $options[self::OPTION_SEARCH] ?? $this->search;
+        $replace = $options[self::OPTION_REPLACE] ?? $this->replace;
+
+        self::validate($search, $replace);
+
+        foreach ($documents as $document) {
+            yield $document->withContent(str_replace($search, $replace, $document->content));
+        }
+    }
+
+    private static function validate(string $search, string $replace): void
+    {
+        if ($search === $replace) {
+            throw new \InvalidArgumentException('Search and replace strings must be different.');
+        }
+    }
+}

src/store/src/Document/Transformer/TextSplitTransformer.php

@@ -29,13 +29,22 @@
     public const OPTION_CHUNK_SIZE = 'chunk_size';
     public const OPTION_OVERLAP = 'overlap';
 
+    public function __construct(
+        private int $chunkSize = 1000,
+        private int $overlap = 200,
+    ) {
+        if ($this->overlap < 0 || $this->overlap >= $this->chunkSize) {
+            throw new InvalidArgumentException(sprintf('Overlap must be non-negative and less than chunk size. Got chunk size: %d, overlap: %d', $this->chunkSize, $this->overlap));
+        }
+    }
+
     /**
      * @param array{chunk_size?: int, overlap?: int} $options
      */
     public function __invoke(iterable $documents, array $options = []): iterable
     {
-        $chunkSize = $options[self::OPTION_CHUNK_SIZE] ?? 1000;
-        $overlap = $options[self::OPTION_OVERLAP] ?? 200;
+        $chunkSize = $options[self::OPTION_CHUNK_SIZE] ?? $this->chunkSize;
+        $overlap = $options[self::OPTION_OVERLAP] ?? $this->overlap;
 
         if ($overlap < 0 || $overlap >= $chunkSize) {
             throw new InvalidArgumentException('Overlap must be non-negative and less than chunk size.');

src/store/tests/Document/TextDocumentTest.php

@@ -247,4 +247,38 @@ public function testExceptionMessageIsCorrect()
 
         new TextDocument(Uuid::v4(), '   ');
     }
+
+    #[TestDox('withContent() creates new instance with updated content')]
+    public function testWithContent()
+    {
+        $id = Uuid::v4();
+        $originalContent = 'Original content';
+        $newContent = 'Updated content';
+        $metadata = new Metadata(['title' => 'Test Document']);
+
+        $originalDocument = new TextDocument($id, $originalContent, $metadata);
+        $updatedDocument = $originalDocument->withContent($newContent);
+
+        // Original document is unchanged
+        $this->assertSame($originalContent, $originalDocument->content);
+        
+        // New document has updated content but same ID and metadata
+        $this->assertSame($newContent, $updatedDocument->content);
+        $this->assertSame($id, $updatedDocument->id);
+        $this->assertSame($metadata, $updatedDocument->metadata);
+        
+        // Different instances
+        $this->assertNotSame($originalDocument, $updatedDocument);
+    }
+
+    #[TestDox('withContent() validates new content')]
+    public function testWithContentValidatesContent()
+    {
+        $document = new TextDocument(Uuid::v4(), 'Valid content');
+
+        $this->expectException(InvalidArgumentException::class);
+        $this->expectExceptionMessage('The content shall not be an empty string.');
+
+        $document->withContent('   ');
+    }
 }