[Examples][Store] Implement indexing pipeline

Author: OskarStark

demo/CLAUDE.md

@@ -36,7 +36,7 @@ composer install
 echo "OPENAI_API_KEY='sk-...'" > .env.local
 
 # Initialize vector store
-symfony console app:blog:embed -vv
+symfony console ai:store:index blog -vv
 
 # Test vector store
 symfony console app:blog:query

demo/README.md

@@ -74,7 +74,7 @@ The [Chroma DB](https://www.trychroma.com/) is a vector store that is used to st
 To initialize the Chroma DB, you need to run the following command:
 
 ```shell
-symfony console app:blog:embed -vv
+symfony console ai:store:index blog -vv
 ```
 
 Now you should be able to run the test command and get some results:

demo/config/packages/ai.yaml

@@ -59,7 +59,11 @@ ai:
                 class: 'Symfony\AI\Platform\Bridge\OpenAi\Embeddings'
                 name: !php/const Symfony\AI\Platform\Bridge\OpenAi\Embeddings::TEXT_ADA_002
     indexer:
-        default:
+        blog:
+            loader: 'App\Blog\FeedLoader'
+            source: 'https://feeds.feedburner.com/symfony/blog'
+            transformers:
+                - 'Symfony\AI\Store\Document\Transformer\TextTrimTransformer'
             vectorizer: 'ai.vectorizer.openai_embeddings'
             store: 'ai.store.chroma_db.symfonycon'
 
@@ -75,3 +79,5 @@ services:
     Symfony\AI\Agent\Toolbox\Tool\Wikipedia: ~
     Symfony\AI\Agent\Toolbox\Tool\SimilaritySearch:
         $vectorizer: '@ai.vectorizer.openai_embeddings'
+    
+    Symfony\AI\Store\Document\Transformer\TextTrimTransformer: ~

demo/src/Blog/Command/EmbedCommand.php

@@ -11,23 +11,33 @@
 
 namespace App\Blog;
 
+use Symfony\AI\Store\Document\LoaderInterface;
+use Symfony\AI\Store\Document\Metadata;
+use Symfony\AI\Store\Document\TextDocument;
+use Symfony\AI\Store\Exception\InvalidArgumentException;
 use Symfony\Component\DomCrawler\Crawler;
 use Symfony\Component\Uid\Uuid;
 use Symfony\Contracts\HttpClient\HttpClientInterface;
 
-class FeedLoader
+final class FeedLoader implements LoaderInterface
 {
     public function __construct(
         private HttpClientInterface $httpClient,
     ) {
     }
 
     /**
-     * @return Post[]
+     * @param ?string              $source  RSS feed URL
+     * @param array<string, mixed> $options
+     *
+     * @return iterable<TextDocument>
      */
-    public function load(): array
+    public function load(?string $source, array $options = []): iterable
     {
-        $result = $this->httpClient->request('GET', 'https://feeds.feedburner.com/symfony/blog');
+        if (null === $source) {
+            throw new InvalidArgumentException('FeedLoader requires a RSS feed URL as source, null given.');
+        }
+        $result = $this->httpClient->request('GET', $source);
 
         $posts = [];
         $crawler = new Crawler($result->getContent());
@@ -44,6 +54,8 @@ public function load(): array
             );
         });
 
-        return $posts;
+        foreach ($posts as $post) {
+            yield new TextDocument($post->id, $post->toString(), new Metadata($post->toArray()));
+        }
     }
 }

demo/src/Blog/Embedder.php

@@ -0,0 +1,47 @@
+<?php
+
+/*
+ * This file is part of the Symfony package.
+ *
+ * (c) Fabien Potencier <fabien@symfony.com>
+ *
+ * For the full copyright and license information, please view the LICENSE
+ * file that was distributed with this source code.
+ */
+
+use Symfony\AI\Platform\Bridge\OpenAi\Embeddings;
+use Symfony\AI\Platform\Bridge\OpenAi\PlatformFactory;
+use Symfony\AI\Store\Bridge\Local\InMemoryStore;
+use Symfony\AI\Store\Document\Loader\TextFileLoader;
+use Symfony\AI\Store\Document\Transformer\TextReplaceTransformer;
+use Symfony\AI\Store\Document\Transformer\TextSplitTransformer;
+use Symfony\AI\Store\Document\Vectorizer;
+use Symfony\AI\Store\Indexer;
+
+require_once dirname(__DIR__).'/bootstrap.php';
+
+$platform = PlatformFactory::create(env('OPENAI_API_KEY'), http_client());
+$store = new InMemoryStore();
+$vectorizer = new Vectorizer($platform, new Embeddings('text-embedding-3-small'));
+$indexer = new Indexer(
+    loader: new TextFileLoader(),
+    vectorizer: $vectorizer,
+    store: $store,
+    source: [
+        dirname(__DIR__, 2).'/fixtures/movies/gladiator.md',
+        dirname(__DIR__, 2).'/fixtures/movies/inception.md',
+        dirname(__DIR__, 2).'/fixtures/movies/jurassic-park.md',
+    ],
+    transformers: [
+        new TextReplaceTransformer(search: '## Plot', replace: '## Synopsis'),
+        new TextSplitTransformer(chunkSize: 500, overlap: 100),
+    ],
+);
+
+$indexer->index();
+
+$vector = $vectorizer->vectorize('Roman gladiator revenge');
+$results = $store->query($vector);
+foreach ($results as $i => $document) {
+    echo sprintf("%d. %s\n", $i + 1, substr($document->id, 0, 40).'...');
+}

demo/src/Blog/FeedLoader.php

@@ -0,0 +1,58 @@
+<?php
+
+/*
+ * This file is part of the Symfony package.
+ *
+ * (c) Fabien Potencier <fabien@symfony.com>
+ *
+ * For the full copyright and license information, please view the LICENSE
+ * file that was distributed with this source code.
+ */
+
+use Symfony\AI\Platform\Bridge\OpenAi\Embeddings;
+use Symfony\AI\Platform\Bridge\OpenAi\PlatformFactory;
+use Symfony\AI\Store\Bridge\Local\InMemoryStore;
+use Symfony\AI\Store\Document\Loader\InMemoryLoader;
+use Symfony\AI\Store\Document\Metadata;
+use Symfony\AI\Store\Document\TextDocument;
+use Symfony\AI\Store\Document\Transformer\TextSplitTransformer;
+use Symfony\AI\Store\Document\Vectorizer;
+use Symfony\AI\Store\Indexer;
+use Symfony\Component\Uid\Uuid;
+
+require_once dirname(__DIR__).'/bootstrap.php';
+
+$platform = PlatformFactory::create(env('OPENAI_API_KEY'), http_client());
+$store = new InMemoryStore();
+$vectorizer = new Vectorizer($platform, new Embeddings('text-embedding-3-small'));
+
+$documents = [
+    new TextDocument(
+        Uuid::v4(),
+        'Artificial Intelligence is transforming the way we work and live. Machine learning algorithms can now process vast amounts of data and make predictions with remarkable accuracy.',
+        new Metadata(['title' => 'AI Revolution'])
+    ),
+    new TextDocument(
+        Uuid::v4(),
+        'Climate change is one of the most pressing challenges of our time. Renewable energy sources like solar and wind power are becoming increasingly important for a sustainable future.',
+        new Metadata(['title' => 'Climate Action'])
+    ),
+];
+
+$indexer = new Indexer(
+    loader: new InMemoryLoader($documents),
+    vectorizer: $vectorizer,
+    store: $store,
+    source: null,
+    transformers: [
+        new TextSplitTransformer(chunkSize: 100, overlap: 20),
+    ],
+);
+
+$indexer->index();
+
+$vector = $vectorizer->vectorize('machine learning artificial intelligence');
+$results = $store->query($vector);
+foreach ($results as $i => $document) {
+    echo sprintf("%d. %s\n", $i + 1, substr($document->id, 0, 40).'...');
+}

examples/indexer/index-file-loader.php

@@ -0,0 +1,32 @@
+# Gladiator (2000)
+
+**IMDB**: https://www.imdb.com/title/tt0172495/
+
+**Director:** Ridley Scott
+
+## Cast
+
+- **Russell Crowe** as Maximus Decimus Meridius
+- **Joaquin Phoenix** as Emperor Commodus
+- **Connie Nielsen** as Lucilla
+- **Oliver Reed** as Proximo
+- **Derek Jacobi** as Senator Gracchus
+- **Djimon Hounsou** as Juba
+- **Richard Harris** as Marcus Aurelius
+- **Ralf Möller** as Hagen
+- **Tommy Flanagan** as Cicero
+- **David Schofield** as Falco
+
+## Plot
+
+A former Roman General sets out to exact vengeance against the corrupt emperor who murdered his family and sent him into slavery.
+
+**Maximus Decimus Meridius** is a powerful Roman general beloved by the people and the aging Emperor **Marcus Aurelius**. As Marcus Aurelius lies dying, he makes known his wish that Maximus should succeed him and return Rome to the former glory of the Republic rather than the corrupt Empire it has become.
+
+However, Marcus Aurelius's son **Commodus** learns of his father's plan and murders him before he can publicly name Maximus as his successor. Commodus then orders the execution of Maximus and his family. Maximus escapes the execution but arrives at his farm too late to save his wife and son.
+
+Wounded and devastated, Maximus is captured by slave traders and forced to become a gladiator. Under the training of **Proximo**, a former gladiator, Maximus becomes a skilled fighter and eventually makes his way to the **Colosseum** in Rome, where he gains fame and the crowd's favor.
+
+Using his newfound popularity with the people, Maximus seeks to avenge the murder of his family and fulfill his promise to Marcus Aurelius to restore Rome to a republic. The film culminates in a final confrontation between Maximus and Commodus in the arena.
+
+The film explores themes of *honor*, *revenge*, *political corruption*, and the struggle between personal desires and duty to the greater good.

examples/indexer/index-inmemory-loader.php

@@ -0,0 +1,28 @@
+# Inception (2010)
+
+**IMDB**: https://www.imdb.com/title/tt1375666/
+
+**Director:** Christopher Nolan
+
+## Cast
+
+- **Leonardo DiCaprio** as Dom Cobb
+- **Marion Cotillard** as Mal Cobb
+- **Tom Hardy** as Eames
+- **Elliot Page** as Ariadne
+- **Ken Watanabe** as Saito
+- **Dileep Rao** as Yusuf
+- **Cillian Murphy** as Robert Fischer Jr.
+- **Tom Berenger** as Peter Browning
+- **Michael Caine** as Professor Stephen Miles
+- **Lukas Haas** as Nash
+
+## Plot
+
+A skilled thief is given a chance at redemption if he can successfully perform inception, the act of planting an idea in someone's subconscious.
+
+**Dom Cobb** is a skilled thief who specializes in *extraction* - stealing secrets from people's subconscious minds while they dream. This unique skill has made him a valuable player in the world of corporate espionage, but it has also cost him everything he loves. Cobb's rare ability has made him a coveted player in this treacherous new world of corporate espionage, but it has also made him an international fugitive and cost him everything he has ever loved.
+
+Now Cobb is being offered a chance at redemption. One last job could give him his life back but only if he can accomplish the impossible - **inception**. Instead of the perfect heist, Cobb and his team of specialists have to pull off the reverse: their task is not to steal an idea but to plant one. If they succeed, it could be the perfect crime.
+
+The film explores themes of *reality*, *dreams*, *memory*, and the nature of consciousness through multiple layers of dream states, creating a complex narrative structure that challenges both characters and audience to question what is real.