Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ChatAnthropic streaming with tools not working and instead forces non-stream response #6045

Closed
5 tasks done
Moe03 opened this issue Jul 11, 2024 · 10 comments · Fixed by #6179 or #6183
Closed
5 tasks done

ChatAnthropic streaming with tools not working and instead forces non-stream response #6045

Moe03 opened this issue Jul 11, 2024 · 10 comments · Fixed by #6179 or #6183
Labels
auto:bug Related to a bug, vulnerability, unexpected error with an existing feature

Comments

@Moe03
Copy link

Moe03 commented Jul 11, 2024

Checked other resources

  • I added a very descriptive title to this issue.
  • I searched the LangChain.js documentation with the integrated search.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangChain.js rather than my code.
  • The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

The following is a more advanced example but it is to showcase the full problem, I've also implemented a fix for it in a pr.

import { HumanMessage, SystemMessage } from "@langchain/core/messages";
import { ChatAnthropic } from "@langchain/anthropic";
import { DynamicStructuredTool } from "@langchain/core/tools";
import { z } from "zod";
import { ChatOpenAI } from "@langchain/openai";
import { AgentExecutor, createToolCallingAgent } from "langchain/agents";
import {
    ChatPromptTemplate,
    MessagesPlaceholder
} from "@langchain/core/prompts";

async function main() {

    const defaultTools = [
        new DynamicStructuredTool({
            name: "GetWeather",
            description: "Return the weather.",
            schema: z.object({
                query: z.string().describe("The city to return weather for,."),
            }),
            func: async (input) => {
                console.log(`STARTING TOOL: GetWeather`)
                // just an example
                return 'sunny';
            },
        }),
    ]

    const chatPrompt = ChatPromptTemplate.fromMessages([
        new MessagesPlaceholder("messages"),
        new MessagesPlaceholder("agent_scratchpad"),
    ]);

    const messagesHistory = [
        new SystemMessage("you are a helpful assistant!"),
        new HumanMessage("Whats the weather like in Cairo?")
    ]

    let llm;

    if (false) {
        llm = new ChatOpenAI({
            model: 'gpt-4o',
            apiKey: OAI_API_KEY,
            // streaming: true,
        });
    } else {
        llm = new ChatAnthropic({
            apiKey: ANTHROPIC_API_KEY,
            model: "claude-3-5-sonnet-20240620",
        });
    }

    const agent = await createToolCallingAgent({
        llm,
        tools: defaultTools,
        prompt: chatPrompt,
    });

    const agentExec = new AgentExecutor({
        agent,
        tools: defaultTools,
    }).withConfig({ runName: "Agent" });

    let llmStream = await agentExec.streamEvents(
        {
            messages: messagesHistory,
        },
        {
            version: "v1"
        }
    );

    for await (const event of llmStream) {
        const eventType = event.event;
        if (eventType === "on_chain_start") {
            // Was assigned when creating the agent with `.withConfig({"runName": "Agent"})` above
            if (event.name === "Agent") {
                console.log("\n-----");
                console.log(
                    `Starting agent: ${event.name} with input: ${JSON.stringify(
                        event.data.input
                    )}`
                );
            }
        } else if (eventType === "on_chain_end") {
            // Was assigned when creating the agent with `.withConfig({"runName": "Agent"})` above
            if (event?.name === "Agent") {
                console.log("\n-----");
                console.log(`Finished agent: ${event?.name}\n`);
                console.log(`Agent output was: ${event?.data?.output}`);
                // if(modelToUse.includes(`claude`)){

                // }
                console.log("\n-----");
            }
        } else if (eventType === "on_llm_stream" || eventType === "on_chat_model_stream" || (eventType === "on_chain_stream")) {
            // console.log(`got llm stream thing: `, )
            console.log("\n-----");
            const content = event?.data?.chunk?.output || event.data?.chunk?.message?.content || event.data?.chunk?.content;
            if (content !== undefined && content !== "") {
                console.log(`| ${content}`);
            }
        } else if (eventType === "on_tool_start") {
            console.log("\n-----");
            console.log(`debug message`)
            console.log(`EVENT: `, event)
            const debugMessage = `Starting tool: ${event.name} with inputs: ${event.data.input}`
            console.log(
                debugMessage
            );
        } else if (eventType === "on_tool_end") {
            console.log("\n-----");
            console.log(`Finished tool: ${event.name}\n`);
            console.log(`Tool output was: ${event.data.output}`);
            const debugMessage = `Finished tool: ${event.name}\nTool output was: ${event.data.output}`
            console.log(
                debugMessage
            );
            console.log("\n-----");
        }
    }
}
main()

We basically needed tools to stream properly on @langchain/anthropic package to enable streaming events across any other langchain method like the agentExecutor I'm using here.

Error Message and Stack Trace (if applicable)

There are no errors but it doesn't stream when it should.

Description

We basically needed tools to stream properly on @langchain/anthropic package to enable streaming events across any other langchain method like the agentExecutor.

Currenlty it doesn't stream and instead wait for the whole tool + response to finish then outputs the response which takes a really long time and is bad UX in realtime apps.

System Info

Node version 18.17.1
yarn version 1.22.22
platform windows

yarn info langchain:
(result was too big)

      },
      import: './chains/retrieval.js',
      require: './chains/retrieval.cjs'
    },
    './chains/sql_db': {
      types: {
        import: './chains/sql_db.d.ts',
        require: './chains/sql_db.d.cts',
        default: './chains/sql_db.d.ts'
      },
      import: './chains/sql_db.js',
      require: './chains/sql_db.cjs'
    },
    './chains/graph_qa/cypher': {
      types: {
        import: './chains/graph_qa/cypher.d.ts',
        require: './chains/graph_qa/cypher.d.cts',
        default: './chains/graph_qa/cypher.d.ts'
      },
      import: './chains/graph_qa/cypher.js',
      require: './chains/graph_qa/cypher.cjs'
    },
    './embeddings/cache_backed': {
      types: {
        import: './embeddings/cache_backed.d.ts',
        require: './embeddings/cache_backed.d.cts',
        default: './embeddings/cache_backed.d.ts'
      },
      import: './embeddings/cache_backed.js',
      require: './embeddings/cache_backed.cjs'
    },
    './embeddings/fake': {
      types: {
        import: './embeddings/fake.d.ts',
        require: './embeddings/fake.d.cts',
        default: './embeddings/fake.d.ts'
      },
      import: './embeddings/fake.js',
      require: './embeddings/fake.cjs'
    },
    './vectorstores/memory': {
      types: {
        import: './vectorstores/memory.d.ts',
        require: './vectorstores/memory.d.cts',
        default: './vectorstores/memory.d.ts'
      },
      import: './vectorstores/memory.js',
      require: './vectorstores/memory.cjs'
    },
    './text_splitter': {
      types: {
        import: './text_splitter.d.ts',
        require: './text_splitter.d.cts',
        default: './text_splitter.d.ts'
      },
      import: './text_splitter.js',
      require: './text_splitter.cjs'
    },
    './memory': {
      types: {
        import: './memory.d.ts',
        require: './memory.d.cts',
        default: './memory.d.ts'
      },
      import: './memory.js',
      require: './memory.cjs'
    },
    './memory/index': {
      types: {
        import: './memory/index.d.ts',
        require: './memory/index.d.cts',
        default: './memory/index.d.ts'
      },
      import: './memory/index.js',
      require: './memory/index.cjs'
    },
    './memory/chat_memory': {
      types: {
        import: './memory/chat_memory.d.ts',
        require: './memory/chat_memory.d.cts',
        default: './memory/chat_memory.d.ts'
      },
      import: './memory/chat_memory.js',
      require: './memory/chat_memory.cjs'
    },
    './document': {
      types: {
        import: './document.d.ts',
        require: './document.d.cts',
        default: './document.d.ts'
      },
      import: './document.js',
      require: './document.cjs'
    },
    './document_loaders/base': {
      types: {
        import: './document_loaders/base.d.ts',
        require: './document_loaders/base.d.cts',
        default: './document_loaders/base.d.ts'
      },
      import: './document_loaders/base.js',
      require: './document_loaders/base.cjs'
    },
    './document_loaders/web/apify_dataset': {
      types: {
        import: './document_loaders/web/apify_dataset.d.ts',
        require: './document_loaders/web/apify_dataset.d.cts',
        default: './document_loaders/web/apify_dataset.d.ts'
      },
      import: './document_loaders/web/apify_dataset.js',
      require: './document_loaders/web/apify_dataset.cjs'
    },
    './document_loaders/web/assemblyai': {
      types: {
        import: './document_loaders/web/assemblyai.d.ts',
        require: './document_loaders/web/assemblyai.d.cts',
        default: './document_loaders/web/assemblyai.d.ts'
      },
      import: './document_loaders/web/assemblyai.js',
      require: './document_loaders/web/assemblyai.cjs'
    },
    './document_loaders/web/azure_blob_storage_container': {
      types: {
        import: './document_loaders/web/azure_blob_storage_container.d.ts',
        require: './document_loaders/web/azure_blob_storage_container.d.cts',
        default: './document_loaders/web/azure_blob_storage_container.d.ts'
      },
      import: './document_loaders/web/azure_blob_storage_container.js',
      require: './document_loaders/web/azure_blob_storage_container.cjs'
    },
    './document_loaders/web/azure_blob_storage_file': {
      types: {
        import: './document_loaders/web/azure_blob_storage_file.d.ts',
        require: './document_loaders/web/azure_blob_storage_file.d.cts',
        default: './document_loaders/web/azure_blob_storage_file.d.ts'
      },
      import: './document_loaders/web/azure_blob_storage_file.js',
      require: './document_loaders/web/azure_blob_storage_file.cjs'
    },
    './document_loaders/web/browserbase': {
      types: {
        import: './document_loaders/web/browserbase.d.ts',
        require: './document_loaders/web/browserbase.d.cts',
        default: './document_loaders/web/browserbase.d.ts'
      },
      import: './document_loaders/web/browserbase.js',
      require: './document_loaders/web/browserbase.cjs'
    },
    './document_loaders/web/cheerio': {
      types: {
        import: './document_loaders/web/cheerio.d.ts',
        require: './document_loaders/web/cheerio.d.cts',
        default: './document_loaders/web/cheerio.d.ts'
      },
      import: './document_loaders/web/cheerio.js',
      require: './document_loaders/web/cheerio.cjs'
    },
    './document_loaders/web/puppeteer': {
      types: {
        import: './document_loaders/web/puppeteer.d.ts',
        require: './document_loaders/web/puppeteer.d.cts',
        default: './document_loaders/web/puppeteer.d.ts'
      },
      import: './document_loaders/web/puppeteer.js',
      require: './document_loaders/web/puppeteer.cjs'
    },
    './document_loaders/web/playwright': {
      types: {
        import: './document_loaders/web/playwright.d.ts',
        require: './document_loaders/web/playwright.d.cts',
        default: './document_loaders/web/playwright.d.ts'
      },
      import: './document_loaders/web/playwright.js',
      require: './document_loaders/web/playwright.cjs'
    },
    './document_loaders/web/college_confidential': {
      types: {
        import: './document_loaders/web/college_confidential.d.ts',
        require: './document_loaders/web/college_confidential.d.cts',
        default: './document_loaders/web/college_confidential.d.ts'
      },
      import: './document_loaders/web/college_confidential.js',
      require: './document_loaders/web/college_confidential.cjs'
    },
    './document_loaders/web/gitbook': {
      types: {
        import: './document_loaders/web/gitbook.d.ts',
        require: './document_loaders/web/gitbook.d.cts',
        default: './document_loaders/web/gitbook.d.ts'
      },
      import: './document_loaders/web/gitbook.js',
      require: './document_loaders/web/gitbook.cjs'
    },
    './document_loaders/web/hn': {
      types: {
        import: './document_loaders/web/hn.d.ts',
        require: './document_loaders/web/hn.d.cts',
        default: './document_loaders/web/hn.d.ts'
      },
      import: './document_loaders/web/hn.js',
      require: './document_loaders/web/hn.cjs'
    },
    './document_loaders/web/imsdb': {
      types: {
        import: './document_loaders/web/imsdb.d.ts',
        require: './document_loaders/web/imsdb.d.cts',
        default: './document_loaders/web/imsdb.d.ts'
      },
      import: './document_loaders/web/imsdb.js',
      require: './document_loaders/web/imsdb.cjs'
    },
    './document_loaders/web/figma': {
      types: {
        import: './document_loaders/web/figma.d.ts',
        require: './document_loaders/web/figma.d.cts',
        default: './document_loaders/web/figma.d.ts'
      },
      import: './document_loaders/web/figma.js',
      require: './document_loaders/web/figma.cjs'
    },
    './document_loaders/web/firecrawl': {
      types: {
        import: './document_loaders/web/firecrawl.d.ts',
        require: './document_loaders/web/firecrawl.d.cts',
        default: './document_loaders/web/firecrawl.d.ts'
      },
      import: './document_loaders/web/firecrawl.js',
      require: './document_loaders/web/firecrawl.cjs'
    },
    './document_loaders/web/github': {
      types: {
        import: './document_loaders/web/github.d.ts',
        require: './document_loaders/web/github.d.cts',
        default: './document_loaders/web/github.d.ts'
      },
      import: './document_loaders/web/github.js',
      require: './document_loaders/web/github.cjs'
    },
    './document_loaders/web/notiondb': {
      types: {
        import: './document_loaders/web/notiondb.d.ts',
        require: './document_loaders/web/notiondb.d.cts',
        default: './document_loaders/web/notiondb.d.ts'
      },
      import: './document_loaders/web/notiondb.js',
      require: './document_loaders/web/notiondb.cjs'
    },
    './document_loaders/web/notionapi': {
      types: {
        import: './document_loaders/web/notionapi.d.ts',
        require: './document_loaders/web/notionapi.d.cts',
        default: './document_loaders/web/notionapi.d.ts'
      },
      import: './document_loaders/web/notionapi.js',
      require: './document_loaders/web/notionapi.cjs'
    },
    './document_loaders/web/pdf': {
      types: {
        import: './document_loaders/web/pdf.d.ts',
        require: './document_loaders/web/pdf.d.cts',
        default: './document_loaders/web/pdf.d.ts'
      },
      import: './document_loaders/web/pdf.js',
      require: './document_loaders/web/pdf.cjs'
    },
    './document_loaders/web/recursive_url': {
      types: {
        import: './document_loaders/web/recursive_url.d.ts',
        require: './document_loaders/web/recursive_url.d.cts',
        default: './document_loaders/web/recursive_url.d.ts'
      },
      import: './document_loaders/web/recursive_url.js',
      require: './document_loaders/web/recursive_url.cjs'
    },
    './document_loaders/web/s3': {
      types: {
        import: './document_loaders/web/s3.d.ts',
        require: './document_loaders/web/s3.d.cts',
        default: './document_loaders/web/s3.d.ts'
      },
      import: './document_loaders/web/s3.js',
      require: './document_loaders/web/s3.cjs'
    },
    './document_loaders/web/sitemap': {
      types: {
        import: './document_loaders/web/sitemap.d.ts',
        require: './document_loaders/web/sitemap.d.cts',
        default: './document_loaders/web/sitemap.d.ts'
      },
      import: './document_loaders/web/sitemap.js',
      require: './document_loaders/web/sitemap.cjs'
    },
    './document_loaders/web/sonix_audio': {
      types: {
        import: './document_loaders/web/sonix_audio.d.ts',
        require: './document_loaders/web/sonix_audio.d.cts',
        default: './document_loaders/web/sonix_audio.d.ts'
      },
      import: './document_loaders/web/sonix_audio.js',
      require: './document_loaders/web/sonix_audio.cjs'
    },
    './document_loaders/web/confluence': {
      types: {
        import: './document_loaders/web/confluence.d.ts',
        require: './document_loaders/web/confluence.d.cts',
        default: './document_loaders/web/confluence.d.ts'
      },
      import: './document_loaders/web/confluence.js',
      require: './document_loaders/web/confluence.cjs'
    },
    './document_loaders/web/couchbase': {
      types: {
        import: './document_loaders/web/couchbase.d.ts',
        require: './document_loaders/web/couchbase.d.cts',
        default: './document_loaders/web/couchbase.d.ts'
      },
      import: './document_loaders/web/couchbase.js',
      require: './document_loaders/web/couchbase.cjs'
    },
    './document_loaders/web/searchapi': {
      types: {
        import: './document_loaders/web/searchapi.d.ts',
        require: './document_loaders/web/searchapi.d.cts',
        default: './document_loaders/web/searchapi.d.ts'
      },
      import: './document_loaders/web/searchapi.js',
      require: './document_loaders/web/searchapi.cjs'
    },
    './document_loaders/web/serpapi': {
      types: {
        import: './document_loaders/web/serpapi.d.ts',
        require: './document_loaders/web/serpapi.d.cts',
        default: './document_loaders/web/serpapi.d.ts'
      },
      import: './document_loaders/web/serpapi.js',
      require: './document_loaders/web/serpapi.cjs'
    },
    './document_loaders/web/sort_xyz_blockchain': {
      types: {
        import: './document_loaders/web/sort_xyz_blockchain.d.ts',
        require: './document_loaders/web/sort_xyz_blockchain.d.cts',
        default: './document_loaders/web/sort_xyz_blockchain.d.ts'
      },
      import: './document_loaders/web/sort_xyz_blockchain.js',
      require: './document_loaders/web/sort_xyz_blockchain.cjs'
    },
    './document_loaders/web/youtube': {
      types: {
        import: './document_loaders/web/youtube.d.ts',
        require: './document_loaders/web/youtube.d.cts',
        default: './document_loaders/web/youtube.d.ts'
      },
      import: './document_loaders/web/youtube.js',
      require: './document_loaders/web/youtube.cjs'
    },
    './document_loaders/fs/directory': {
      types: {
        import: './document_loaders/fs/directory.d.ts',
        require: './document_loaders/fs/directory.d.cts',
        default: './document_loaders/fs/directory.d.ts'
      },
      import: './document_loaders/fs/directory.js',
      require: './document_loaders/fs/directory.cjs'
    },
    './document_loaders/fs/multi_file': {
      types: {
        import: './document_loaders/fs/multi_file.d.ts',
        require: './document_loaders/fs/multi_file.d.cts',
        default: './document_loaders/fs/multi_file.d.ts'
      },
      import: './document_loaders/fs/multi_file.js',
      require: './document_loaders/fs/multi_file.cjs'
    },
    './document_loaders/fs/buffer': {
      types: {
        import: './document_loaders/fs/buffer.d.ts',
        require: './document_loaders/fs/buffer.d.cts',
        default: './document_loaders/fs/buffer.d.ts'
      },
      import: './document_loaders/fs/buffer.js',
      require: './document_loaders/fs/buffer.cjs'
    },
    './document_loaders/fs/chatgpt': {
      types: {
        import: './document_loaders/fs/chatgpt.d.ts',
        require: './document_loaders/fs/chatgpt.d.cts',
        default: './document_loaders/fs/chatgpt.d.ts'
      },
      import: './document_loaders/fs/chatgpt.js',
      require: './document_loaders/fs/chatgpt.cjs'
    },
    './document_loaders/fs/text': {
      types: {
        import: './document_loaders/fs/text.d.ts',
        require: './document_loaders/fs/text.d.cts',
        default: './document_loaders/fs/text.d.ts'
      },
      import: './document_loaders/fs/text.js',
      require: './document_loaders/fs/text.cjs'
    },
    './document_loaders/fs/json': {
      types: {
        import: './document_loaders/fs/json.d.ts',
        require: './document_loaders/fs/json.d.cts',
        default: './document_loaders/fs/json.d.ts'
      },
      import: './document_loaders/fs/json.js',
      require: './document_loaders/fs/json.cjs'
    },
    './document_loaders/fs/srt': {
      types: {
        import: './document_loaders/fs/srt.d.ts',
        require: './document_loaders/fs/srt.d.cts',
        default: './document_loaders/fs/srt.d.ts'
      },
      import: './document_loaders/fs/srt.js',
      require: './document_loaders/fs/srt.cjs'
    },
    './document_loaders/fs/pdf': {
      types: {
        import: './document_loaders/fs/pdf.d.ts',
        require: './document_loaders/fs/pdf.d.cts',
        default: './document_loaders/fs/pdf.d.ts'
      },
      import: './document_loaders/fs/pdf.js',
      require: './document_loaders/fs/pdf.cjs'
    },
    './document_loaders/fs/docx': {
      types: {
        import: './document_loaders/fs/docx.d.ts',
        require: './document_loaders/fs/docx.d.cts',
        default: './document_loaders/fs/docx.d.ts'
      },
      import: './document_loaders/fs/docx.js',
      require: './document_loaders/fs/docx.cjs'
    },
    './document_loaders/fs/epub': {
      types: {
        import: './document_loaders/fs/epub.d.ts',
        require: './document_loaders/fs/epub.d.cts',
        default: './document_loaders/fs/epub.d.ts'
      },
      import: './document_loaders/fs/epub.js',
      require: './document_loaders/fs/epub.cjs'
    },
    './document_loaders/fs/csv': {
      types: {
        import: './document_loaders/fs/csv.d.ts',
        require: './document_loaders/fs/csv.d.cts',
        default: './document_loaders/fs/csv.d.ts'
      },
      import: './document_loaders/fs/csv.js',
      require: './document_loaders/fs/csv.cjs'
    },
    './document_loaders/fs/notion': {
      types: {
        import: './document_loaders/fs/notion.d.ts',
        require: './document_loaders/fs/notion.d.cts',
        default: './document_loaders/fs/notion.d.ts'
      },
      import: './document_loaders/fs/notion.js',
      require: './document_loaders/fs/notion.cjs'
    },
    './document_loaders/fs/obsidian': {
      types: {
        import: './document_loaders/fs/obsidian.d.ts',
        require: './document_loaders/fs/obsidian.d.cts',
        default: './document_loaders/fs/obsidian.d.ts'
      },
      import: './document_loaders/fs/obsidian.js',
      require: './document_loaders/fs/obsidian.cjs'
    },
    './document_loaders/fs/unstructured': {
      types: {
        import: './document_loaders/fs/unstructured.d.ts',
        require: './document_loaders/fs/unstructured.d.cts',
        default: './document_loaders/fs/unstructured.d.ts'
      },
      import: './document_loaders/fs/unstructured.js',
      require: './document_loaders/fs/unstructured.cjs'
    },
    './document_loaders/fs/openai_whisper_audio': {
      types: {
        import: './document_loaders/fs/openai_whisper_audio.d.ts',
        require: './document_loaders/fs/openai_whisper_audio.d.cts',
        default: './document_loaders/fs/openai_whisper_audio.d.ts'
      },
      import: './document_loaders/fs/openai_whisper_audio.js',
      require: './document_loaders/fs/openai_whisper_audio.cjs'
    },
    './document_loaders/fs/pptx': {
      types: {
        import: './document_loaders/fs/pptx.d.ts',
        require: './document_loaders/fs/pptx.d.cts',
        default: './document_loaders/fs/pptx.d.ts'
      },
      import: './document_loaders/fs/pptx.js',
      require: './document_loaders/fs/pptx.cjs'
    },
    './document_transformers/openai_functions': {
      types: {
        import: './document_transformers/openai_functions.d.ts',
        require: './document_transformers/openai_functions.d.cts',
        default: './document_transformers/openai_functions.d.ts'
      },
      import: './document_transformers/openai_functions.js',
      require: './document_transformers/openai_functions.cjs'
    },
    './sql_db': {
      types: {
        import: './sql_db.d.ts',
        require: './sql_db.d.cts',
        default: './sql_db.d.ts'
      },
      import: './sql_db.js',
      require: './sql_db.cjs'
    },
    './callbacks': {
      types: {
        import: './callbacks.d.ts',
        require: './callbacks.d.cts',
        default: './callbacks.d.ts'
      },
      import: './callbacks.js',
      require: './callbacks.cjs'
    },
    './output_parsers': {
      types: {
        import: './output_parsers.d.ts',
        require: './output_parsers.d.cts',
        default: './output_parsers.d.ts'
      },
      import: './output_parsers.js',
      require: './output_parsers.cjs'
    },
    './output_parsers/expression': {
      types: {
        import: './output_parsers/expression.d.ts',
        require: './output_parsers/expression.d.cts',
        default: './output_parsers/expression.d.ts'
      },
      import: './output_parsers/expression.js',
      require: './output_parsers/expression.cjs'
    },
    './retrievers/contextual_compression': {
      types: {
        import: './retrievers/contextual_compression.d.ts',
        require: './retrievers/contextual_compression.d.cts',
        default: './retrievers/contextual_compression.d.ts'
      },
      import: './retrievers/contextual_compression.js',
      require: './retrievers/contextual_compression.cjs'
    },
    './retrievers/document_compressors': {
      types: {
        import: './retrievers/document_compressors.d.ts',
        require: './retrievers/document_compressors.d.cts',
        default: './retrievers/document_compressors.d.ts'
      },
      import: './retrievers/document_compressors.js',
      require: './retrievers/document_compressors.cjs'
    },
    './retrievers/ensemble': {
      types: {
        import: './retrievers/ensemble.d.ts',
        require: './retrievers/ensemble.d.cts',
        default: './retrievers/ensemble.d.ts'
      },
      import: './retrievers/ensemble.js',
      require: './retrievers/ensemble.cjs'
    },
    './retrievers/multi_query': {
      types: {
        import: './retrievers/multi_query.d.ts',
        require: './retrievers/multi_query.d.cts',
        default: './retrievers/multi_query.d.ts'
      },
      import: './retrievers/multi_query.js',
      require: './retrievers/multi_query.cjs'
    },
    './retrievers/multi_vector': {
      types: {
        import: './retrievers/multi_vector.d.ts',
        require: './retrievers/multi_vector.d.cts',
        default: './retrievers/multi_vector.d.ts'
      },
      import: './retrievers/multi_vector.js',
      require: './retrievers/multi_vector.cjs'
    },
    './retrievers/parent_document': {
      types: {
        import: './retrievers/parent_document.d.ts',
        require: './retrievers/parent_document.d.cts',
        default: './retrievers/parent_document.d.ts'
      },
      import: './retrievers/parent_document.js',
      require: './retrievers/parent_document.cjs'
    },
    './retrievers/time_weighted': {
      types: {
        import: './retrievers/time_weighted.d.ts',
        require: './retrievers/time_weighted.d.cts',
        default: './retrievers/time_weighted.d.ts'
      },
      import: './retrievers/time_weighted.js',
      require: './retrievers/time_weighted.cjs'
    },
    './retrievers/document_compressors/chain_extract': {
      types: {
        import: './retrievers/document_compressors/chain_extract.d.ts',
        require: './retrievers/document_compressors/chain_extract.d.cts',
        default: './retrievers/document_compressors/chain_extract.d.ts'
      },
      import: './retrievers/document_compressors/chain_extract.js',
      require: './retrievers/document_compressors/chain_extract.cjs'
    },
    './retrievers/document_compressors/embeddings_filter': {
      types: {
        import: './retrievers/document_compressors/embeddings_filter.d.ts',
        require: './retrievers/document_compressors/embeddings_filter.d.cts',
        default: './retrievers/document_compressors/embeddings_filter.d.ts'
      },
      import: './retrievers/document_compressors/embeddings_filter.js',
      require: './retrievers/document_compressors/embeddings_filter.cjs'
    },
    './retrievers/hyde': {
      types: {
        import: './retrievers/hyde.d.ts',
        require: './retrievers/hyde.d.cts',
        default: './retrievers/hyde.d.ts'
      },
      import: './retrievers/hyde.js',
      require: './retrievers/hyde.cjs'
    },
    './retrievers/score_threshold': {
      types: {
        import: './retrievers/score_threshold.d.ts',
        require: './retrievers/score_threshold.d.cts',
        default: './retrievers/score_threshold.d.ts'
      },
      import: './retrievers/score_threshold.js',
      require: './retrievers/score_threshold.cjs'
    },
    './retrievers/self_query': {
      types: {
        import: './retrievers/self_query.d.ts',
        require: './retrievers/self_query.d.cts',
        default: './retrievers/self_query.d.ts'
      },
      import: './retrievers/self_query.js',
      require: './retrievers/self_query.cjs'
    },
    './retrievers/self_query/chroma': {
      types: {
        import: './retrievers/self_query/chroma.d.ts',
        require: './retrievers/self_query/chroma.d.cts',
        default: './retrievers/self_query/chroma.d.ts'
      },
      import: './retrievers/self_query/chroma.js',
      require: './retrievers/self_query/chroma.cjs'
    },
    './retrievers/self_query/functional': {
      types: {
        import: './retrievers/self_query/functional.d.ts',
        require: './retrievers/self_query/functional.d.cts',
        default: './retrievers/self_query/functional.d.ts'
      },
      import: './retrievers/self_query/functional.js',
      require: './retrievers/self_query/functional.cjs'
    },
    './retrievers/self_query/pinecone': {
      types: {
        import: './retrievers/self_query/pinecone.d.ts',
        require: './retrievers/self_query/pinecone.d.cts',
        default: './retrievers/self_query/pinecone.d.ts'
      },
      import: './retrievers/self_query/pinecone.js',
      require: './retrievers/self_query/pinecone.cjs'
    },
    './retrievers/self_query/supabase': {
      types: {
        import: './retrievers/self_query/supabase.d.ts',
        require: './retrievers/self_query/supabase.d.cts',
        default: './retrievers/self_query/supabase.d.ts'
      },
      import: './retrievers/self_query/supabase.js',
      require: './retrievers/self_query/supabase.cjs'
    },
    './retrievers/self_query/weaviate': {
      types: {
        import: './retrievers/self_query/weaviate.d.ts',
        require: './retrievers/self_query/weaviate.d.cts',
        default: './retrievers/self_query/weaviate.d.ts'
      },
      import: './retrievers/self_query/weaviate.js',
      require: './retrievers/self_query/weaviate.cjs'
    },
    './retrievers/self_query/vectara': {
      types: {
        import: './retrievers/self_query/vectara.d.ts',
        require: './retrievers/self_query/vectara.d.cts',
        default: './retrievers/self_query/vectara.d.ts'
      },
      import: './retrievers/self_query/vectara.js',
      require: './retrievers/self_query/vectara.cjs'
    },
    './retrievers/matryoshka_retriever': {
      types: {
        import: './retrievers/matryoshka_retriever.d.ts',
        require: './retrievers/matryoshka_retriever.d.cts',
        default: './retrievers/matryoshka_retriever.d.ts'
      },
      import: './retrievers/matryoshka_retriever.js',
      require: './retrievers/matryoshka_retriever.cjs'
    },
    './cache/file_system': {
      types: {
        import: './cache/file_system.d.ts',
        require: './cache/file_system.d.cts',
        default: './cache/file_system.d.ts'
      },
      import: './cache/file_system.js',
      require: './cache/file_system.cjs'
    },
    './stores/doc/base': {
      types: {
        import: './stores/doc/base.d.ts',
        require: './stores/doc/base.d.cts',
        default: './stores/doc/base.d.ts'
      },
      import: './stores/doc/base.js',
      require: './stores/doc/base.cjs'
    },
    './stores/doc/in_memory': {
      types: {
        import: './stores/doc/in_memory.d.ts',
        require: './stores/doc/in_memory.d.cts',
        default: './stores/doc/in_memory.d.ts'
      },
      import: './stores/doc/in_memory.js',
      require: './stores/doc/in_memory.cjs'
    },
    './stores/file/in_memory': {
      types: {
        import: './stores/file/in_memory.d.ts',
        require: './stores/file/in_memory.d.cts',
        default: './stores/file/in_memory.d.ts'
      },
      import: './stores/file/in_memory.js',
      require: './stores/file/in_memory.cjs'
    },
    './stores/file/node': {
      types: {
        import: './stores/file/node.d.ts',
        require: './stores/file/node.d.cts',
        default: './stores/file/node.d.ts'
      },
      import: './stores/file/node.js',
      require: './stores/file/node.cjs'
    },
    './stores/message/in_memory': {
      types: {
        import: './stores/message/in_memory.d.ts',
        require: './stores/message/in_memory.d.cts',
        default: './stores/message/in_memory.d.ts'
      },
      import: './stores/message/in_memory.js',
      require: './stores/message/in_memory.cjs'
    },
    './storage/encoder_backed': {
      types: {
        import: './storage/encoder_backed.d.ts',
        require: './storage/encoder_backed.d.cts',
        default: './storage/encoder_backed.d.ts'
      },
      import: './storage/encoder_backed.js',
      require: './storage/encoder_backed.cjs'
    },
    './storage/in_memory': {
      types: {
        import: './storage/in_memory.d.ts',
        require: './storage/in_memory.d.cts',
        default: './storage/in_memory.d.ts'
      },
      import: './storage/in_memory.js',
      require: './storage/in_memory.cjs'
    },
    './storage/file_system': {
      types: {
        import: './storage/file_system.d.ts',
        require: './storage/file_system.d.cts',
        default: './storage/file_system.d.ts'
      },
      import: './storage/file_system.js',
      require: './storage/file_system.cjs'
    },
    './hub': {
      types: {
        import: './hub.d.ts',
        require: './hub.d.cts',
        default: './hub.d.ts'
      },
      import: './hub.js',
      require: './hub.cjs'
    },
    './util/document': {
      types: {
        import: './util/document.d.ts',
        require: './util/document.d.cts',
        default: './util/document.d.ts'
      },
      import: './util/document.js',
      require: './util/document.cjs'
    },
    './util/math': {
      types: {
        import: './util/math.d.ts',
        require: './util/math.d.cts',
        default: './util/math.d.ts'
      },
      import: './util/math.js',
      require: './util/math.cjs'
    },
    './util/time': {
      types: {
        import: './util/time.d.ts',
        require: './util/time.d.cts',
        default: './util/time.d.ts'
      },
      import: './util/time.js',
      require: './util/time.cjs'
    },
    './experimental/autogpt': {
      types: {
        import: './experimental/autogpt.d.ts',
        require: './experimental/autogpt.d.cts',
        default: './experimental/autogpt.d.ts'
      },
      import: './experimental/autogpt.js',
      require: './experimental/autogpt.cjs'
    },
    './experimental/openai_assistant': {
      types: {
        import: './experimental/openai_assistant.d.ts',
        require: './experimental/openai_assistant.d.cts',
        default: './experimental/openai_assistant.d.ts'
      },
      import: './experimental/openai_assistant.js',
      require: './experimental/openai_assistant.cjs'
    },
    './experimental/openai_files': {
      types: {
        import: './experimental/openai_files.d.ts',
        require: './experimental/openai_files.d.cts',
        default: './experimental/openai_files.d.ts'
      },
      import: './experimental/openai_files.js',
      require: './experimental/openai_files.cjs'
    },
    './experimental/babyagi': {
      types: {
        import: './experimental/babyagi.d.ts',
        require: './experimental/babyagi.d.cts',
        default: './experimental/babyagi.d.ts'
      },
      import: './experimental/babyagi.js',
      require: './experimental/babyagi.cjs'
    },
    './experimental/generative_agents': {
      types: {
        import: './experimental/generative_agents.d.ts',
        require: './experimental/generative_agents.d.cts',
        default: './experimental/generative_agents.d.ts'
      },
      import: './experimental/generative_agents.js',
      require: './experimental/generative_agents.cjs'
    },
    './experimental/plan_and_execute': {
      types: {
        import: './experimental/plan_and_execute.d.ts',
        require: './experimental/plan_and_execute.d.cts',
        default: './experimental/plan_and_execute.d.ts'
      },
      import: './experimental/plan_and_execute.js',
      require: './experimental/plan_and_execute.cjs'
    },
    './experimental/chains/violation_of_expectations': {
      types: {
        import: './experimental/chains/violation_of_expectations.d.ts',
        require: './experimental/chains/violation_of_expectations.d.cts',
        default: './experimental/chains/violation_of_expectations.d.ts'
      },
      import: './experimental/chains/violation_of_expectations.js',
      require: './experimental/chains/violation_of_expectations.cjs'
    },
    './experimental/masking': {
      types: {
        import: './experimental/masking.d.ts',
        require: './experimental/masking.d.cts',
        default: './experimental/masking.d.ts'
      },
      import: './experimental/masking.js',
      require: './experimental/masking.cjs'
    },
    './experimental/prompts/custom_format': {
      types: {
        import: './experimental/prompts/custom_format.d.ts',
        require: './experimental/prompts/custom_format.d.cts',
        default: './experimental/prompts/custom_format.d.ts'
      },
      import: './experimental/prompts/custom_format.js',
      require: './experimental/prompts/custom_format.cjs'
    },
    './experimental/prompts/handlebars': {
      types: {
        import: './experimental/prompts/handlebars.d.ts',
        require: './experimental/prompts/handlebars.d.cts',
        default: './experimental/prompts/handlebars.d.ts'
      },
      import: './experimental/prompts/handlebars.js',
      require: './experimental/prompts/handlebars.cjs'
    },
    './experimental/tools/pyinterpreter': {
      types: {
        import: './experimental/tools/pyinterpreter.d.ts',
        require: './experimental/tools/pyinterpreter.d.cts',
        default: './experimental/tools/pyinterpreter.d.ts'
      },
      import: './experimental/tools/pyinterpreter.js',
      require: './experimental/tools/pyinterpreter.cjs'
    },
    './evaluation': {
      types: {
        import: './evaluation.d.ts',
        require: './evaluation.d.cts',
        default: './evaluation.d.ts'
      },
      import: './evaluation.js',
      require: './evaluation.cjs'
    },
    './smith': {
      types: {
        import: './smith.d.ts',
        require: './smith.d.cts',
        default: './smith.d.ts'
      },
      import: './smith.js',
      require: './smith.cjs'
    },
    './runnables/remote': {
      types: {
        import: './runnables/remote.d.ts',
        require: './runnables/remote.d.cts',
        default: './runnables/remote.d.ts'
      },
      import: './runnables/remote.js',
      require: './runnables/remote.cjs'
    },
    './indexes': {
      types: {
        import: './indexes.d.ts',
        require: './indexes.d.cts',
        default: './indexes.d.ts'
      },
      import: './indexes.js',
      require: './indexes.cjs'
    },
    './schema/query_constructor': {
      types: {
        import: './schema/query_constructor.d.ts',
        require: './schema/query_constructor.d.cts',
        default: './schema/query_constructor.d.ts'
      },
      import: './schema/query_constructor.js',
      require: './schema/query_constructor.cjs'
    },
    './schema/prompt_template': {
      types: {
        import: './schema/prompt_template.d.ts',
        require: './schema/prompt_template.d.cts',
        default: './schema/prompt_template.d.ts'
      },
      import: './schema/prompt_template.js',
      require: './schema/prompt_template.cjs'
    },
    './package.json': './package.json'
  },
  gitHead: 'd0e1b796a9d07892092d7a34961e9e81f3ce8535',
  dist: {
    integrity: 'sha512-iZ0l7BDVfoifqZlDl1gy3JP5mIdhYjWiToPlDnlmfHD748cw3okvF0gZo0ruT4nbftnQcaM7JzPUiNC43UPfgg==', 
    shasum: '1341bdd7166f4f6da0b9337f363e409a79523dbb',
    tarball: 'https://registry.npmjs.org/langchain/-/langchain-0.2.9.tgz',
    fileCount: 1978,
    unpackedSize: 3988163,
    signatures: [
      {
        keyid: 'SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA',
        sig: 'MEQCIHfweA2hh5nLILOHPlFMG30cXQhmTqPm/65AlZXmyYVKAiBvKxaSQkX1TIYLJC3ADmFqgqqctAcxNHT/6G8G3mMILw=='   
      }
    ]
  },
  directories: {}
}
Done in 2.03s.
@dosubot dosubot bot added the auto:bug Related to a bug, vulnerability, unexpected error with an existing feature label Jul 11, 2024
@dosubot DosuBot
Copy link

dosubot bot commented Jul 11, 2024

Hey @Moe03! I'm here to help you with any bugs, questions, or contributions. Let's tackle this together while we wait for a human maintainer.

To enable proper streaming of events in the @langchain/anthropic package, you can utilize the stream method to yield chunks of data as they are generated. This method ensures that events are streamed in real-time rather than waiting for the entire tool and response to finish.

Here is an example of how you can use the stream method to achieve this:

import { ChatAnthropic } from "@langchain/anthropic";
import { AgentExecutor, createToolCallingAgent } from "langchain/agents";
import { ChatPromptTemplate, MessagesPlaceholder } from "@langchain/core/prompts";
import { HumanMessage, SystemMessage } from "@langchain/core/messages";
import { DynamicStructuredTool } from "@langchain/core/tools";
import { z } from "zod";

async function main() {
    const defaultTools = [
        new DynamicStructuredTool({
            name: "GetWeather",
            description: "Return the weather.",
            schema: z.object({
                query: z.string().describe("The city to return weather for."),
            }),
            func: async (input) => {
                console.log(`STARTING TOOL: GetWeather`);
                return 'sunny';
            },
        }),
    ];

    const chatPrompt = ChatPromptTemplate.fromMessages([
        new MessagesPlaceholder("messages"),
        new MessagesPlaceholder("agent_scratchpad"),
    ]);

    const messagesHistory = [
        new SystemMessage("you are a helpful assistant!"),
        new HumanMessage("Whats the weather like in Cairo?")
    ];

    const llm = new ChatAnthropic({
        apiKey: ANTHROPIC_API_KEY,
        model: "claude-3-5-sonnet-20240620",
        streaming: true, // Enable streaming
    });

    const agent = await createToolCallingAgent({
        llm,
        tools: defaultTools,
        prompt: chatPrompt,
    });

    const agentExec = new AgentExecutor({
        agent,
        tools: defaultTools,
    }).withConfig({ runName: "Agent" });

    let llmStream = await agentExec.streamEvents(
        {
            messages: messagesHistory,
        },
        {
            version: "v1"
        }
    );

    for await (const event of llmStream) {
        const eventType = event.event;
        if (eventType === "on_chain_start") {
            if (event.name === "Agent") {
                console.log("\n-----");
                console.log(`Starting agent: ${event.name} with input: ${JSON.stringify(event.data.input)}`);
            }
        } else if (eventType === "on_chain_end") {
            if (event?.name === "Agent") {
                console.log("\n-----");
                console.log(`Finished agent: ${event?.name}\n`);
                console.log(`Agent output was: ${event?.data?.output}`);
                console.log("\n-----");
            }
        } else if (eventType === "on_llm_stream" || eventType === "on_chat_model_stream" || eventType === "on_chain_stream") {
            console.log("\n-----");
            const content = event?.data?.chunk?.output || event.data?.chunk?.message?.content || event.data?.chunk?.content;
            if (content !== undefined && content !== "") {
                console.log(`| ${content}`);
            }
        } else if (eventType === "on_tool_start") {
            console.log("\n-----");
            console.log(`Starting tool: ${event.name} with inputs: ${event.data.input}`);
        } else if (eventType === "on_tool_end") {
            console.log("\n-----");
            console.log(`Finished tool: ${event.name}\n`);
            console.log(`Tool output was: ${event.data.output}`);
        }
    }
}

main();

This example ensures that chunks of data are yielded and events are sent in real-time, improving the performance of real-time applications [1][2][3].

To continue talking to Dosu, mention @dosu.

@Moe03 Moe03 mentioned this issue Jul 11, 2024
Closed
@jacoblee93
Copy link
Collaborator

Tracking in #6056

@jacoblee93
Copy link
Collaborator

This should now be fixed!

@snlamm
Copy link
Contributor

snlamm commented Jul 23, 2024

I ran into the same problem and after upgrading to @langchain/anthropic v0.2.6, handleLLMNewToken is still not triggering for me, unfortunately.

@jacoblee93
Copy link
Collaborator

@bracesproul can you have a look?

@jacoblee93 jacoblee93 reopened this Jul 23, 2024
@snlamm
Copy link
Contributor

snlamm commented Jul 23, 2024 •
edited
Loading

@bracesproul Thanks in advance!
I think I've found the lines at issue in langchain-anthropic/src/chat_model.ts. The problem is with the extractToken function -

Basically, extractToken expects a string but when tools are used it's being passed an object instead.

Here's the step-by-step:
First, the code sets coerceContentToString to always be false whenever a tool is used (which seems good and correct).

    // this will _always_ return false if a tool is being used
    const coerceContentToString = !_toolsInParams({
      ...params,
      ...formattedMessages,
      stream: false,
    });

Then, the streamed tool chunk content is set as the following object in _makeMessageChunkFromAnthropicEvent:

  } else if (
    data.type === "content_block_delta" &&
    data.delta.type === "input_json_delta"
  ) {
    return {
      chunk: new AIMessageChunk({
        content: fields.coerceContentToString
          ? ""
          : [
              {
                index: data.index,
                input: data.delta.partial_json, // this input field is where the streamed token is
                type: data.delta.type,
              },
            ],
        additional_kwargs: {},
      }),
      usageData: usageDataCopy,
    };
  }

The next step is where the issue lies. Basically, we have the extractToken function that's being passed the chunk content. However, it expects the content field to always be a string, whereas in this case it's an object (the one defined above)! Therefore, because it can't parse an object, it coerces the token value to always be undefined.

 const token = extractToken(chunk); // this will always be undefined, as it does not know how to handle the chunk content as an object

Because token is undefined, the callback is never triggered:

      if (token) {
        await runManager?.handleLLMNewToken(token); // this is unreachable
      }

In reality, though, if extractToken knew how to parse it, the token can be found in chunk.content[0].input.

For example:

[ { index: 0, input: 'opportu', type: 'input_json_delta' } ]

Is this helpful in terms of tracking it down?

@bracesproul
Copy link
Collaborator

Hey @snlamm I believe this PR should account for the edge case you're mentioning here #6179

Let me know if you're looking for something else.

@bracesproul bracesproul mentioned this issue Jul 23, 2024
Merged
@snlamm
Copy link
Contributor

snlamm commented Jul 23, 2024 •
edited
Loading

@bracesproul that looks great! Looks like it accounts for it 🙏 (added a comment on the PR itself re: making sure it's handling the needed type)

@snlamm snlamm mentioned this issue Jul 23, 2024
Merged
@snlamm
Copy link
Contributor

snlamm commented Jul 23, 2024

@bracesproul Ah looks like the new implementation introduces it's own bug 😅 . Opened a 1-liner to fix at #6183

@snlamm
Copy link
Contributor

snlamm commented Jul 23, 2024

Can confirm it works now - thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
auto:bug Related to a bug, vulnerability, unexpected error with an existing feature
Projects
None yet
Milestone
No milestone
4 participants
@jacoblee93 @snlamm @bracesproul @Moe03