Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add support for Message Batches in anthropic_sdk_dart #585

Merged
merged 1 commit into from
Oct 22, 2024

Conversation

davidmigloz
Copy link
Owner

@davidmigloz davidmigloz commented Oct 22, 2024

The Message Batches API is a powerful, cost-effective way to asynchronously process large volumes of Messages requests. This approach is well-suited to tasks that do not require immediate responses, reducing costs by 50% while increasing throughput.

Refer to the official documentation for more information.

Prepare and create your batch:

const batchRequest = CreateMessageBatchRequest(
  requests: [
    BatchMessageRequest(
      customId: 'request1',
      params: CreateMessageRequest(
        model: Model.model(Models.claudeInstant12),
        temperature: 0,
        maxTokens: 1024,
        messages: [
          Message(
            role: MessageRole.user,
            content: MessageContent.text(
                'List the numbers from 1 to 9 in order.'),
          ),
        ],
      ),
    ),
    BatchMessageRequest(
      customId: 'request2',
      params: CreateMessageRequest(
        model: Model.model(Models.claudeInstant12),
        temperature: 0,
        maxTokens: 1024,
        messages: [
          Message(
            role: MessageRole.user,
            content: MessageContent.text(
                'List the numbers from 10 to 19 in order.'),
          ),
        ],
      ),
    ),
  ],
);
var batch = await client.createMessageBatch(request: batchRequest);
print(batch.id);

Tracking your batch:

do {
  await Future<void>.delayed(const Duration(seconds: 5));
  batch = await client.retrieveMessageBatch(id: batch.id);
} while (batch.processingStatus == MessageBatchProcessingStatus.inProgress);

Retrieving batch results:

batch = await client.retrieveMessageBatch(id: batch.id);
print(batch.resultsUrl);

@davidmigloz davidmigloz self-assigned this Oct 22, 2024
@davidmigloz davidmigloz added t:enhancement New feature or request p:anthropic_sdk_dart anthropic_sdk_dart package. labels Oct 22, 2024
@davidmigloz davidmigloz added this to the v0.8.0 milestone Oct 22, 2024
@davidmigloz davidmigloz merged commit a41270a into main Oct 22, 2024
3 checks passed
@davidmigloz davidmigloz deleted the MessageBatches branch October 22, 2024 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
p:anthropic_sdk_dart anthropic_sdk_dart package. t:enhancement New feature or request
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

1 participant