Skip to content

fix(provider/groq): surface cached_tokens from prompt_tokens_details#13192

Open
armandmcqueen wants to merge 2 commits intovercel:mainfrom
armandmcqueen:groq_report_cache_usage
Open

fix(provider/groq): surface cached_tokens from prompt_tokens_details#13192
armandmcqueen wants to merge 2 commits intovercel:mainfrom
armandmcqueen:groq_report_cache_usage

Conversation

@armandmcqueen
Copy link

Background

groq now has models with input cache support. For example: https://console.groq.com/docs/model/moonshotai/kimi-k2-instruct-0905

The AI SDK was not correctly extracting the response from groq into the usage object.

Summary

Pull the prompt cache data from the appropriate response field.

Manual Verification

Here is a raw response object from a call with cached input tokens

"rawResponse": {
    "id": "chatcmpl-a46c2c95-4e88-4fca-8165-3bb46e74bb83",
    "object": "chat.completion",
    "created": 1772843463,
    "model": "openai/gpt-oss-20b",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "The CAP theorem states that a distributed system can simultaneously provide at most two of the following three guarantees: Consistency, Availability, and Partition tolerance.",
          "reasoning": "We need one sentence answer. Keep concise."
        },
        "logprobs": null,
        "finish_reason": "stop"
      }
    ],
    "usage": {
      "queue_time": 0.041304244,
      "prompt_tokens": 3829,
      "prompt_time": 0.025152422,
      "completion_tokens": 48,
      "completion_time": 0.049817351,
      "total_tokens": 3877,
      "total_time": 0.074969773,
      "prompt_tokens_details": {
        "cached_tokens": 3584
      },
      "completion_tokens_details": {
        "reasoning_tokens": 10
      }
    },
    "usage_breakdown": null,
    "x_groq": {
      "id": "req_xxxx",
      "seed": 1536934315
    },
    "service_tier": "on_demand"
  },

Checklist

  • Tests have been added / updated (for bug fixes / features)
  • Documentation has been added / updated (for bug fixes / features)
  • A patch changeset for relevant packages has been added (for bug fixes / features - run pnpm changeset in the project root)
  • I have reviewed this pull request (self-review)

Future Work

Related Issues

@tigent tigent bot added ai/provider related to a provider package. Must be assigned together with at least one `provider/*` label bug Something isn't working as documented provider/groq Issues related to the @ai-sdk/groq provider labels Mar 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai/provider related to a provider package. Must be assigned together with at least one `provider/*` label bug Something isn't working as documented provider/groq Issues related to the @ai-sdk/groq provider

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant