Add support for grounding using google_search #29
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Owner
|
https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/ground-gemini says:
Need to implement that too. |
simonw
changed the title
Owner
|
Also need to figure out how we deal with this requirement: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/grounding-search-suggestions
|
Owner
|
WIP implementation of selectively making this option available for certain models: diff --git a/llm_gemini.py b/llm_gemini.py
index 605eae6..d378320 100644
--- a/llm_gemini.py
+++ b/llm_gemini.py
@@ -42,7 +42,7 @@ def register_models(register):
"gemini-exp-1206",
"gemini-2.0-flash-exp",
]:
- register(GeminiPro(model_id), AsyncGeminiPro(model_id))
+ register(GeminiPro(model_id, True), AsyncGeminiPro(model_id, True))
def resolve_type(attachment):
@@ -93,10 +93,6 @@ class _SharedGemini:
description="Enables the model to generate and run Python code",
default=None,
)
- google_search: Optional[bool] = Field(
- description="Enables the model to use Google Search to improve the accuracy and recency of responses from the model",
- default=None,
- )
temperature: Optional[float] = Field(
description=(
"Controls the randomness of the output. Use higher values for "
@@ -137,8 +133,16 @@ class _SharedGemini:
default=None,
)
- def __init__(self, model_id):
+ def __init__(self, model_id, can_google_search=False):
self.model_id = model_id
+ self.can_google_search = can_google_search
+ if can_google_search:
+ class Options(self.Options):
+ google_search: Optional[bool] = Field(
+ description="Enables the model to use Google Search to improve the accuracy and recency of responses from the model",
+ default=None,
+ )
+ self.Options = Options
def build_messages(self, prompt, conversation):
messages = []
@@ -184,7 +188,7 @@ class _SharedGemini:
}
if prompt.options and prompt.options.code_execution:
body["tools"] = [{"codeExecution": {}}]
- if prompt.options and prompt.options.google_search:
+ if prompt.options and self.can_google_search and prompt.options.google_search:
body["tools"] = [{"google_search": {}}]
if prompt.system:
body["systemInstruction"] = {"parts": [{"text": prompt.system}]} |
|
Any update on when this gets released? |
Owner
|
That |
Owner
|
Should I block attachments from the models that say they are "text only"? I'm tempted to leave them supported and let the Gemini API return an error - especially since that API might change in the future. |
Owner
|
OK, got this working: llm -m gemini-1.5-pro-latest -o google_search 1 'top sports news today for chess, one story, headline only'It output:
Here's the full response: llm logs -c --json[
{
"id": "01jj60b6c2wr52v6xyqrawp3pv",
"model": "gemini-1.5-pro-latest",
"prompt": "top sports news today for chess, one story, headline only",
"system": null,
"prompt_json": null,
"options_json": {
"google_search": true
},
"response": "Praggnanandhaa beats India No. 1 Arjun Erigasi to seize top spot at Tata Steel Chess 2025.\n",
"response_json": [
{
"candidates": [
{
"content": {
"parts": [
{
"text": "Prag"
}
],
"role": "model"
}
}
],
"usageMetadata": {
"promptTokenCount": 13,
"totalTokenCount": 13
},
"modelVersion": "gemini-1.5-pro-002"
},
{
"candidates": [
{
"content": {
"parts": [
{
"text": "gnanandhaa beats India No. 1 Arjun Erigasi to seize top"
}
],
"role": "model"
},
"safetyRatings": [
{
"category": "HARM_CATEGORY_HATE_SPEECH",
"probability": "NEGLIGIBLE"
},
{
"category": "HARM_CATEGORY_DANGEROUS_CONTENT",
"probability": "NEGLIGIBLE"
},
{
"category": "HARM_CATEGORY_HARASSMENT",
"probability": "NEGLIGIBLE"
},
{
"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
"probability": "NEGLIGIBLE"
}
]
}
],
"usageMetadata": {
"promptTokenCount": 13,
"totalTokenCount": 13
},
"modelVersion": "gemini-1.5-pro-002"
},
{
"candidates": [
{
"content": {
"parts": [
{
"text": " spot at Tata Steel Chess 2025.\n"
}
],
"role": "model"
},
"finishReason": "STOP",
"safetyRatings": [
{
"category": "HARM_CATEGORY_HATE_SPEECH",
"probability": "NEGLIGIBLE"
},
{
"category": "HARM_CATEGORY_DANGEROUS_CONTENT",
"probability": "NEGLIGIBLE"
},
{
"category": "HARM_CATEGORY_HARASSMENT",
"probability": "NEGLIGIBLE"
},
{
"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
"probability": "NEGLIGIBLE"
}
],
"groundingMetadata": {
"searchEntryPoint": {
"renderedContent": "<style>\n.container {\n align-items: center;\n border-radius: 8px;\n display: flex;\n font-family: Google Sans, Roboto, sans-serif;\n font-size: 14px;\n line-height: 20px;\n padding: 8px 12px;\n}\n.chip {\n display: inline-block;\n border: solid 1px;\n border-radius: 16px;\n min-width: 14px;\n padding: 5px 16px;\n text-align: center;\n user-select: none;\n margin: 0 8px;\n -webkit-tap-highlight-color: transparent;\n}\n.carousel {\n overflow: auto;\n scrollbar-width: none;\n white-space: nowrap;\n margin-right: -12px;\n}\n.headline {\n display: flex;\n margin-right: 4px;\n}\n.gradient-container {\n position: relative;\n}\n.gradient {\n position: absolute;\n transform: translate(3px, -9px);\n height: 36px;\n width: 9px;\n}\n@media (prefers-color-scheme: light) {\n .container {\n background-color: #fafafa;\n box-shadow: 0 0 0 1px #0000000f;\n }\n .headline-label {\n color: #1f1f1f;\n }\n .chip {\n background-color: #ffffff;\n border-color: #d2d2d2;\n color: #5e5e5e;\n text-decoration: none;\n }\n .chip:hover {\n background-color: #f2f2f2;\n }\n .chip:focus {\n background-color: #f2f2f2;\n }\n .chip:active {\n background-color: #d8d8d8;\n border-color: #b6b6b6;\n }\n .logo-dark {\n display: none;\n }\n .gradient {\n background: linear-gradient(90deg, #fafafa 15%, #fafafa00 100%);\n }\n}\n@media (prefers-color-scheme: dark) {\n .container {\n background-color: #1f1f1f;\n box-shadow: 0 0 0 1px #ffffff26;\n }\n .headline-label {\n color: #fff;\n }\n .chip {\n background-color: #2c2c2c;\n border-color: #3c4043;\n color: #fff;\n text-decoration: none;\n }\n .chip:hover {\n background-color: #353536;\n }\n .chip:focus {\n background-color: #353536;\n }\n .chip:active {\n background-color: #464849;\n border-color: #53575b;\n }\n .logo-light {\n display: none;\n }\n .gradient {\n background: linear-gradient(90deg, #1f1f1f 15%, #1f1f1f00 100%);\n }\n}\n</style>\n<div class=\"container\">\n <div class=\"headline\">\n <svg class=\"logo-light\" width=\"18\" height=\"18\" viewBox=\"9 9 35 35\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\">\n <path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M42.8622 27.0064C42.8622 25.7839 42.7525 24.6084 42.5487 23.4799H26.3109V30.1568H35.5897C35.1821 32.3041 33.9596 34.1222 32.1258 35.3448V39.6864H37.7213C40.9814 36.677 42.8622 32.2571 42.8622 27.0064V27.0064Z\" fill=\"#4285F4\"/>\n <path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M26.3109 43.8555C30.9659 43.8555 34.8687 42.3195 37.7213 39.6863L32.1258 35.3447C30.5898 36.3792 28.6306 37.0061 26.3109 37.0061C21.8282 37.0061 18.0195 33.9811 16.6559 29.906H10.9194V34.3573C13.7563 39.9841 19.5712 43.8555 26.3109 43.8555V43.8555Z\" fill=\"#34A853\"/>\n <path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M16.6559 29.8904C16.3111 28.8559 16.1074 27.7588 16.1074 26.6146C16.1074 25.4704 16.3111 24.3733 16.6559 23.3388V18.8875H10.9194C9.74388 21.2072 9.06992 23.8247 9.06992 26.6146C9.06992 29.4045 9.74388 32.022 10.9194 34.3417L15.3864 30.8621L16.6559 29.8904V29.8904Z\" fill=\"#FBBC05\"/>\n <path fill-rule=\"evenodd\" clip-rule=\"evenodd\" d=\"M26.3109 16.2386C28.85 16.2386 31.107 17.1164 32.9095 18.8091L37.8466 13.8719C34.853 11.082 30.9659 9.3736 26.3109 9.3736C19.5712 9.3736 13.7563 13.245 10.9194 18.8875L16.6559 23.3388C18.0195 19.2636 21.8282 16.2386 26.3109 16.2386V16.2386Z\" fill=\"#EA4335\"/>\n </svg>\n <svg class=\"logo-dark\" width=\"18\" height=\"18\" viewBox=\"0 0 48 48\" xmlns=\"http://www.w3.org/2000/svg\">\n <circle cx=\"24\" cy=\"23\" fill=\"#FFF\" r=\"22\"/>\n <path d=\"M33.76 34.26c2.75-2.56 4.49-6.37 4.49-11.26 0-.89-.08-1.84-.29-3H24.01v5.99h8.03c-.4 2.02-1.5 3.56-3.07 4.56v.75l3.91 2.97h.88z\" fill=\"#4285F4\"/>\n <path d=\"M15.58 25.77A8.845 8.845 0 0 0 24 31.86c1.92 0 3.62-.46 4.97-1.31l4.79 3.71C31.14 36.7 27.65 38 24 38c-5.93 0-11.01-3.4-13.45-8.36l.17-1.01 4.06-2.85h.8z\" fill=\"#34A853\"/>\n <path d=\"M15.59 20.21a8.864 8.864 0 0 0 0 5.58l-5.03 3.86c-.98-2-1.53-4.25-1.53-6.64 0-2.39.55-4.64 1.53-6.64l1-.22 3.81 2.98.22 1.08z\" fill=\"#FBBC05\"/>\n <path d=\"M24 14.14c2.11 0 4.02.75 5.52 1.98l4.36-4.36C31.22 9.43 27.81 8 24 8c-5.93 0-11.01 3.4-13.45 8.36l5.03 3.85A8.86 8.86 0 0 1 24 14.14z\" fill=\"#EA4335\"/>\n </svg>\n <div class=\"gradient-container\"><div class=\"gradient\"></div></div>\n </div>\n <div class=\"carousel\">\n <a class=\"chip\" href=\"https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUBnsYvH7Kd1eE0C2w8qENULacXFV13Sk6XfQX3nBZXnpxoAmL_UVbyzNq5vT0C6hqNqoxBIILAGzVefdpcbDThw7NBlPIfGmsiKWL4K3Y-q-yADqtCh5WG84zOegmES9AUoiht5-OCsGAqNFJtOnscduANDmeA1fJcT2ifNxFR976VSlyMt_tfKRdOdyvbSKffJR3oQNg9zFCbjqnE41F-L2A==\">top sports news today chess</a>\n </div>\n</div>\n"
},
"groundingChunks": [
{
"web": {
"uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUBnsYsrQrQGrr-V-l-mIKul_Ky1rCxauWXIYu5IQ2KGt_hz9XU5ECN7UKUou6xrSeT-HoQrL0nf2-Kk0VEjbDaws3gHveOwyKbFdcKfGBQGc6s6FRAQjENKy0qf2HmvMmiujkqx_dk4QX-Vz4ZP_w==",
"title": "indiatimes.com"
}
}
],
"groundingSupports": [
{
"segment": {
"endIndex": 90,
"text": "Praggnanandhaa beats India No. 1 Arjun Erigasi to seize top spot at Tata Steel Chess 2025."
},
"groundingChunkIndices": [
0
],
"confidenceScores": [
"Decimal('0.99521')"
]
}
],
"retrievalMetadata": {},
"webSearchQueries": [
"top sports news today chess"
]
}
}
],
"modelVersion": "gemini-1.5-pro-002"
}
],
"conversation_id": "01jj60b6c0ry2ajcf7dtezg9kc",
"duration_ms": 1490,
"datetime_utc": "2025-01-22T03:19:23.306704",
"input_tokens": 13,
"output_tokens": 29,
"token_details": null,
"conversation_name": "top sports news today for chess\u2026",
"conversation_model": "gemini-1.5-pro-latest",
"attachments": []
}
] |
simonw
added a commit
that referenced
this pull request
Jan 22, 2025
|
Thanks @simonw! I will try it out. |
|
Hmm is there a special requirement for this feature? I am using my api key from https://aistudio.google.com/apikey and most models gave me quota error. Using 2.0-flash-exp gave me this removing the google_search option and everything works just fine. Tried this on both my personal and corporate account 🤔 |
|
@sluongng Just dealt with this and finally got it figured out. In AI Studio it was showing my API Key's plan as Free with an option to "Link Billing". No matter what I did I would get that "Resource has been exhausted" error. I figured it must be payment related, so I clicked through to link billing, added my credit card number, poked around a bit, but was still getting that error. Tried different models, same error you saw about google_search_retrieval. But when I went back to AI Studio, I saw my API Key was still marked as "Free". I clicked through and sure enough, it had not connected the billing account I had just setup for this explicit purpose. As soon as I linked that billing account to my API Key, the FWIW I have a US credit card and US address and this was the command I ran: |
|
Yeah, that's a really good pointer. I was able to navigate to https://aistudio.google.com/plan_information and see this That's a bit unfortunate because I am already paying for Gemini through my Google One subscription and the $WORK account is not gonna enable extra billing for Aistudio 🤔 I do have access to ChatGPT Plus Search though, so guess I will not be touching this for a while. |
lokkju
pushed a commit
to lokkju/llm-gemini-code-assist
that referenced
this pull request
Nov 15, 2025
Co-authored-by: Simon Willison <swillison@gmail.com>
lokkju
pushed a commit
to lokkju/llm-gemini-code-assist
that referenced
this pull request
Nov 15, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for grounding which will use google search to improve the accuracy of the responses