ProviderStrategy fails with JSONDecodeError for Gemini on VertexAI

Package (Required)

• langchain-google-genai
• langchain-google-vertexai
• langchain-google-community
• Other / not sure / general

Checked other resources

• I added a descriptive title to this issue
• I searched the LangChain documentation and API reference (linked above)
• I used the GitHub search to find a similar issue and didn't find it
• I am sure this is a bug and not a question or request for help

Example Code (Python)

"""
Minimal reproducible example showing ProviderStrategy JSONDecodeError with Gemini.

This script demonstrates:
1. ToolStrategy works perfectly ✅
2. ProviderStrategy fails with JSONDecodeError ❌
3. Both use the same model, schema, and prompt

Run with:
    export GCP_PROJECT_ID="your-project-id"
    python reproduce_provider_strategy_bug.py

Requirements:
    pip install langchain langchain-google-vertexai
"""
import asyncio
import os
from langchain_google_vertexai import ChatVertexAI
from langchain.agents import create_agent
from langchain.agents.structured_output import ProviderStrategy, ToolStrategy

# Simple JSON schema for testing
SCHEMA = {
    "type": "object",
    "properties": {
        "sentiment": {
            "type": "string",
            "enum": ["positive", "negative", "neutral"],
            "description": "The sentiment of the text"
        },
        "confidence": {
            "type": "number",
            "minimum": 0,
            "maximum": 1,
            "description": "Confidence score"
        }
    },
    "required": ["sentiment", "confidence"],
    "additionalProperties": False
}

PROMPT = "Analyze this review: 'This product is absolutely amazing!'"


async def test_tool_strategy_works():
    """✅ This works - proves setup is correct"""
    print("\n" + "=" * 80)
    print("TEST 1: ToolStrategy with Gemini")
    print("=" * 80)

    model = ChatGoogleGenerativeAI(
        model="gemini-2.5-flash",
        temperature=0.0,
        credentials=None,  # Force API key usage instead of GCP credentials
    )

    agent = create_agent(
        model=model,
        tools=[],
        response_format=ToolStrategy(schema=SCHEMA, handle_errors=True),
    )

    try:
        result = await agent.ainvoke({
            "messages": [{"role": "user", "content": PROMPT}]
        })

        print("✅ SUCCESS - ToolStrategy works!")
        print(f"Structured response: {result.get('structured_response')}")
        print(f"Response type: {type(result.get('structured_response'))}")
    except Exception as e:
        print(f"❌ UNEXPECTED FAILURE: {type(e).__name__}")
        print(f"Error: {str(e)}")


async def test_provider_strategy_fails():
    """❌ This fails - JSONDecodeError at char 0"""
    print("\n" + "=" * 80)
    print("TEST 2: ProviderStrategy with Gemini")
    print("=" * 80)

    model = ChatGoogleGenerativeAI(
        model="gemini-2.5-flash",
        temperature=0.0,
        credentials=None,  # Force API key usage instead of GCP credentials
    )

    try:
        agent = create_agent(
            model=model,
            tools=[],
            response_format=ProviderStrategy(schema=SCHEMA),  # ❌ This fails
        )

        result = await agent.ainvoke({
            "messages": [{"role": "user", "content": PROMPT}]
        })

        print("✅ UNEXPECTED SUCCESS!")
        print(f"Structured response: {result.get('structured_response')}")
        print("(If you see this, the bug may have been fixed!)")

    except Exception as e:
        print(f"❌ FAILED: {type(e).__name__}")
        print(f"Error message: {str(e)}")
        print()
        print("Full traceback:")
        import traceback
        traceback.print_exc()


async def main():
    """Run both tests to demonstrate the issue"""
    print("\n" + "=" * 80)
    print("BUG REPRODUCTION: ProviderStrategy JSONDecodeError with Gemini")
    print("=" * 80)

    # Check environment
    if not os.getenv("GCP_PROJECT_ID"):
        print()
        print("ERROR: GCP_PROJECT_ID environment variable not set")
        print()
        print("Please set it before running:")
        print("  export GCP_PROJECT_ID='your-project-id'")
        print()
        return

    print(f"Project ID: {os.getenv('GCP_PROJECT_ID')}")
    print(f"Model: gemini-2.5-flash")
    print(f"Schema: Simple sentiment analysis")

    # Run both tests
    await test_tool_strategy_works()
    await test_provider_strategy_fails()

    # Summary
    print("\n" + "=" * 80)
    print("SUMMARY")
    print("=" * 80)
    print("✅ ToolStrategy works - proves model, schema, and auth are correct")
    print("❌ ProviderStrategy fails with JSONDecodeError at char 0")
    print()
    print("This demonstrates that ProviderStrategy doesn't properly support")
    print("Gemini's native structured output capabilities, despite Gemini")
    print("officially supporting responseSchema.")
    print()
    print("See: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output")
    print("=" * 80 + "\n")


if __name__ == "__main__":
    asyncio.run(main())

Error Message and Stack Trace (if applicable)

BUG REPRODUCTION: ProviderStrategy JSONDecodeError with Gemini
================================================================================
API Key: Set
Model: gemini-2.5-flash
Schema: Simple sentiment analysis

================================================================================
TEST 1: ToolStrategy with Gemini
================================================================================
Key 'additionalProperties' is not supported in schema, ignoring
SUCCESS - ToolStrategy works!

Full result structure:
  Keys: ['messages', 'structured_response']

Structured response:
{'confidence': 0.99, 'sentiment': 'positive'}

Response type: <class 'dict'>

Messages (3 total):
  [0] HumanMessage - Analyze this review: 'This product is absolutely amazing!'
  [1] AIMessage -
  [2] ToolMessage - Returning structured response: {'sentiment': 'positive', 'confidence': 0.99}

================================================================================
TEST 2: ProviderStrategy with Gemini
================================================================================
FAILED: StructuredOutputValidationError
Error message: Failed to parse structured output for tool 'response_format': Native structured output expected valid JSON for response_format, but parsing failed: Expecting value: line 1 column 1 (char 0)..

Full traceback:
Traceback (most recent call last):
  File "D:\Project\Opensource\langchain_google\langchain_g\Lib\site-packages\langchain\agents\structured_output.py", line 380, in parse
    data = json.loads(raw_text)
  File "C:\Users\rohan\AppData\Local\Programs\Python\Python313\Lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "C:\Users\rohan\AppData\Local\Programs\Python\Python313\Lib\json\decoder.py", line 345, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\rohan\AppData\Local\Programs\Python\Python313\Lib\json\decoder.py", line 363, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\Project\Opensource\langchain_google\langchain_g\Lib\site-packages\langchain\agents\factory.py", line 888, in _handle_model_output
    structured_response = provider_strategy_binding.parse(output)
  File "D:\Project\Opensource\langchain_google\langchain_g\Lib\site-packages\langchain\agents\structured_output.py", line 387, in parse
    raise ValueError(msg) from e
ValueError: Native structured output expected valid JSON for response_format, but parsing failed: Expecting value: line 1 column 1 (char 0).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Project\Opensource\langchain_google\tp.py", line 107, in test_provider_strategy_fails
    result = await agent.ainvoke({
             ^^^^^^^^^^^^^^^^^^^^^
        "messages": [{"role": "user", "content": PROMPT}]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    })
    ^^
  File "D:\Project\Opensource\langchain_google\langchain_g\Lib\site-packages\langgraph\pregel\main.py", line 3158, in ainvoke
    async for chunk in self.astream(
    ...<29 lines>...
            chunks.append(chunk)
  File "D:\Project\Opensource\langchain_google\langchain_g\Lib\site-packages\langgraph\pregel\main.py", line 2971, in astream
    async for _ in runner.atick(
    ...<13 lines>...
            yield o
  File "D:\Project\Opensource\langchain_google\langchain_g\Lib\site-packages\langgraph\pregel\_runner.py", line 304, in atick
    await arun_with_retry(
    ...<15 lines>...
    )
  File "D:\Project\Opensource\langchain_google\langchain_g\Lib\site-packages\langgraph\pregel\_retry.py", line 137, in arun_with_retry
    return await task.proc.ainvoke(task.input, config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Project\Opensource\langchain_google\langchain_g\Lib\site-packages\langgraph\_internal\_runnable.py", line 705, in ainvoke
    input = await asyncio.create_task(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
        step.ainvoke(input, config, **kwargs), context=context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "D:\Project\Opensource\langchain_google\langchain_g\Lib\site-packages\langgraph\_internal\_runnable.py", line 473, in ainvoke
    ret = await self.afunc(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Project\Opensource\langchain_google\langchain_g\Lib\site-packages\langchain\agents\factory.py", line 1182, in amodel_node
    response = await _execute_model_async(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Project\Opensource\langchain_google\langchain_g\Lib\site-packages\langchain\agents\factory.py", line 1158, in _execute_model_async
    handled_output = _handle_model_output(output, effective_response_format)
  File "D:\Project\Opensource\langchain_google\langchain_g\Lib\site-packages\langchain\agents\factory.py", line 894, in _handle_model_output
    raise validation_error
langchain.agents.structured_output.StructuredOutputValidationError: Failed to parse structured output for tool 'response_format': Native structured output expected valid JSON for response_format, but parsing failed: Expecting value: line 1 column 1 (char 0)..
During task with name 'model' and id 'ab355218-bff0-1a8f-16fa-e028fe835f3d'

================================================================================
SUMMARY
================================================================================
ToolStrategy works - proves model, schema, and auth are correct
ProviderStrategy fails with JSONDecodeError at char 0

This demonstrates that ProviderStrategy doesn't properly support
Gemini's native structured output capabilities, despite Gemini
officially supporting responseSchema.

See: https://ai.google.dev/gemini-api/docs/structured-output
================================================================================

Description

Problem Summary When using ProviderStrategy with Gemini models on Google Cloud VertexAI, LangChain raises a JSONDecodeError: Expecting value: line 1 column 1 (char 0). However, ToolStrategy works perfectly with the exact same configuration.

What I'm trying to do I want to use ProviderStrategy with Gemini models on VertexAI because:

Gemini supports native structured output via the responseSchema parameter (Google Cloud Documentation) ProviderStrategy is more reliable than ToolStrategy per LangChain docs, as it uses provider-native capabilities instead of tool calling Better performance - native JSON output is faster and more reliable than tool-based extraction What I expect to happen ProviderStrategy should:

Configure the Gemini request with responseMimeType="application/json" and responseSchema= Receive a properly formatted JSON response Parse it successfully and return structured output Just like it does for OpenAI's response_format parameter.

What actually happens ProviderStrategy:

Accepts the configuration without complaint Makes a request to Gemini (exact request format unclear) Receives a response that doesn't match expected format Fails with JSONDecodeError: Expecting value: line 1 column 1 (char 0) Meanwhile, ToolStrategy with the same model/schema/prompt works perfectly, proving the setup is correct.

This issue was originally raised by @alejandroem1 in the langchain package. Link: https://github.com/langchain-ai/langchain-google/issues/1395. @ccurme advised that this should be implemented in langchain-google because Gemini response_format needs to be handled there.

A small change made in the example code is I have used ChatGoogleGenerativeAI() instead of ChatVertexAI(). After this PR gets accepted I will be happy to fix it ChatVertexAI() as well.