Claude in Vertex AI does not support URL-based image ingestion

Description

Passing an image as an URL works with Anthropic's API and with Gemini (via Vertex) too. But when using Claude via Vertex AI we get an error. (It works with base64) Google's Python ADK seems to have the same problem : https://github.com/google/adk-python/issues/2466 It has already been noted and fixed by vertex AI : https://github.com/vercel/ai/pull/6618

Steps to Reproduce

image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/2/2e/Morenas_negras_%28Muraena_augusti%29%2C_franja_marina_Teno-Rasca%2C_Tenerife%2C_Espa%C3%B1a%2C_2022-01-09%2C_DD_52.jpg/1000px-Morenas_negras_%28Muraena_augusti%29%2C_franja_marina_Teno-Rasca%2C_Tenerife%2C_Espa%C3%B1a%2C_2022-01-09%2C_DD_52.jpg"


def multiply(a: int, b: int) -> int:
    """Multiply two numbers"""
    return a * b


agent = create_react_agent(
    model=agent_model,
    # pass the tool that can update state
    tools=[multiply],
)

print(
    agent.invoke(
        {
            "messages": [
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": "Describe this image"},
                        {
                            "type": "image",
                            "source_type": "url",
                            "url": image_url,
                        },
                    ],
                }
            ]
        }
    )
)

Expected Behavior

Claude on Vertex should accept URL-based images, just like Anthropic's API and Gemini on Vertex

Actual Behavior

Error returned:

Retrying langchain_google_vertexai.model_garden.ChatAnthropicVertex._generate.<locals>._completion_with_retry_inner in 4.0 seconds as it raised BadRequestError: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.0.content.1.image.source.base64.data: URL sources are not supported'}}.
Retrying langchain_google_vertexai.model_garden.ChatAnthropicVertex._generate.<locals>._completion_with_retry_inner in 4.0 seconds as it raised BadRequestError: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.0.content.1.image.source.base64.data: URL sources are not supported'}}.

Environment

langchain version: 0.3.27

Python version: 3.13

Vertex AI model: claude-sonnet-4@20250514