vertexAI: response_schema can't parse TypedDicts in generation config

[ShouryaRSharma]

Description of the bug:

If a TypedDict is supplied in the response_schema field, as shown in the examples of the doc here, we get the following error:

python3.10/site-packages/vertexai/generative_models/_generative_mode
ls.py", line 2200, in _fix_schema_dict_for_gapic_in_place
    if "type" in schema_dict:
TypeError: argument of type '_TypedDictMeta' is not iterable

It appears this functionality is supported in the genai library but not vertex AI?

Actual vs expected behavior:

Expected behaviour:

When a typedDict is passed as the input for response_schema, we should receive structured outputs in that format based on the documentation provided above. However, for the following code:

class CodeOutput(TypedDict):
    line_number: int
    original_line: str
    new_line: str
    conventional_commit_message: str
    description: str

And the following generation_config -

 self._generation_config = GenerationConfig(
            response_mime_type="application/json",
            response_schema=CodeOutput,
        )
        
self._model.generate_content(
                content,
                safety_settings=self._safety_config,
                generation_config=self._generation_config,
                stream=False,
            )

I receive the error:

File "/Users/shourya.sharma/dev/scrapesmith/src/scrapesmith/client.py", line 122, in __init__
   self._generation_config = self._setup_generation_config()
 File "/Users/shourya.sharma/dev/scrapesmith/src/scrapesmith/client.py", line 142, in _setup_generation_config
   return GenerationConfig(
 File "/Users/shourya.sharma/Library/Caches/pypoetry/virtualenvs/scrapesmith-9e6PYqsH-py3.10/lib/python3.10/site-packages/vertexai/generative_models/_generative_mode
ls.py", line 1747, in __init__
   raw_schema = FunctionDeclaration(
 File "/Users/shourya.sharma/Library/Caches/pypoetry/virtualenvs/scrapesmith-9e6PYqsH-py3.10/lib/python3.10/site-packages/vertexai/generative_models/_generative_mode
ls.py", line 2161, in __init__
   _fix_schema_dict_for_gapic_in_place(parameters)
 File "/Users/shourya.sharma/Library/Caches/pypoetry/virtualenvs/scrapesmith-9e6PYqsH-py3.10/lib/python3.10/site-packages/vertexai/generative_models/_generative_mode
ls.py", line 2200, in _fix_schema_dict_for_gapic_in_place
   if "type" in schema_dict:
TypeError: argument of type '_TypedDictMeta' is not iterable

Current dev environment:

• MacOS Darwin (M2 Macbook Pro)
• Python 3.10.14
• VertexAI version 1.71.1

Any other information you'd like to share?

No response

[manojssmk]

Hi @ShouryaRSharma

You seem to be referencing the Python SDK code for Vertex AI's Generative AI. If you're looking for guidance on structured output in Vertex AI, you can refer to this link

Thanks

[ShouryaRSharma]

Is there a reason the 2 APIs (Vertex AI vs GenAI) don't have parity or same featureset for gemini models?

[manojssmk]

A new Python SDK, Google Gen AI SDK, has been released, supporting both Vertex AI and Google Generative AI. Check out the link for more details.

[github-actions]

Marking this issue as stale since it has been open for 14 days with no activity. This issue will be closed if no further activity occurs.