Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unexpected behavior with some Spanish prompts than finish reason is OTHER #593

Open
mdarino opened this issue Oct 9, 2024 · 2 comments
Open
Assignees
Labels
component:python sdk Issue/PR related to Python SDK status:triaged Issue/PR triaged to the corresponding sub-team type:bug Something isn't working

Comments

@mdarino
Copy link

mdarino commented Oct 9, 2024

Description of the bug:

I have a Python script that generates menus by reading them from a list and completing a prompt (the prompts are in Spanish). However, for some reason, certain prompts return a ValueError. After some testing, I was able to reduce the issue to a small script, shown below:

import google.generativeai as genai
import os


genai.configure(api_key=os.environ["GEMINI_API_KEY"])

model = genai.GenerativeModel(model_name="gemini-1.5-flash")
response = model.generate_content(
    'Genera una lista de ingredientes del siguiente menu: "Cafe con leche"',
    generation_config = genai.GenerationConfig(
        max_output_tokens=1000,
        temperature=0.1,
    )
)
print(response)
print(response.text)

When I change the menu (e.g., to "Té con leche"), it fails, and the error message is not clear enough for me to understand the cause, it shows "finish_reason": "OTHER"

Actual vs expected behavior:

Some examples:

1 -

import google.generativeai as genai
import os


genai.configure(api_key=os.environ["GEMINI_API_KEY"])

model = genai.GenerativeModel(model_name="gemini-1.5-flash")
response = model.generate_content(
    'Genera una lista de ingredientes del siguiente menu: "Cafe con leche"',
    generation_config = genai.GenerationConfig(
        max_output_tokens=1000,
        temperature=0.1,
    )
)
print(response)
print(response.text)

RESULT OK:

response:
GenerateContentResponse(
    done=True,
    iterator=None,
    result=protos.GenerateContentResponse({
      "candidates": [
        {
          "content": {
            "parts": [
              {
                "text": "## Ingredientes para Caf\u00e9 con Leche:\n\n* **Caf\u00e9:** (molido o en grano)\n* **Leche:** (entera, semidesnatada, desnatada, vegetal)\n* **Agua:** (para preparar el caf\u00e9)\n* **Az\u00facar:** (opcional)\n"
              }
            ],
            "role": "model"
          },
          "finish_reason": "STOP",
          "index": 0,
          "safety_ratings": [
            {
              "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
              "probability": "NEGLIGIBLE"
            },
            {
              "category": "HARM_CATEGORY_HATE_SPEECH",
              "probability": "NEGLIGIBLE"
            },
            {
              "category": "HARM_CATEGORY_HARASSMENT",
              "probability": "NEGLIGIBLE"
            },
            {
              "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
              "probability": "NEGLIGIBLE"
            }
          ]
        }
      ],
      "usage_metadata": {
        "prompt_token_count": 16,
        "candidates_token_count": 60,
        "total_token_count": 76
      }
    }),
)
## Ingredientes para Café con Leche:

* **Café:** (molido o en grano)
* **Leche:** (entera, semidesnatada, desnatada, vegetal)
* **Agua:** (para preparar el café)
* **Azúcar:** (opcional)

2 -

import google.generativeai as genai
import os


genai.configure(api_key=os.environ["GEMINI_API_KEY"])

model = genai.GenerativeModel(model_name="gemini-1.5-flash")
response = model.generate_content(
    'Genera una lista de ingredientes del siguiente menu: "Te con leche"',
    generation_config = genai.GenerationConfig(
        max_output_tokens=1000,
        temperature=0.1,
    )
)
print(response)
print(response.text)
response:
GenerateContentResponse(
    done=True,
    iterator=None,
    result=protos.GenerateContentResponse({
      "candidates": [
        {
          "finish_reason": "OTHER",
          "index": 0
        }
      ],
      "usage_metadata": {
        "prompt_token_count": 16,
        "total_token_count": 16
      }
    }),
)
Traceback (most recent call last):
  File "/home/mdarino/Desktop/foodcards/ai_menus/error.py", line 16, in <module>
    print(response.text)
          ^^^^^^^^^^^^^
  File "/home/mdarino/Desktop/foodcards/ai_menus/.venv/lib/python3.12/site-packages/google/generativeai/types/generation_types.py", line 495, in text
    raise ValueError(msg)
ValueError: Invalid operation: The `response.text` quick accessor requires the response to contain a valid `Part`, but none were returned. The candidate's [finish_reason](https://ai.google.dev/api/generate-content#finishreason) is 5.

Any other information you'd like to share?

Another example:

Prompt: 'Genera una lista de ingredientes del siguiente menu: "Pizza Mexicana (carne, jalapeños, frijoles, mozzarella)"',

RESULT NOT OK:

response:
GenerateContentResponse(
    done=True,
    iterator=None,
    result=protos.GenerateContentResponse({
      "candidates": [
        {
          "finish_reason": "OTHER",
          "index": 0
        }
      ],
      "usage_metadata": {
        "prompt_token_count": 27,
        "total_token_count": 27
      }
    }),
)
Traceback (most recent call last):
  File "/home/mdarino/Desktop/foodcards/ai_menus/error.py", line 16, in <module>
    print(response.text)
          ^^^^^^^^^^^^^
  File "/home/mdarino/Desktop/foodcards/ai_menus/.venv/lib/python3.12/site-packages/google/generativeai/types/generation_types.py", line 495, in text
    raise ValueError(msg)
ValueError: Invalid operation: The `response.text` quick accessor requires the response to contain a valid `Part`, but none were returned. The candidate's [finish_reason](https://ai.google.dev/api/generate-content#finishreason) is 5.

But Prompt: 'Genera una lista de ingredientes del siguiente menu: "Pizza de camarones (camarones, ajo, mozzarella)"',

RETURN OK:

response:
GenerateContentResponse(
    done=True,
    iterator=None,
    result=protos.GenerateContentResponse({
      "candidates": [
        {
          "content": {
            "parts": [
              {
                "text": "## Ingredientes para Pizza de Camarones:\n\n* **Base de pizza:** (Harina, agua, levadura, aceite de oliva, sal)\n* **Salsa de tomate:** (Tomates, cebolla, ajo, especias)\n* **Camarones:** (Frescos o congelados)\n* **Ajo:** (Picado o en polvo)\n* **Mozzarella:** (Rallada o en rebanadas)\n* **Aceite de oliva:** (Para rociar)\n* **Or\u00e9gano:** (Para espolvorear)\n* **Sal y pimienta:** (Al gusto)\n\n**Opcionales:**\n\n* **Ceboll\u00edn:** (Picado para decorar)\n* **Perejil:** (Picado para decorar)\n* **Queso parmesano:** (Rallado para espolvorear)\n* **Piment\u00f3n dulce:** (Para espolvorear)\n* **Aj\u00ed picante:** (Para agregar un toque picante)\n"
              }
            ],
            "role": "model"
          },
          "finish_reason": "STOP",
          "index": 0,
          "safety_ratings": [
            {
              "category": "HARM_CATEGORY_SEXUALLY_EXPLICIT",
              "probability": "NEGLIGIBLE"
            },
            {
              "category": "HARM_CATEGORY_HATE_SPEECH",
              "probability": "NEGLIGIBLE"
            },
            {
              "category": "HARM_CATEGORY_HARASSMENT",
              "probability": "NEGLIGIBLE"
            },
            {
              "category": "HARM_CATEGORY_DANGEROUS_CONTENT",
              "probability": "NEGLIGIBLE"
            }
          ]
        }
      ],
      "usage_metadata": {
        "prompt_token_count": 25,
        "candidates_token_count": 205,
        "total_token_count": 230
      }
    }),
)
## Ingredientes para Pizza de Camarones:

* **Base de pizza:** (Harina, agua, levadura, aceite de oliva, sal)
* **Salsa de tomate:** (Tomates, cebolla, ajo, especias)
* **Camarones:** (Frescos o congelados)
* **Ajo:** (Picado o en polvo)
* **Mozzarella:** (Rallada o en rebanadas)
* **Aceite de oliva:** (Para rociar)
* **Orégano:** (Para espolvorear)
* **Sal y pimienta:** (Al gusto)

**Opcionales:**

* **Cebollín:** (Picado para decorar)
* **Perejil:** (Picado para decorar)
* **Queso parmesano:** (Rallado para espolvorear)
* **Pimentón dulce:** (Para espolvorear)
* **Ají picante:** (Para agregar un toque picante)

@mdarino
Copy link
Author

mdarino commented Oct 9, 2024

In English: 'Generate a list of ingredients for the following menu: "Te con leche"'
Work well
On www.gemini.google.com, all prompts respond as expected (just in case I tested them)

@gmKeshari gmKeshari added type:bug Something isn't working status:triaged Issue/PR triaged to the corresponding sub-team component:python sdk Issue/PR related to Python SDK labels Oct 10, 2024
@gmKeshari gmKeshari self-assigned this Oct 10, 2024
@gmKeshari
Copy link

Hi @mdarino

I have escalated this to internal team.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:python sdk Issue/PR related to Python SDK status:triaged Issue/PR triaged to the corresponding sub-team type:bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants