Model's maximum context length error using Agents and Azure provider.

[lxDaniel]

Hi, I'm consistently getting this error ONLY when using Agents.
Interestingly, it works perfectly well with Azure Assistants (I need the features that Agents offer tho).:

The librechat-api log indicates the following:
2024-12-26 09:01:49 error: [handleAbortError] AI response error; aborting request: 400 This model's maximum context length is 128000 tokens. However, your messages resulted in 202177 tokens (202122 in the messages, 55 in the functions). Please reduce the length of the messages or functions.
The token count may vary, the largest number I got so far is around 450k.

I'm using Librechat v0.76, last commit as of today (12/26) (not like it worked at some point, it didn't), to interact with an Azure AI Search Index using Librechat's Agents.
AzureOpenAI provider, gpt-4o model (the error also happens with gpt-35-turbo-16k).
My librechat.yml file looks like this:

endpoints:
  agents:
    disableBuilder: false
  azureOpenAI:
    titleModel: "gpt-35-turbo-16k"
    titleConvo: true
    plugins: true
    agents: true
    assistants: true
    groups:
      - group: "eastus"
        apiKey: myapikey
        agents: true
        assistants: true
        instanceName: "myinstance"
        version: "2024-08-01-preview"
        models:
          gpt-4o:
            deploymentName: mydeploy-4o
          gpt-35-turbo-16k:
            deploymentName: mydeploy-35
  azureAssistants:
    capabilities: ["code_interpreter", "actions", "tools"]

Any guidance is welcomed!

[danny-avila]

Can you share a conversation where this happens to you? Use export feature from top right, you can redact the content if you wish.

[lxDaniel]

Sure, I dm'ed it to you on discord.
If it's not the more appropriate way to do it, please let me know.

[danny-avila]

Your Azure AI Search tool is returning 100k tokens per response. Context window will quickly fill up. I am fixing an issue where tokens were not being counted for tool calls but there is no counting done during the run, only before the run starts.

[danny-avila]

FWIW it seems you are erroring mid-run probably from consecutive ai search calls, since the context window is about 100k at this point:

[danny-avila]

Assistants likely discard or truncate tokens mid-run if tool outputs start exceeding the context, will think about how to best handle this for agents.