Feature request: add support for OpenAI O1/o1-mini model

[Sander-Chen]

I'm currently trying to use OpenAI's o1 and o1-mini models through chainforge's custom provider, but encountering difficulties with the implementation. While o1-mini works occasionally, o1-preview-2024-09-12 consistently returns a 400 Bad Request error when making API calls.

Current Setup

• Using custom provider implementation
• Successfully connected to OpenAI API
• o1-mini works in some cases
• o1-preview-2024-09-12 and o1-preview fails with 400 error

Feature Request

Would it be possible to add native support for OpenAI's o1 model family in chainforge? This would:

• Ensure proper parameter handling for these models
• Provide better integration with chainforge's existing OpenAI support
• Help users avoid implementation issues with custom providers

Impact

Native support would benefit users who want to:

• Experiment with OpenAI's latest models
• Compare results between different model versions
• Build reliable workflows using these models

Let me know if you need any additional information. Looking forward to your response!

I let AI to help me built a custom provider file, but only worked for o1-mini, not sure why o1-preview is not working:

# -*- coding: utf-8 -*-
from chainforge.providers import provider
import requests
import urllib3
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# Disable SSL warnings
urllib3.disable_warnings()

THIRD_PARTY_GPT_SETTINGS_SCHEMA = {
  "settings": {
    "temperature": {
      "type": "number",
      "title": "temperature",
      "description": "Controls the 'creativity' or randomness of the response.",
      "default": 1,
      "minimum": 0,
      "maximum": 1.0,
      "multipleOf": 0.1,
    },
    "max_completion_tokens": {
      "type": "integer",
      "title": "max_completion_tokens",
      "description": "Maximum number of tokens to generate in the response.",
      "default": 65536,
      "minimum": 1,
      "maximum": 65536,
    },
    "presence_penalty": {
      "type": "number",
      "title": "Presence Penalty",
      "description": "Penalize new tokens based on their presence in the text so far.",
      "default": 0,
      "minimum": -2.0,
      "maximum": 2.0,
      "multipleOf": 0.1,
    },
    "frequency_penalty": {
      "type": "number",
      "title": "Frequency Penalty",
      "description": "Penalize new tokens based on their frequency in the text so far.",
      "default": 0,
      "minimum": -2.0,
      "maximum": 2.0,
      "multipleOf": 0.1,
    },
  },
  "ui": {
    "temperature": {
      "ui:help": "Defaults to 1.",
      "ui:widget": "range"
    },
    "max_completion_tokens": {
      "ui:help": "Defaults to 100.",
      "ui:widget": "range"
    },
  }
}

@provider(name="o1-model",
          emoji="\U0001F680",
          models=["o1-mini", "o1-preview-2024-09-12"],
          rate_limit="sequential",
          settings_schema=THIRD_PARTY_GPT_SETTINGS_SCHEMA)
def third_party_gpt_v2_completion(prompt: str, model: str, temperature: float = 1, max_completion_tokens: int = 1000, **kwargs) -> str:
    url = "https://api.openai.com/v1/chat/completions"
    
    # Create a session and configure the retry mechanism
    session = requests.Session()
    retry = Retry(
        total=5,  # Increase the number of retries
        backoff_factor=0.5,
        status_forcelist=[408, 429, 500, 502, 503, 504]
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('https://', adapter)
    
    headers = {
        "Content-Type": "application/json",
        "Authorization": "Bearer YOUR_API_KEY"  # Replace YOUR_API_KEY with your actual API key
    }
    data = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": temperature,
        "max_completion_tokens": max_completion_tokens
    }
    
    # Disable SSL verification
    response = session.post(url, headers=headers, json=data, verify=False, timeout=360)
    response.raise_for_status()
    result = response.json()
    return result["choices"][0]["message"]["content"]

[ianarawjo]

Did you check out the most recent update pushed yesterday? Try that first.

[Sander-Chen]

Hi, I have updated my chainforge to the newest version: 25.0.1

And I saw your update notes:
Custom model names to all providers (#324)

• Add custom widget using datalist to set model not present in dropdown enum list

• Require LLMProvider explicitly passed when querying LLMs.

• Finished UI datalist widget for react-jsonschema-form. Tested custom model endpoints in settings windows.

• Added o1+ model hack to strip system message, since OpenAI o1+ models do not yet support system messages (and the developer command on their API does not currently work...)

---

However, it's still not working.

I tried to let o3 mini to read the "ModelSettingSchemas.tsx file", it did give me some suggestions, but the code it modified is still not working:(.

I really expect that you can either:

1. add support for o1 model series.
2. Otherwise, you can give me a temporary template for the o1 series model, so that I can modify it by myself?

I understand UI changes might take time, so a working template would be extremely helpful as an interim solution.

Thank you for your help!

[ianarawjo]

I’ve tested the o1 models myself in the new version and then do work. Make sure your browser is refreshed.

[Sander-Chen]

Thank you for your prompt response!
After performing a hard browser cache refresh and extensive testing, I've successfully implemented the o1 models through the revamped built-in model configuration interface. This workflow improvement is exceptionally valuable for:

• Future new model adoption (e.g. upcoming o2/o3 series)
• Simplifying provider-agnostic model management
• Reducing custom provider dependencies

Documentation Enhancement Proposal

While discovering the manual model addition feature through trial-and-error, I noticed:

1. The official changelog last updated on 2024-12-20 lacks these UI updates
2. First-time users might struggle to discover the inline model override capability

Suggested improvements:

• Add version-specific UI change highlights in release notes
• Include tooltip guidance near the model selector:

  Tip: Type any valid model identifier to create custom endpoints  

Cross-Node Model Persistence Issue

Current behavior observation:

1. Adding o1-preview-2024-09-12 in PromptNode_A works
2. Subsequent PromptNode_B shows empty model selection dropdown

Proposed solution architecture:

graph LR
A[Global Settings] --> B(Model Registry)
B --> C{Model Type}
C -->|Built-in| D[Standard Models]
C -->|Custom| E[User-defined Models]
E --> F[Sync Across Nodes]

Loading

This would enable:

• Centralized model configuration management
• Automatic propagation to all nodes
• Version-controlled model presets

Thank you again for your quick support - seeing how rapidly these improvements are being implemented is what makes ChainForge such a joy to use! The new model override workflow has already saved me hours of configuration time.

These suggestions simply come from my excitement to see more users benefit from these great UI enhancements. Keep up the amazing work shaping the future of LLM experimentation tools! 🚀