Skip to content

Development

Jump to bottom
Atomie CHEN edited this page Jun 30, 2024 · 2 revisions

Example scripts are placed in examples folder.

Sync & Async API

Using OpenAIClient

Each API method of OpenAIClient returns a Requestor, and you can execute its call() or acall() to get synchronous or asynchronous API calls.

Synchronous API usage:

from handyllm import OpenAIClient
with OpenAIClient(api_key='<your-key>') as client:
    response = client.chat(
      	model="gpt-4-turbo",
      	messages=[{"role": "user", "content": "please tell me a joke"}]
    ).call()  ## note .call() here
    print(response['choices'][0]['message']['content'])

Asynchronous API usage:

async with OpenAIClient('async', api_key='<your-key>') as client_async:
    response = await client_async.chat(
      	model="gpt-4-turbo",
      	messages=[{"role": "user", "content": "please tell me a joke"}]
    ).acall()  ## note .acall() here
    print(response['choices'][0]['message']['content'])

You can instantiate a client that supports both modes:

client = OpenAIClient('sync')  ## only supports sync APIs
client = OpenAIClient('async')  ## only supports async APIs
client = OpenAIClient('both')  ## supports both versions

Legacy: Using OpenAIAPI proxy

Important

This is not recommended anymore. Use OpenAIClient instead.

Under the hood it connects to a module client and only provides synchronous APIs, without call().

from handyllm import OpenAIAPI
OpenAIAPI.api_key = '<your-key>'
response = OpenAIAPI.chat(
    model="gpt-4-turbo",
    messages=[{"role": "user", "content": "please tell me a joke"}]
)  ## no .call() here
print(response['choices'][0]['message']['content'])

OpenAI API Request (Beyond OpenAI)

Any endpoint that provides APIs compatible with OpenAI API specifications are supported, for example Moonshot AI API and other self-hosted endpoints.

Azure APIs, which unfortunately do not exactly follow OpenAI specifications, are seamlessly supported.

Endpoints

Each API request will connect to an endpoint along with some API configurations, which include:

Description Value
api_type API type. Defaults to openai, which means OpenAI API compatible. str: openai / azure
api_base API base url. Defaults to OpenAI base url. str
api_key API key. str
organization Organization. str
api_version API version. Must be provided for Azure end-points. str
model_engine_map Map model name to engine name. Useful for Azure end-points if you have custom model names. dict

An Endpoint object contains these information. An EndpointManager acts like a list and can be used to rotate the next endpoint. See test_endpoint.py.

Methods for configuring endpoint info (values will be inferred in top-town order):

Configuration method Description
API keyword parameters e.g.: chat(api_key='xxx', ...)
API endpoint keyword parameter Providing an Endpoint, e.g.: chat(endpoint=MyEndpoint)
API endpoint_manager keyword parameter Providing an EndpointManager, e.g.: chat(endpoint_manager=MyEndpointManager)
OpenAIClient instance (or OpenAIAPI) variables e.g.: client.api_key = 'xxx' / OpenAIAPI.api_key = 'xxx'
Environment variables OPENAI_API_KEY, OPENAI_ORGANIZATION/OPENAI_ORG_ID, OPENAI_API_BASE, OPENAI_API_TYPE, OPENAI_API_VERSION, MODEL_ENGINE_MAP.

Tip

Azure OpenAI APIs are supported: Specify api_type='azure', and set api_base and api_key accordingly. Set model_engine_map if you want to use model parameter instead of engine/deployment_id. See test_azure.py. Please refer to Azure OpenAI Service Documentation for details.

Logger

You can pass custom logger and log_marks (a string or a collection of strings) to chat/completions to get input and output logging.

Timeout control

This toolkit supports client-side timeout control:

from handyllm import OpenAIClient
client = OpenAIClient()
prompt = [{
    "role": "user",
    "content": "please tell me a joke"
    }]
response = client.chat(
    model="gpt-3.5-turbo",
    messages=prompt,
    timeout=10
    ).call()
print(response['choices'][0]['message']['content'])

Stream response

Stream response of chat/completions/finetunes_list_events can be achieved using steam parameter:

from handyllm import OpenAIClient, stream_chat

client = OpenAIClient()
response = client.chat(
    model="gpt-3.5-turbo",
    messages=prompt,
    timeout=10,
    stream=True
    ).call()

# you can use this to stream the response text
for text in stream_chat(response):
    print(text, end='')

# or you can use this to get the whole response
# for chunk in response:
#     if 'content' in chunk['choices'][0]['delta']:
#         print(chunk['choices'][0]['delta']['content'], end='')

Supported APIs

  • chat
  • completions
  • edits
  • embeddings
  • models_list
  • models_retrieve
  • moderations
  • images_generations
  • images_edits
  • images_variations
  • audio_speech
  • audio_transcriptions
  • audtio_translations
  • files_list
  • files_upload
  • files_delete
  • files_retrieve
  • files_retrieve_content
  • finetunes_create
  • finetunes_list
  • finetunes_retrieve
  • finetunes_cancel
  • finetunes_list_events
  • finetunes_delete_model

Please refer to OpenAI official API reference for details.

Audio Speech (TTS)

The response of non-stream mode is the binary content:

response = client.audio_speech(
    model='tts-1',
    input="Hello, world! oh yes. This is a test. Sync speech no-stream version.",
    voice='alloy',
).call()
with open('output-sync.mp3', 'wb') as f:
    f.write(response)

Stream mode:

from handyllm import stream_to_file, astream_to_file

response = client.audio_speech(
    model='tts-1',
    input="Hello, world! oh yes. This is a test. Sync speech stream version.",
    voice='alloy',
    stream=True,
    chunk_size=1024,
).call()
stream_to_file(response, 'output-sync-stream.mp3')

response = await client.audio_speech(
    model='tts-1',
    input="Hello, world! oh no. This is a test. Async speech stream version.",
    voice='alloy',
    stream=True,
    chunk_size=1024,
).acall()
await astream_to_file(response, 'output-async-stream.mp3')

Note: when using Azure, Azure TTS API needs both deployment_id and model arguments (see Azure OpenAI text-to-speech quickstart):

with OpenAIClient(
    api_type='azure', 
    api_base=os.getenv("AZURE_OPENAI_ENDPOINT"), 
    api_key=os.getenv("AZURE_OPENAI_KEY"), 
    api_version=os.getenv("AZURE_OPENAI_API_VERSION")
    ) as client:
    response = client.audio_speech(
        deployment_id='tts',  # replace with your deployment name
        model='tts-1',  # required: the model you choose
        input="Hello, world! oh yes. This is a test. Sync speech no-stream version.",
        voice='alloy',
    ).call()
    with open('output-sync-azure.mp3', 'wb') as f:
        f.write(response)

Chat Prompt

Important

This is the legacy documentations. Please refer to hprompt files.

Prompt Conversion

PromptConverter can convert this text file prompt.txt into a structured prompt for chat API calls:

$system$
You are a helpful assistant.

$user$
Please help me merge the following two JSON documents into one.

$assistant$
Sure, please give me the two JSON documents.

$user$
{
    "item1": "It is really a good day."
}
{
    "item2": "Indeed."
}
%output_format%
%misc1%
%misc2%

from handyllm import PromptConverter
converter = PromptConverter()

# `msgs` can be used as the `messages` parameter for OpenAI API
msgs = converter.rawfile2msgs('prompt.txt')

# variables wrapped in %s can be replaced at runtime
new_msgs = converter.msgs_replace_variables(
    msgs, 
    {
        r'%misc1%': 'Note1: do not use any bad word.',
        r'%misc2%': 'Note2: be optimistic.',
    }
)

Important

About the prompt format, each role key (e.g. $system$ / $user$ / $assistant) should be placed in a separate line.

Substitute

PromptConverter can also substitute placeholder variables like %output_format% stored in text files to make multiple prompts modular. A substitute map substitute.txt looks like this:

%output_format%
Please output a SINGLE JSON object that contains all items from the two input JSON objects.

%variable1%
Placeholder text.

%variable2%
Placeholder text.

from handyllm import PromptConverter
converter = PromptConverter()
converter.read_substitute_content('substitute.txt')  # read substitute map
msgs = converter.rawfile2msgs('prompt.txt')  # variables are substituted already
Clone this wiki locally