Python: How the Semantic Kernel handles the system prompt under the hood

[lucasmsoares96]

When analyzing the system prompt examples in the semantic kernel, some things were not clear.

python/samples/concepts/chat_completion/chat_gpt_api.py

     1  # Copyright (c) Microsoft. All rights reserved.
     2
     3  import asyncio
     4
     5  from semantic_kernel import Kernel
     6  from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion
     7  from semantic_kernel.contents import ChatHistory
     8  from semantic_kernel.functions import KernelArguments
     9
    10  system_message = """
    11  You are a chat bot. Your name is Mosscap and
    12  you have one goal: figure out what people need.
    13  Your full name, should you need to know it, is
    14  Splendid Speckled Mosscap. You communicate
    15  effectively, but you tend to answer with long
    16  flowery prose.
    17  """
    18
    19  kernel = Kernel()
    20
    21  service_id = "chat-gpt"
    22  kernel.add_service(OpenAIChatCompletion(service_id=service_id, ai_model_id="gpt-3.5-turbo"))
    23
    24  settings = kernel.get_prompt_execution_settings_from_service_id(service_id)
    25  settings.max_tokens = 2000
    26  settings.temperature = 0.7
    27  settings.top_p = 0.8
    28
    29  chat_function = kernel.add_function(
    30      plugin_name="ChatBot",
    31      function_name="Chat",
    32      prompt="{{$chat_history}}{{$user_input}}",
    33      template_format="semantic-kernel",
    34      prompt_execution_settings=settings,
    35  )
    36
    37  chat_history = ChatHistory(system_message=system_message)
    38  chat_history.add_user_message("Hi there, who are you?")
    39  chat_history.add_assistant_message("I am Mosscap, a chat bot. I'm trying to figure out what people need")
    40  chat_history.add_user_message("I want to find a hotel in Seattle with free wifi and a pool.")
    41
    42
    43  async def chat() -> bool:
    44      try:
    45          user_input = input("User:> ")
    46      except KeyboardInterrupt:
    47          print("\n\nExiting chat...")
    48          return False
    49      except EOFError:
    50          print("\n\nExiting chat...")
    51          return False
    52
    53      if user_input == "exit":
    54          print("\n\nExiting chat...")
    55          return False
    56
    57      answer = await kernel.invoke(chat_function, KernelArguments(user_input=user_input, chat_history=chat_history))
    58      chat_history.add_user_message(user_input)
    59      chat_history.add_assistant_message(str(answer))
    60      print(f"Mosscap:> {answer}")
    61      return True
    62
    63
    64  async def main() -> None:
    65      chatting = True
    66      while chatting:
    67          chatting = await chat()
    68
    69
    70  if __name__ == "__main__":
    71      asyncio.run(main())

Analyzing this code, on line 32 a prompt is defined that interpolates two variables: chat_history and user_input. This prompt will be used to create a function and add it to the kernel.

    29  chat_function = kernel.add_function(
    30      plugin_name="ChatBot",
    31      function_name="Chat",
    32      prompt="{{$chat_history}}{{$user_input}}",      #    <<<<<<
    33      template_format="semantic-kernel",
    34      prompt_execution_settings=settings,
    35  )

Then a ChatHistory is defined and some messages are added to it.

    37  chat_history = ChatHistory(system_message=system_message)
    38  chat_history.add_user_message("Hi there, who are you?")
    39  chat_history.add_assistant_message("I am Mosscap, a chat bot. I'm trying to figure out what people need")
    40  chat_history.add_user_message("I want to find a hotel in Seattle with free wifi and a pool.")

And finally, on line 57 a message is sent to the OpenAI API passing user_input and chat_history as KernelArguments:

    57      answer = await kernel.invoke(chat_function, KernelArguments(user_input=user_input, chat_history=chat_history))

which will be replaced on line 32:

    32      prompt="{{$chat_history}}{{$user_input}}",

This way it seems that the semantic kernel adds the ChatHistory messages as direct text inside a single message with user role instead of sending a list of messages with different user, system and assistant roles for each message, as demonstrated in the documentation:

Create chat completion
Quickstart

Am I right? If so, how do I actually send a message with system role to the OpenAI API? Is there any example about this? If not, how does the Semantic Kernel handle this role conversion? I didn't find anything in the documentation or examples about this.

[moonbox3]

Hi @lucasmsoares96, thanks for your question. You can send a system role message to the OpenAI API in two ways:

• By constructing the ChatHistory object constructor with the system_message kwarg
• By creating a ChatHistory object and adding the system message via the add_system_message("<some string here>") method

Note: we recently refactored our concept samples so the sample you were referring to chat_gpt_api.py is now called simple_chatbot_kernel_function.py.

Let's look at how the system message is handled. If I start out with this sample and ask a question "Why is the sky blue in one sentence?" you can see that when we form the messages dictionary for the OpenAI request, we have:

Once the response is returned, and we ask another question, we see the following messages with their roles in the dictionary:

We can see that as the chat history grows, we keep that context as the user continues to add their input. This is why the kernel function's prompt is configured as prompt="{{$chat_history}}{{$user_input}}".

It does look like the prompt would only contain "direct text" as you mentioned, but because the ChatHistory knows what the underlying roles are, the prompt is first rendered depending on the type of prompt template (Semantic Kernel, Jinja2, or Handlebars) and then it is converted again back into a role to message dictionary once we send it to the model.

One clarification based on This prompt will be used to create a function and add it to the kernel. is that the prompt specified as part of the kernel function is what is rendered each time that function is run. So it isn't that the prompt leads to the function, but rather the function has this prompt format, which continuously renders the current chat history first, and then includes the user message at the end (as we see in the screenshots above).

Does this help answer your question?

[moonbox3]

Hi @lucasmsoares96, please feel free to re-open this issue if you need further help around your original question. Thank you.

[lucasmsoares96]

Thank you very much for the response, @moonbox3. It clarified a lot, but there are still some gaps. For example, in the screenshots you provided, the function's prompt didn't appear in the message history. This makes sense since it's a simple prompt, but how does the model manage to execute a more complex prompt like ./concepts/search/google_search_plugin.py without adding it to the history?

    54      # Following example demonstrates the use of the plugin within a semantic function
    55      prompt = """
    56      Answer the question using only the data that is provided in the data section.
    57      Do not use any prior knowledge to answer the question.
    58      Data: {{WebSearch.SearchAsync "What is semantic kernel?"}}
    59      Question: What is semantic kernel?
    60      Answer:
    61      """
    62
    63      qna = kernel.add_function(
    64          plugin_name="qa",
    65          function_name="qna",
    66          prompt=prompt,
    67          prompt_execution_settings=PromptExecutionSettings(temperature=0.2),
    68      )