Gemma/Gemma_with_Langfun_and_LlamaCpp.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2024 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "tuOe1ymfHZPu"
      },
      "outputs": [],
      "source": [
        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FyE6mpefBU_G"
      },
      "source": [
        "## Guide to using Gemma 2 with Langfun, PyGlove and llama.cpp\n",
        "\n",
        "[Langfun](https://github.com/google/langfun) is a versatile library designed to bridge the gap between natural language processing (NLP) and structured data manipulation. It allows you to effortlessly parse, interpret, and convert unstructured text into well-defined Python objects. This capability is invaluable for applications that require seamless integration between language models and data-driven systems.\n",
        "\n",
        "[PyGlove](https://github.com/google/pyglove) is a powerful library for flexible and scalable configuration management and automated machine learning (AutoML). It provides tools for defining complex schemas and object representations, enabling you to create highly customizable and maintainable code structures. PyGlove's compatibility with Langfun ensures that the objects you generate from natural language prompts are both robust and easy to use.\n",
        "\n",
        "[llama.cpp](https://github.com/ggerganov/llama.cpp) is a C++ implementation of Meta AI's LLaMA and other large language model architectures, designed for efficient performance on local machines or within environments like Google Colab. It enables you to run large language models without needing extensive computational resources.\n",
        "\n",
        "[Gemma](https://ai.google.dev/gemma) is a family of lightweight, state-of-the-art open-source language models from Google. Built from the same research and technology used to create the Gemini models, Gemma models are text-to-text, decoder-only large language models (LLMs), available in English, with open weights, pre-trained variants, and instruction-tuned variants.\n",
        "Gemma models are well-suited for various text-generation tasks, including question-answering, summarization, and reasoning. Their relatively small size makes it possible to deploy them in environments with limited resources such as a laptop, desktop, or your cloud infrastructure, democratizing access to state-of-the-art AI models and helping foster innovation for everyone.\n",
        "\n",
        "By combining `Langfun`, `PyGlove`, `llama.cpp` and `Gemma`, you can create applications that not only understand and process natural language but also convert that understanding into structured, actionable data. This integration allows you to build sophisticated systems that can interact with external APIs, databases, and other services using well-defined objects, streamlining workflows and enhancing functionality.\n",
        "\n",
        "In this guide, you'll learn how to set up your environment, configure the Gemma 2 model, and implement various real-world use cases.\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/gemma-cookbook/blob/main/Gemma/Gemma_with_Langfun_and_LlamaCpp.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FaqZItBdeokU"
      },
      "source": [
        "## Setup\n",
        "\n",
        "### Select the Colab runtime\n",
        "To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model. In this case, you can use a T4 GPU:\n",
        "\n",
        "1. In the upper-right of the Colab window, select **▾ (Additional connection options)**.\n",
        "2. Select **Change runtime type**.\n",
        "3. Under **Hardware accelerator**, select **T4 GPU**.\n",
        "\n",
        "### Gemma setup\n",
        "\n",
        "**Before you dive into the tutorial, let's get you set up with Gemma:**\n",
        "\n",
        "1. **Hugging Face Account:**  If you don't already have one, you can create a free Hugging Face account by clicking [here](https://huggingface.co/join).\n",
        "2. **Gemma Model Access:** Head over to the [Gemma model page](https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315) and accept the usage conditions.\n",
        "3. **Colab with Gemma Power:**  For this tutorial, you'll need a Colab runtime with enough resources to handle the Gemma 9B model. Choose an appropriate runtime when starting your Colab session.\n",
        "4. **Hugging Face Token:**  Generate a Hugging Face access (preferably `write` permission) token by clicking [here](https://huggingface.co/settings/tokens). You'll need this token later in the tutorial.\n",
        "\n",
        "**Once you've completed these steps, you're ready to move on to the next section where you'll set up environment variables in your Colab environment.**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CY2kGtsyYpHF"
      },
      "source": [
        "### Configure your HF token\n",
        "\n",
        "Add your Hugging Face token to the Colab Secrets manager to securely store it.\n",
        "\n",
        "1. Open your Google Colab notebook and click on the 🔑 Secrets tab in the left panel. <img src=\"https://storage.googleapis.com/generativeai-downloads/images/secrets.jpg\" alt=\"The Secrets tab is found on the left panel.\" width=50%>\n",
        "2. Create a new secret with the name `HF_TOKEN`.\n",
        "3. Copy/paste your token key into the Value input box of `HF_TOKEN`.\n",
        "4. Toggle the button on the left to allow notebook access to the secret.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hFhJSxWB5KXd"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from google.colab import userdata\n",
        "\n",
        "# Note: `userdata.get` is a Colab API. If you're not using Colab, set the env\n",
        "# vars as appropriate for your system.\n",
        "os.environ[\"HF_TOKEN\"] = userdata.get(\"HF_TOKEN\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iwjo5_Uucxkw"
      },
      "source": [
        "### Install dependencies\n",
        "\n",
        "You'll need to install a few Python packages for Langfun and PyGlove and some dependencies to interact with HuggingFace along with `llama.cpp` and run the model.\n",
        "\n",
        "Run the following cell to install or upgrade it:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vJsTBy_j4oqj"
      },
      "outputs": [],
      "source": [
        "# Install ccache, a compiler cache to speed up compilations\n",
        "!apt install -y ccache\n",
        "\n",
        "# Install Langfun and PyGlove Python libraries quietly\n",
        "!pip install -q langfun pyglove\n",
        "\n",
        "# Upgrade and install the huggingface_hub library for interacting with Hugging Face\n",
        "!pip install --upgrade -q huggingface_hub\n",
        "\n",
        "# Clone the llama.cpp repository from GitHub\n",
        "!rm -rf llama.cpp\n",
        "!git clone https://github.com/ggerganov/llama.cpp.git\n",
        "\n",
        "# Navigate into the llama.cpp directory and compile the project with CUDA support\n",
        "!cd llama.cpp && make -j8 GGML_CUDA=1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-kQON-KLW2_C"
      },
      "source": [
        "You are compiling the `llama.cpp` binary, which may take some time. Please keep your browser tab active to prevent Colab from shutting down due to inactivity, and return once the compilation is complete to resume working with the notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2_bahJBmwvSp"
      },
      "source": [
        "### Logging into Hugging Face Hub\n",
        "\n",
        "Next, to authenticate and gain access to download the Gemma 2 model, you need to log in to the Hugging Face Hub using your access token."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pd08YwW75gvm"
      },
      "outputs": [],
      "source": [
        "from huggingface_hub import login\n",
        "\n",
        "login(os.environ[\"HF_TOKEN\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_-NAW6VXOeBt"
      },
      "source": [
        "### Downloading the Gemma 2 Model\n",
        "Once you're logged in, you can download the Gemma 2 model files from Hugging Face. The [Gemma 2 model](https://huggingface.co/bartowski/gemma-2-9b-it-GGUF) is available in **GGUF** format, which is optimized for use with `llama.cpp`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MuWXSuqFPSQW"
      },
      "outputs": [],
      "source": [
        "from huggingface_hub import hf_hub_download\n",
        "\n",
        "# Specify the repository and filename\n",
        "repo_id = 'bartowski/gemma-2-9b-it-GGUF'  # Repository containing the GGUF model\n",
        "filename = 'gemma-2-9b-it-Q6_K.gguf'        # The GGUF model file\n",
        "\n",
        "# Download the model file to the current directory\n",
        "hf_hub_download(repo_id=repo_id, filename=filename, local_dir='.')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lr3K9wqMH0K-"
      },
      "source": [
        "### Running the Gemma 2 model using `llama.cpp`\n",
        "\n",
        "You will initialize the model using the `llama.cpp` library by first loading a pre-trained Gemma 2 model from HuggingFace. Next, you'll prompt the model by asking a simple question so that you can test if `llama.cpp` is working with the model that you just downloaded."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VywSmn34_lhL"
      },
      "outputs": [],
      "source": [
        "!# Define your prompt\n",
        "prompt = \"Q: I have 3 apples and I eat 2. How many apples do I have left?\\nA:\"\n",
        "model_path = \"gemma-2-9b-it-Q6_K.gguf\"\n",
        "\n",
        "# Run the model using llama.cpp\n",
        "!./llama.cpp/llama-cli -m \"{model_path}\" -p \"{prompt}\" \\\n",
        "  --color --chat-template gemma --temp 0.7 -ngl 9999"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hD4ykO5YG-Ll"
      },
      "source": [
        "Next, you'll run the server with `llama-server` to serve the Gemma 2 model, allowing for API-based interactions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zjn6oCNlEi2s"
      },
      "outputs": [],
      "source": [
        "!./llama.cpp/llama-server \\\n",
        "  --model \"{model_path}\" \\\n",
        "  --chat-template gemma \\\n",
        "  --host 0.0.0.0 \\\n",
        "  --port 8000 \\\n",
        "  --chat-template gemma \\\n",
        "  --n-gpu-layers 9999 > server.log 2>&1 &\n",
        "\n",
        "# Add a delay to allow the server to warm up before making any API requests to\n",
        "# the server\n",
        "!sleep 30"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZgU1KxkyYFDL"
      },
      "source": [
        "Hint: You can use `tail -f server.log` to preview the command's output if necessary."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "v3hoGJF6Wi8_"
      },
      "outputs": [],
      "source": [
        "from google.colab.output import eval_js\n",
        "\n",
        "# (Optional) View the llama.cpp server interface\n",
        "# Uncomment the line below to generate a URL for the server\n",
        "# print(eval_js(\"google.colab.kernel.proxyPort(8000)\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3zV5b1-XJalg"
      },
      "source": [
        "Note: Uncommenting the print statement will display the proxied URL, which you can open in a new browser tab to interact with the server."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "import-libraries"
      },
      "source": [
        "### Import Libraries\n",
        "\n",
        "Import the necessary Python libraries for `Langfun` and `PyGlove`, which are essential for defining schemas and interacting with the language model. These libraries provide the foundational tools needed to define data schemas and query the language model effectively."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "import-packages"
      },
      "outputs": [],
      "source": [
        "import langfun as lf\n",
        "import pyglove as pg"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "format-prompt"
      },
      "source": [
        "To programmatically add control tokens to any prompt for use with an instruction-tuned model, you can create a function that wraps the prompt with the necessary control tokens. Here’s an example Python function that can be reused to apply this formatting:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "define-prompt-function"
      },
      "outputs": [],
      "source": [
        "def format_gemma_prompt(user_input):\n",
        "    \"\"\"\n",
        "    Formats a prompt for the Gemma instruction-tuned model by adding control tokens.\n",
        "    Here you'll be instructing the model to adhere to Langfun and PyGlove.\n",
        "\n",
        "    Parameters:\n",
        "    user_input (str): The input from the user to be formatted.\n",
        "\n",
        "    Returns:\n",
        "    str: The formatted prompt with Gemma control tokens.\n",
        "    \"\"\"\n",
        "    return f\"\"\"\\\n",
        "    You are an AI assistant designed to work seamlessly with Langfun and PyGlove. Your task is to parse user inputs into complex object representations compatible with these libraries. Always output as an object and nothing else. Do not include any additional text, explanations, or comments.\n",
        "\n",
        "    Guidelines:\n",
        "    Output Format: Always provide your output as an object representation suitable for Langfun and PyGlove.\n",
        "    Consistency: Do not deviate from producing the object. Ensure your output is valid and can be used directly in code.\n",
        "    No Extra Text: Exclude any additional explanations or comments outside of the object.\n",
        "\n",
        "    Technical Details:\n",
        "    Your goal is to help parse user inputs into object representations that can be directly utilized within Langfun and PyGlove code.\n",
        "\n",
        "    <start_of_turn>user\n",
        "    {user_input}\n",
        "    <end_of_turn>\n",
        "    <start_of_turn>model\n",
        "    \"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "using-langfun"
      },
      "source": [
        "### Using Langfun with Gemma 2 from Google AI\n",
        "\n",
        "Langfun allows you to define a clear schema and effortlessly convert natural language inputs into structured objects. These objects can then be directly plugged into external systems, databases, or external APIs, facilitating seamless data flow and integration without the need for manual parsing or data cleaning.\n",
        "\n",
        "First, let's try a simple use case by asking Langfun to represent a simple mathematical statement's final answer using a Python class `Answer`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wQK5MrmxNjOF"
      },
      "outputs": [],
      "source": [
        "class Answer(pg.Object):\n",
        "  result: int\n",
        "\n",
        "prompt = 'The result of one plus two is three'\n",
        "result = lf.query(\n",
        "    prompt=format_gemma_prompt(prompt),\n",
        "    schema=Answer,\n",
        "    lm=lf.llms.LlamaCppRemote(\"http://0.0.0.0:8000\")\n",
        ")\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yv_zEDSHKE6g"
      },
      "source": [
        "To break down what's happening here.\n",
        "\n",
        "* `lf.query` first sends the formatted prompt to the language model and parses the response into the specified schema.\n",
        "\n",
        "* `format_gemma_prompt(prompt)` formats the prompt by adding necessary control tokens and instructions.\n",
        "\n",
        "* `schema=Answer` specifies that the output should conform to the `Answer` schema.\n",
        "\n",
        "* `lm=lf.llms.LlamaCppRemote(\"http://0.0.0.0:8000\")` makes Langfun use the local `llama.cpp` server running the Gemma 2 9B IT quantized model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "thxM1dtFefLP"
      },
      "source": [
        "Next, you'll take a look at real-world use cases demonstrating how you can leverage Langfun and Gemma 2 to build robust applications."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rkh5Jyz5BlBk"
      },
      "source": [
        "### Parsing a Job Description into a Structured Object\n",
        "\n",
        "\n",
        "Automating the recruitment process can significantly enhance efficiency and accuracy in HR departments. By parsing job descriptions into structured objects, you can seamlessly integrate this data into your HR systems, enabling functionalities like resume matching, job postings, and analytics.\n",
        "\n",
        "Let's start first by defining a `Job` schema using PyGlove, then provide a job description as input. Langfun, powered by the Gemma 2 model, processes the text and extracts relevant information, populating the `Job` object accordingly.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "parse-job-description"
      },
      "outputs": [],
      "source": [
        "# Define the Job schema\n",
        "class Job(pg.Object):\n",
        "    title: str\n",
        "    company: str\n",
        "    location: str\n",
        "    responsibilities: list[str]\n",
        "    requirements: list[str]\n",
        "\n",
        "# Job description text\n",
        "job_description = \"\"\"\n",
        "We are looking for a Software Engineer to join our team at TechCorp in San Francisco.\n",
        "Responsibilities include developing software solutions, collaborating with cross-functional teams, and participating in code reviews.\n",
        "Requirements: Bachelor's degree in Computer Science, 3+ years of experience in software development, proficiency in Python and JavaScript.\n",
        "\"\"\"\n",
        "\n",
        "# Parse the job description\n",
        "result = lf.query(\n",
        "    prompt=format_gemma_prompt(job_description),\n",
        "    schema=Job,\n",
        "    lm=lf.llms.LlamaCppRemote(\"http://0.0.0.0:8000\")\n",
        ")\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c4TQL0qNGedG"
      },
      "source": [
        "This structured data can then be used directly in your HR systems or external APIs for various purposes such as job matching and integration with Applicant Tracking Systems (ATS) for streamlined recruitment workflows."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "job-description-explanation"
      },
      "source": [
        "### Enabling Automated Handling of Support Tickets\n",
        "\n",
        "In this second example, you'll create a support ticket by incorporating variables into your prompt. In customer service applications, generating support tickets efficiently is crucial for timely issue resolution. By creating structured support tickets from user inputs, you can integrate seamlessly with ticketing systems, prioritize issues, and assign them to the appropriate teams.\n",
        "\n",
        "Define a `SupportTicket` schema and use variables like `customer_name` and `issue` in your prompt. Langfun processes these variables within the context, generating a structured `SupportTicket` object that can be integrated into your customer service systems or external APIs. This lets you automatically create tickets in a system, easily track ticket statuses and generate reports."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fO19wo73Raje"
      },
      "outputs": [],
      "source": [
        "customer_name = 'John Doe'\n",
        "issue = 'I am unable to log into my account.'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "create-support-ticket"
      },
      "outputs": [],
      "source": [
        "# Define the SupportTicket schema\n",
        "class SupportTicket(pg.Object):\n",
        "    customer_name: str\n",
        "    issue: str\n",
        "    steps_taken: list[str]\n",
        "    resolution: str | None\n",
        "\n",
        "# Prompt with variables\n",
        "prompt = 'Create a support ticket for {{customer_name}} who reported: \"{{issue}}\" and propose steps to be taken along with a final resolution'\n",
        "\n",
        "# Generate the support ticket\n",
        "result = lf.query(\n",
        "    prompt=format_gemma_prompt(prompt),\n",
        "    customer_name=customer_name,\n",
        "    issue=issue,\n",
        "    schema=SupportTicket,\n",
        "    lm=lf.llms.LlamaCppRemote(\"http://0.0.0.0:8000\")\n",
        ")\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iy3mKTOzHbpK"
      },
      "source": [
        "In a scenario where your customer service team receives numerous inquiries daily, by automating the creation of support tickets, you can ensure that each issue is documented consistently and efficiently. The structured `SupportTicket` objects can be directly fed into your CRM or ticketing software, facilitating faster response times and better customer satisfaction.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "support-ticket-explanation"
      },
      "source": [
        "### Parsing Customer Feedback into Structured Data\n",
        "\n",
        "Understanding customer feedback is essential for improving products and services. You'll start by first defining a `Feedback` schema and provide customer feedback text. Langfun processes the feedback, determining sentiment and identifying key topics, and outputs a structured `Feedback` object. This allows you to parse customer feedback into structured data, aiding in sentiment analysis, report generation and topic modeling."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "parse-feedback"
      },
      "outputs": [],
      "source": [
        "# Define the Feedback schema\n",
        "from typing import Literal\n",
        "\n",
        "class Feedback(pg.Object):\n",
        "    customer_id: int\n",
        "    feedback_text: str\n",
        "    sentiment: Literal['positive', 'negative', 'mixed']\n",
        "    topics: list[str]\n",
        "\n",
        "# Customer feedback text\n",
        "feedback_text = 'I love the new features in the app, but it crashes frequently when I try to upload photos.'\n",
        "\n",
        "# Context for the model\n",
        "context = f'Parse the following customer (id: 1) feedback into an object representation (Feedback): {feedback_text}'\n",
        "\n",
        "# Parse the feedback\n",
        "result = lf.query(\n",
        "    prompt=format_gemma_prompt(context),\n",
        "    schema=Feedback,\n",
        "    lm=lf.llms.LlamaCppRemote(\"http://0.0.0.0:8000\")\n",
        ")\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fM2w-_agEVIL"
      },
      "source": [
        "This example shows how the model can extract valuable insights from customer feedback, which is crucial for product improvement. Imagine you manage a mobile app with a large user base. By systematically parsing feedback, you can identify which features are well-received and which areas require attention. For instance, detecting frequent crashes related to photo uploads allows your development team to prioritize bug fixes, enhancing the overall user experience.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "feedback-explanation"
      },
      "source": [
        "### Classifying Support Tickets Using `Literal` Types\n",
        "\n",
        "Efficiently categorizing and prioritizing support tickets can significantly enhance your ticket triaging system. Proper classification ensures that issues are addressed by the right teams promptly, improving resolution times and customer satisfaction.\n",
        "\n",
        "Define a `Ticket` schema with specific categories and priorities. Provide a support ticket description, and Langfun processes it to classify the ticket accordingly. Next, let's perform classification tasks using `Literal` types from the `typing` module, which is useful for ticket triaging systems."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "classify-ticket"
      },
      "outputs": [],
      "source": [
        "# Define the Ticket schema\n",
        "class Ticket(pg.Object):\n",
        "    id: int\n",
        "    issue: str\n",
        "    priority: Literal['low', 'medium', 'high']\n",
        "    category: Literal['billing', 'technical', 'account', 'other']\n",
        "    assigned_to: str | None\n",
        "\n",
        "# Support ticket description\n",
        "ticket_description = 'Ticket ID: 12345. Customer reports that they are unable to update their billing information on the website.'\n",
        "\n",
        "# Context for classification\n",
        "context = f'Classify the following support ticket into the Ticket object: {ticket_description}'\n",
        "\n",
        "# Classify the ticket\n",
        "result = lf.query(\n",
        "    prompt=format_gemma_prompt(context),\n",
        "    schema=Ticket,\n",
        "    lm=lf.llms.LlamaCppRemote(\"http://0.0.0.0:8000\")\n",
        ")\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0L4XZrBcEjM-"
      },
      "source": [
        "In a customer service center, tickets often vary widely in nature and urgency. By automatically classifying tickets, you ensure that billing issues are handled by a finance team, technical problems by the IT department, and so forth. Prioritizing tickets based on their severity helps in managing workloads effectively and ensuring that high-priority issues are resolved swiftly. This demonstrates the model's ability to categorize and prioritize support tickets automatically.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "workout-plan-explanation"
      },
      "source": [
        "###  Chain-of-Thought Reasoning\n",
        "\n",
        "Chain-of-thought (CoT) reasoning involves breaking down complex problems into a series of logical steps. This approach enhances the model's ability to solve intricate tasks by systematically reasoning through each component.\n",
        "\n",
        "In this example, you'll demonstrate the model's capability in performing CoT reasoning for a simple puzzle.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sYiTwC8S_ypU"
      },
      "outputs": [],
      "source": [
        "from typing import Optional, List\n",
        "\n",
        "class Step(pg.Object):\n",
        "  description: str\n",
        "  step_output: float\n",
        "\n",
        "class Solution(pg.Object):\n",
        "  question: str\n",
        "  steps: list[Step]\n",
        "  final_answer: int\n",
        "\n",
        "\n",
        "# Puzzle question\n",
        "question = (\n",
        "  'Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. '\n",
        "  'She sells the remainder at the farmers\\' market daily for $2 per fresh duck egg. '\n",
        "  'How much in dollars does she make every day at the farmers\\' market? '\n",
        ")\n",
        "\n",
        "# Assume gemma2_llm is your initialized Gemma 9b model\n",
        "result = lf.query(\n",
        "    prompt=format_gemma_prompt(question),\n",
        "    schema=Solution,\n",
        "    lm=lf.llms.LlamaCppRemote(\"http://0.0.0.0:8000\")\n",
        ")\n",
        "print(result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "financial-calculations-explanation"
      },
      "source": [
        "This demonstrates the model's capability in performing chain-of-thought reasoning.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A90firK-Aa0W"
      },
      "source": [
        "In this notebook, you've explored how to leverage **Langfun**, **PyGlove**, **llama.cpp** and the **Gemma 2** model from **Hugging Face** to address a variety of real-world scenarios. Through practical use cases—including parsing job descriptions, creating support tickets, analyzing customer feedback, classifying support tickets, and performing chain-of-thought reasoning—you've seen how Langfun effortlessly converts unstructured text into structured, actionable objects. This showcases the model's effectiveness in real-world scenarios, enhancing the capabilities of your applications and seamlessly integrating with external systems.\n",
        "\n",
        "Feel free to extend these examples to fit your specific needs and explore the full potential of Langfun and Gemma in your projects!\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "Gemma_with_Langfun_and_LlamaCpp.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}