Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIG] How to add feedback documentatation #17438

Merged
merged 14 commits into from
Oct 10, 2024
Merged
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions src/content/docs/ai-gateway/evaluations/add-human-feedback.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
---
pcx_content_type: how-to
title: Add Human Feedback
sidebar:
order: 3
badge:
text: Beta
---

Human feedback is a valuable metric to assess the performance of your AI models. By incorporating human feedback, you can gain deeper insights into how the model's responses are perceived and how well it performs from a user-centric perspective. This feedback can then be used in evaluations to calculate performance metrics, driving optimization and ultimately enhancing the reliability, accuracy, and efficiency of your AI application.

Human feedback measures the performance of your dataset based on direct human input. The metric is calculated as the percentage of positive feedback (thumbs up) given on logs, which are annotated in the Logs tab of the Cloudflare dashboard. This feedback helps refine model performance by considering real-world evaluations of its output.

This tutorial will guide you through the process of adding human feedback to your evaluations in AI Gateway.

## 1. Log into the dashboard
daisyfaithauma marked this conversation as resolved.
Show resolved Hide resolved

1. Log into the [Cloudflare dashboard](https://dash.cloudflare.com/) and select your account.
2. Navigate to the AI Gateway section.
daisyfaithauma marked this conversation as resolved.
Show resolved Hide resolved

## 2. Access the Logs tab

1. In the dashboard, go to the Logs tab under the AI Gateway section.
daisyfaithauma marked this conversation as resolved.
Show resolved Hide resolved
2. The Logs tab displays all logs associated with your datasets. These logs show key information, including:
- Timestamp: When the interaction occurred.
- Status: Whether the request was successful, cached, or failed.
- Model: The model used in the request.
- Tokens: The number of tokens consumed by the response.
- Cost: The cost based on token usage.
- Duration: The time taken to complete the response.
- Feedback: Where you can provide human feedback on each log.

## 3. Provide human feedback

1. In the Logs tab, click on the log entry you want to review. This expands the log, allowing you to see more detailed information.
daisyfaithauma marked this conversation as resolved.
Show resolved Hide resolved
2. In the expanded log, you can view additional details such as:
- The user prompt.
- The model's response.
daisyfaithauma marked this conversation as resolved.
Show resolved Hide resolved
- HTTP response details.
- Endpoint information.
3. You will see two icons:
- Thumbs up: Indicates positive feedback.
- Thumbs down: Indicates negative feedback.
4. Click either the thumbs up or thumbs down icon based on how you rate the model's response for that particular log entry.
daisyfaithauma marked this conversation as resolved.
Show resolved Hide resolved

## 4. Evaluate human feedback

After providing feedback on your logs, it becomes part of the evaluation process.
daisyfaithauma marked this conversation as resolved.
Show resolved Hide resolved

When you run an evaluation (as outlined in the [Set Up Evaluations](/ai-gateway/evaluations/set-up-evaluations/) guide), the human feedback metric will be calculated based on the percentage of logs that received thumbs-up feedback.

:::note[Note]

You need to select human feedback as an evaluator to receive its metrics.

:::

## 5. Review results

After running the evaluation,review results on the Evaluations tab.
daisyfaithauma marked this conversation as resolved.
Show resolved Hide resolved
You will be able to see the model's performance based on cost, speed, and now human feedback, represented as the percentage of positive feedback (thumbs up).
daisyfaithauma marked this conversation as resolved.
Show resolved Hide resolved

The human feedback score is displayed as a percentage, showing how manyy of the dataset's responses were rated positively.
daisyfaithauma marked this conversation as resolved.
Show resolved Hide resolved

For more information on running evaluations, refer to the documentation [Set Up Evaluations](/ai-gateway/evaluations/set-up-evaluations/).
Loading