Skip to content

Commit

Permalink
rename to chatvertexai and fix hyphenation
Browse files Browse the repository at this point in the history
  • Loading branch information
dylanbouchard committed Jan 4, 2025
1 parent 957fc67 commit 9d94d74
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions paper/paper.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Furthermore, LangFair is designed for real-world LLM-based systems that require


# Generation of Evaluation Datasets
The `langfair.generator` module offers two classes, `ResponseGenerator` and `CounterfactualGenerator`, which aim to enable user-friendly construction of evaluation datasets for text generation use cases.
The `langfair.generator` module offers two classes, `ResponseGenerator` and \hyphenateddigits[\unknown]{`CounterfactualGenerator`}, which aim to enable user-friendly construction of evaluation datasets for text generation use cases.


### `ResponseGenerator` class
Expand Down Expand Up @@ -90,14 +90,14 @@ When LLMs are used to solve classification problems, traditional machine learnin
# Semi-Automated Evaluation

### `AutoEval` class
To streamline assessments for text generation use cases, the `AutoEval` class conducts a multi-step process (each step is described in detail above) for a comprehensive fairness assessment. Specifically, these steps include metric selection (based on whether FTU is satsified), evaluation dataset generation from user-provided prompts with a user-provided LLM, and computation of applicable fairness metrics. To implement, the user is required to supply a list of prompts and an instance of `langchain` LLM. Below we provide a basic example demonstrating the execution of `AutoEval.evaluate` with a `gemini-pro` instance.^[Note that this example assumes the user has already set up their VertexAI credentials and sampled a list of prompts from their use case prompts.]
To streamline assessments for text generation use cases, the `AutoEval` class conducts a multi-step process (each step is described in detail above) for a comprehensive fairness assessment. Specifically, these steps include metric selection (based on whether FTU is satsified), evaluation dataset generation from user-provided prompts with a user-provided LLM, and computation of applicable fairness metrics. To implement, the user is required to supply a list of prompts and an instance of a `langchain` LLM. Below we provide a basic example demonstrating the execution of `AutoEval.evaluate` with a `gemini-pro` instance.^[Note that this example assumes the user has already set up their VertexAI credentials and sampled a list of prompts from their use case prompts.]


```python
from langchain_google_vertexai import VertexAI
from langchain_google_vertexai import ChatVertexAI
from langfair.auto import AutoEval

llm = VertexAI(model_name='gemini-pro')
llm = ChatVertexAI(model_name='gemini-pro')
auto_object = AutoEval(prompts=prompts, langchain_llm=llm)
results = await auto_object.evaluate()
```
Expand Down

0 comments on commit 9d94d74

Please sign in to comment.