ragas: pass rubric in via RubricScore.SingleTurnPrompt.instruction #211
Conversation
|
@RobotSail Here is what the prompts and internal responses look like at without this PR and with this PR: What should the final prompt look like? There's a lot baked by ragas that's easy to see in the output when this PR is not applied. |
alimaredia
force-pushed
the
pass-in-rubric
branch
from
ad55824 to
f8277b1
Compare
mergify
bot
added
dependencies
There was a problem hiding this comment.
Thanks for this PR @alimaredia.
I'm a little confused by this PR, because it seems like there are a lot of things being changed which may not be intended to be.
Could you please update this PR to only be changing the intended behavior (making sure that the old ragas rubric template can still be used)?
Additionally, we should somehow make the old rubric usage be a toggle-able behavior, since there is reason for which the users may want to use the new rubric.
src/instructlab/eval/ragas.py
Outdated
| @@ -88,29 +96,25 @@ def __init__( | |||
| self.judge_openai_api_key = judge_openai_api_key | |||
|
|
|||
| @staticmethod | |||
| def _validate_dataset(df: DataFrame): | |||
| def validate_dataset(df: DataFrame): | |||
There was a problem hiding this comment.
Why are we exposing this as a public dataset? What is the use-case for this?
src/instructlab/eval/ragas.py
Outdated
| required_keys = {"user_input", "reference"} | ||
| missing_keys = required_keys - set(df.columns) | ||
| if missing_keys: | ||
| required_keys = {"user_input", "reference", "response"} |
There was a problem hiding this comment.
Why are you changing response to be required? This method's only job is to ensure that the given dataset is not missing any of the values which the (InstructLab) evaluation method cannot proceed without.
src/instructlab/eval/ragas.py
Outdated
| required_keys = {"user_input", "reference", "response"} | ||
|
|
||
| columns_list = set(df.columns) | ||
| if not columns_list.issubset(required_keys): |
There was a problem hiding this comment.
This is incorrect. We use a set difference to ensure that the list of required keys are contained within the list of columns provided in the dataset.
In your code, you could have a dataset that passes in {"user_input"} and this method would allow it to be passed because ["user_input"] is mathematically a subset of ["user_input", "reference"], which this method would allow.
| ) | ||
|
|
||
| def run( | ||
| self, | ||
| dataset: List[Sample] | Path, | ||
| student_model: ModelConfig | None = None, | ||
| dataset: List[Sample] | DataFrame, |
There was a problem hiding this comment.
How come we are getting rid of the Path type here?
| run_config = run_config if run_config else self.run_config | ||
| student_openai_client = ( |
Before ragas v0.2.11 RubricScores.rubrics wasn't being applied properly. This commit sets that as the minimum version for this library. A change in v0.2.11 from previous versions was a change in the prompt for domain specific knowledge evaluation with reference. The prompt from previous versions is now explicitly passed in. Signed-off-by: Ali Maredia <[email protected]>
alimaredia
force-pushed
the
pass-in-rubric
branch
from
f8277b1 to
5d47841
Compare
alimaredia
force-pushed
the
pass-in-rubric
branch
from
5d47841 to
d803496
Compare
Refactor RagasEvaluator Class for use for `ilab` interface. Signed-off-by: Ali Maredia <[email protected]>
alimaredia
force-pushed
the
pass-in-rubric
branch
from
d803496 to
1a3f6f6
Compare
In ragas, RubricScores.rubrics isn't being used anywhere except in repr thus it was not being passed into the prompt for the judge to evaluate responses against reference answers.
Passing a string rubric into SingleTurnPrompt.instruction allows us to pass the rubric into the prompt sent
to the judge.