The scripts in this directory are used to create copies of the 2021 ARPA RFP evaluation rubric to be used by evaluators and then to summarize the results of the evaluation. In the evaluation there were 70 proposals in response to the RFP, each of which was to be evaluated by 5 people, for a total of 350 evaluations.
To prepare to run, simply run the command:
pip install --upgrade -r requirements.txt
There are four separate scripts, although the last three update separate tabs within a single output file:
- create_evaluator_sheets.py - Create evaluation spreadsheets for all evaluators and a spreadsheet with the evaluator/proposal/evaluation mappings.
- detail_reports.py - Create the All Data and Evaluation Status tabs in the summary report spreadsheet.
- summary_reports.py - Create the Summary and Potential Issues tabs in the summary report spreadsheet.
- check_stats.py - Create the StatCheck tab in the summary report spreadsheet.
All input and output files are Google Sheets. IDs and other parameters are defined in the inputs.json file.
This script uses the master input spreadsheet to get evaluator assignments and the README and evaluation rubric pages to use for each evaluator spreadsheet. It then creates one spreadsheet per evaluator, with a README tab and one tab per assigned evaluation (each evaluation tab is a copy of the evaluation rubric page in the input spreadsheet, with information about the specific proposal inserted at the top). At the end of the process an Evaluator Mappings is generated (see example here).
This script uses the Evaluator Mappings file generated above to read all evaluations and output a tab with the status of each evaluation (Evaluation Status tab) as either Complete, Partial or Not Started, and a tab with data on each individual question of each evaluation (All Data tab). See sample here.
This script uses the Evaluator Mappings file generated above as well as the Score Weighting information from the master input spreadsheet to compute overall scores of all the proposals in the Summary tab as well as a list of proposals with potential issues (e.g., a very wide spread of individual scores) in the Potential Issues tab. Only fully completed evaluations are included in calculating these statistics.
This script computes score statistics for all evaluations using a different approach than above and then compares the two results as a check for correctness.