Workflow Stages

It’s important to consider Rich Context in the sense of a workflow: ML models solve specific use cases at different stages of the workflow. So there are needs for multiple kinds of modeling to be researched and evaluated. The structure of this leaderboard competition is capable of managing multiple use cases – with one leaderboard for each identified use case – as an ongoing, parallelized research effort.

Overall, a three-step process is being used to apply the results of this competition and extend the corpus:

1. Track progress for specific machine learning use cases, based on the current corpus.
2. Apply leading ML models to identify datasets in a broader set of research publications.
3. Have the publication authors explicitly confirm or reject the inferred dataset annotations.

The latter step introduces opportunities to use semi-supervised learning (aka, “human-in-the-loop” approaches) to improve research metadata. For example, RePEc communications help provide means for that annotation process and feedback directly from authors.

See also:

• Human-in-the-loop AI for scholarly infrastructure
• New initiative to help with discovery of dataset use in scholarly work
• Active Learning Use Cases
• Coleridge Initiative: Publication Submission