Context and problem
A dataset of business process models needs quality labels. Manual expert annotation does not scale, and unchecked model labels are not trustworthy enough to train on.
Labelling target
The labelling target is the fit between a student’s process model and its task, not syntax or linting.
Approach
- For each item the judge receives the task text, an LLM-generated rubric, a reference solution and the student solution.
- Two datasets: one with reference models, one without.
- Judge outputs are checked for agreement with human experts before they count as labels.
- Validated labels then serve as fine-tuning data for a faster grading model.
Principle
The model does the repetitive checking. Measured agreement with human experts decides when its labels can be trusted, not the fluency of its output.
Status
In progress. No agreement figures are published yet.
Transfer
The same validate-before-trust pattern applies to any LLM-assisted classification or review task in an organisation.