# LLM judges for BPMN process model annotation

> LLM judges label a process model dataset, but their labels only become training data after agreement with human experts is measured.

- Role: Pipeline design and validation
- Period: In progress, 2026
- Status: In progress
- Stack: LLM-as-judge, Rubric design, Human expert validation, Fine-tuning, Python

## Context and problem

A dataset of business process models needs quality labels. Manual expert annotation does not scale, and unchecked model labels are not trustworthy enough to train on.

## Labelling target

The labelling target is the fit between a student's process model and its task, not syntax or linting.

## Approach

- For each item the judge receives the task text, an LLM-generated rubric, a reference solution and the student solution.
- Two datasets: one with reference models, one without.
- Judge outputs are checked for agreement with human experts before they count as labels.
- Validated labels then serve as fine-tuning data for a faster grading model.

## Principle

The model does the repetitive checking. Measured agreement with human experts decides when its labels can be trusted, not the fluency of its output.

## Status

In progress. No agreement figures are published yet.

## Transfer

The same validate-before-trust pattern applies to any LLM-assisted classification or review task in an organisation.