Anurag Jha
← All products

NeuroDiscovery AI · 2026 · Internal · Model training, human feedbackPrototype

Reinforcement learning module

inside NeuroAudit

Reviewers mark a handful of model outputs and say why. A judge has to prove it agrees with them before any training starts. Then the model trains on their feedback, and each round trains on top of the last.

My roleDesigned it and built the prototype in Claude Code against the GRPO control plane API. The training backend belongs to the ML team.

3steps: setup, human feedback, trainingProduct
0.70judge agreement needed before a trainer is spawnedGRPO control plane contract
339 / 339tests passing on the prototype, light and darkBuild notes, Sep 2026
Round 2 trains on top of round 1, on the same axes. Real prototype; the training environment is synthetic.

The challenge

Every rejection a reviewer made in NeuroAudit was lost. Nothing turned reviewer judgment into a better extraction model.

What shipped

A three step loop (setup, human feedback, training) with a judge gate that can refuse to start, a live training workspace modelled on Weights and Biases, and auto evaluation on held out encounters.

What moved

A full loop from reviewer marks to a trained adapter, with two refusal gates and rounds compared on the same axes

The judge has to earn the reviewer's ranking before it is trusted to score anything. If its agreement with the human marks is below 0.70, or the marks have no spread, the server refuses and nothing is spawned.

A

Mark, gate, train, repeat

Each round is a run. Round 2 starts near round 1's plateau, with less headroom, on the same charts.

  1. 01

    Setup

    Name the model, pick the indexes, define the criterion, the prompt and the output format the model must return. The page shows the exact request it will send.

    Setup. Index names blurred.
  2. 02

    Human feedback

    Five samples per encounter. The reviewer marks each out of 10 and gives a reason; the rank the API needs is derived from the marks.

    Marking samples, with the encounter beside them.
  3. 03

    The judge gate

    Agreement and spread update live as you mark. Start training stays locked until both clear their thresholds.

    The loop, explained on the module's home.
  4. 04

    Training workspace

    Reward, format pass rate, judge spread, KL divergence and advantage, per step, with alerts when the judge starts losing discrimination.

    Two rounds on the same axes. Synthetic environment.
    The run table: configuration on the left, outcome on the right.
    The same workspace in light.
    01 / 03
  5. 05

    Auto evaluation

    Held out encounters are scored by the judge, with the best sample's output shown in full.

    Encounters scored and the best sample.

How it was evaluated

  • Judge gate: Pearson agreement between the judge and the reviewer's marks must reach 0.70
  • Spread gate: marks must vary by at least 1.0, or there is no ranking to learn from
  • Format pass rate against the output schema at every step
  • Held out evaluation every 25 steps, with a plateau stop

The prototype runs on a stand in control plane that emits the real API message shapes. The curves on screen are synthetic, so no reward gain is claimed.

What I learned

  • A gate that can refuse is a product feature: it stops a bad training run before it costs anything.
  • Show the request body. Engineers trust a UI that shows exactly what it sends.