Model Trainer (Post-Training and Evals)

Lyzr AI · Bengaluru, India

Model Trainer (Post-Training and Evals)

The job

Take an agentic task that currently runs on a frontier model, fine-tune an open-weight model on it, and prove with evals that the small model does the task better and cheaper. Then do it again on the next task, and the next.

You will work on real enterprise workloads. Some of it runs on customer GPUs, some on ours. You own the model from data through to the eval report that convinces a customer to switch traffic over.

What you will do

• Build training sets from real agent traffic and traces, not scraped benchmarks
• Run LoRA, QLoRA, DoRA, full fine-tunes, CPT, DPO and GRPO depending on what the task needs
• Design task-specific evals for agentic behavior: tool calling accuracy, multi-step reliability, output format adherence, groundedness. Benchmarks are not the bar, the customer's task is
• Get the model serving on vLLM and measure real latency and token cost against the frontier baseline
• Work inside ShadowLM, our open-source fine-tuning SDK, and improve it as you go

What we are looking for

• You have post-trained open-weight models yourself and can talk about what failed, not just what worked
• Comfortable across llama, qwen, mistral, gemma, phi, deepseek and whatever ships next month
• Strong on evals. You know that most fine-tuning projects die because nobody built an honest measurement first
• You can read a paper on Monday and have it running on Wednesday
• Practical about hardware. You know what fits on one GPU and what does not

Nice to have

• Reward modeling or RL for agents
• Quantization and distillation
• Experience getting a model past an enterprise security review