Applied ML engineer, Evals & data (5+ Years, 60L)

This role is for one of our partner companies, YC backed

About the role

We’re looking for a Senior Applied ML Engineer to own the quality and evaluation systems behind an AI video agent.

You’ll be responsible for answering a difficult question: how do we know the agent is actually getting better?

You’ll turn subjective agent behavior into measurable signals and build the datasets, evals, graders, and feedback loops needed to continuously improve production performance.

This is an applied ML role for someone who has shipped LLM or agentic systems to real users and understands what happens after the initial prototype works.

You’ll work directly with the founders and product team.

What you'll own

• Agent quality and evaluation: Define measurable standards for useful, reliable agent behavior across the video editing workflow.
• Evaluation systems: Build offline and online evals, model-based graders, human review systems, and automated quality checks.
• Production datasets: Turn user interactions, telemetry, failures, and human feedback into reliable datasets for model and product iteration.
• Multimodal evaluation: Measure retrieval and agent performance across video, audio, images, and graphics while working within real-time constraints.
• Regression testing: Build release gates and regression suites that prevent quality degradation as models, prompts, tools, and workflows change.
• Model selection: Evaluate and select models based on production performance across quality, latency, reliability, and cost.
• Agent learning loops: Build systems to evaluate and improve next-action prediction, creator memory, brand alignment, pacing, and technical output quality.

You'll be a great fit if

• Have shipped an LLM agent or agentic product to real users and owned its production quality.
• Have built evals for LLM or agent behavior, translating subjective quality requirements into metrics, graders, and test suites.
• Have created datasets from production usage, including telemetry, human feedback, failures, or user behavior.
• Know how to evaluate models across quality, latency, reliability, and cost rather than optimizing for benchmark performance alone.
• Have worked on problems where outputs are inherently subjective, such as preference, style, creative quality, ranking, or personalization.
• Are comfortable working across applied research and product engineering, turning uncertain findings into systems that ship.
• Are strong in Python or TypeScript and can build production-quality systems that support rapid model iteration.

Why this role is interesting

Video agents create an unusually difficult ML evaluation problem.

There often isn’t a single objectively correct output. A good result depends on context, creator intent, brand, pacing, visual quality, and what the user is trying to accomplish.

You’ll build the systems that measure those behaviors, identify where the agent fails, and systematically improve its performance.

If you’ve built production agent systems and enjoy turning messy model behavior into measurable engineering problems, we’d love to hear from you.