Data Science Lead - job 1 of 4

Job Summary

A company is looking for a Data Science Lead to spearhead the design, experimentation, and optimization of their LLM-driven platform.

Key Responsibilities

• Own experimentation strategy for LLM-powered agents, designing and deploying pipelines for A/B testing and real-time feedback loops
• Build frameworks for analyzing AI trace behavior and iterating on model improvements
• Define key performance indicators (KPIs) that connect product goals with AI behavior

Required Qualifications

• Hands-on experience with LLMs in production
• Strong understanding of evaluation systems for agents and pipelines
• Experience in defining success metrics and building analytics infrastructure
• Ability to navigate ambiguity in fast-paced startup environments
• Collaborative approach with engineering and product teams