Circle is hiring a remote Lead Engineer AI Platform!
Circle, a fully remote company of roughly 270 people spread across more than 30 countries, is hiring a Lead Engineer for its AI Platform group. The position sits with the AI Quality engineering team, whose mandate is not model research or training but the measurement and diagnostics layer around production AI systems. This is a hands-on player-coach role: the successful candidate will write code while also setting technical direction and managing a growing team. The core problem is trust — determining whether Circle's planners, tool-calling agents, and sub-agents genuinely perform well across single calls and extended multi-turn conversations, then pinpointing where quality degrades and what to do about it. The work spans evaluation pipelines and scoring, tracing failures through planning and tool execution, dataset construction with human and synthetic data, and structured experimentation on prompts, models, caching, and routing to improve cost and latency. Close collaboration with the AI Core engineering team is central, since the goal is giving product engineers real confidence that their changes improve quality. The environment is asynchronous, documentation-heavy, and deliberately fast-moving, with significant ambiguity and a high volume of experiments — many of which will fail early on. Compensation is benchmarked to U.S. rates globally, with equity and 35 days of paid time off, plus twice-yearly company retreats. English proficiency at CEFR C2 or ILR Level 5 is required.
Key responsibilities
- Own the evaluation infrastructure end to end, including CI/CD pipelines, scorers, and datasets covering everything from a single tool call to a full multi-turn agent conversation
- Trace and diagnose failures across the agent pipeline, distinguishing planning problems from execution problems and isolating tool-selection and tool-trajectory issues in complex product areas
- Convert diagnostic findings into prototypes that either resolve the issue directly or surface clear priorities for the AI Core engineering team
- Build and expand evaluation datasets by standing up annotation workflows and generating simulated or AI-produced conversations to cover product surface faster than manual labeling
- Design and run structured experiments across prompts, models, and the agent harness, including benchmarking new and open-source models against the production baseline
- Create the decision framework for when to switch models, and pursue latency and cost reductions through model swaps, caching, and routing based on plan complexity
- Lead and grow the AI Quality engineering team, setting technical direction and managing day-to-day priorities while remaining hands-on in the codebase
- Partner closely with AI Core engineering so product teams have confidence that their changes measurably improve quality
Key requirements
- Seven or more years building and shipping production software, ideally including LLM-powered agents that take real actions in a product
- Direct experience with complex, tool-using agent systems — multiple tools, planning or orchestration layers, sub-agents — rather than simple single-turn assistants, with a concrete shipped example and an explanation of how its quality was validated
- Comfort with Ruby on Rails and Python, or the ability to become productive in them quickly; Rails underpins the production system and the agent stack, while Python is valuable for data and evaluation work
- Prior experience building evaluation or observability infrastructure for ML or AI systems, such as eval pipelines, scorers, dashboards, or CI/CD for evals, and familiarity with frameworks like Braintrust, LangSmith, or comparable tools
- Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation, including turning raw examples into trustworthy datasets via human annotation, generated data, or simulation
- A fast learner who experiments aggressively and lets results settle debates, including accepting that many early experiments will not pan out
- Comfort operating amid ambiguity and picking up new technologies as projects demand
- Strong alignment with Circle's company values
- English proficiency at CEFR Level C2 or ILR Level 5 across speaking, writing, and reading
- Seniority: Lead
- Work mode: Remote
- Key skills: Ruby on Rails, Python, LLM agent evaluation, Observability infrastructure, CI/CD for evals, Dataset design, Annotation workflows, Prompt and model experimentation, Model routing and caching, Braintrust, LangSmith, Team leadership
Want more jobs?
Join our mailing list to receive updates about new data science and Lead Engineer AI Platform jobs.
