Ads Harness Engineer

Circle Labs is the company behind shapes.inc, a place to collaborate. We’re building multiplayer AI: shapes are collaborative assistants that take part in group conversations and do things with you and your people. Our next work includes helping groups set shared goals, delegate to specialist shapes and external agents, and follow progress through to completion. Making things, figuring things out, and sharing plans give people more reasons to spend time together.

We’re a small team in San Francisco. We use the product every day, talk to users, and work across design, engineering, and research.

Build the experiments and tools that tell us whether advertising is useful in a shared conversation on shapes.inc. Working with our team in San Francisco, you will study how intent, candidate ads and generated presentation fit the moment when people are spending time or doing things together. You will turn those questions into repeatable evaluations and help research and engineering distinguish meaningful quality improvements from changes in a single metric.

• Build a reusable evaluation harness for intent interpretation, ad matching and generated presentation, with versioned inputs and reproducible results.
• Create evaluation cases from authorized sources that cover different preferences within a group, changing plans, shared purchase decisions, repeated recommendations, and situations without a useful ad match.
• Define and calibrate measures of relevance, factual support, repetition and conversational fit using both human review and automated checks.
• Run controlled comparisons of prompts, AI Engines and matching approaches; inspect disagreements and investigate why a result changed.
• Connect offline findings with production feedback and serving outcomes while accounting for missing inventory, provider failures, latency and evaluation cost.
• Work with the Ads Lead to turn research findings into regression checks and practical rollout recommendations.

• Strong programming skills and experience building reliable tools for experiments, data analysis or system evaluation.
• Ability to turn an ambiguous quality question into a measurable test and explain what the result does and does not establish.
• Practical understanding of experimental design, sampling and sources of bias in evaluation data.
• Experience evaluating AI systems, search, recommendations or another product where quality requires more than a simple correct answer.
• Careful judgment when working with conversation data and the ability to communicate findings clearly to product and engineering teams.

• Experience with ad relevance, contextual matching or recommendation evaluation.
• Experience building human review workflows and calibrating automated evaluators against human judgments.
• Familiarity with prompt experiments, generated-content evaluation or measuring quality in conversations with multiple people.

• Competitive pay and equity
• Health, dental, and vision coverage
• Access to frontier AI Engines

Email jobs@shapes.inc with the role you’re interested in and the most impressive thing you’ve built or studied. Include links to relevant projects, a portfolio, or research, and tell us what you contributed.

Email jobs@shapes.inc