• This role will own the product vision, strategy, and execution for critical evaluation workflows for risk detection & mitigation • Your mandate is twofold: • Design principled evaluation frameworks that determine the right size and shape of holdouts, control groups, and manual review samples—without degrading model performance or decision quality. • Drive automation that meaningfully reduces manual reviews, operational burden, and customer friction, while preserving the labels and signals required to keep models accurate and resilient over time. • Success in this role is measured by sustained reductions in manual review volume, improved approval rates, stable or improving loss performance, and evaluation systems that scale as risk vectors evolve • A typical day involves reviewing how models, holdouts, and manual reviews interact across the decisioning funnel—examining false positives, label coverage, approval lift, and downstream loss impact. • You will work closely with Data Science, Machine Learning, Risk Engineering, and Operations to understand where current evaluation approaches over-sample good users, introduce bias, or create unnecessary friction. • You will define the end-to-end product vision for risk evaluations and learning loops, building a multi-year roadmap that balances statistical rigor, operational efficiency, customer experience, and regulatory expectations. • You will own requirements and execution for systems that: • Dynamically size and manage holdouts • Optimize when and how manual reviews are invoked • Preserve high-quality labels without over-reliance on human review • Enable faster, safer iteration on risk models and policies • You will partner deeply with Fraud & Safety, Trust, Legal, Policy, Customer Support, and Payments Operations to align on decision principles and ensure evaluation strategies are understood, trusted, and actionable across the company. • Collaborate with Payments Operations and Support to reduce manual effort, handle edge cases better, and unlock high-quality decisioning at scale. • Key to success for this role is to partner deeply with other Airbnb platform organizations such as Fraud & Safety, AirCover, Legal, Customer Support, Policy Enforcement & Payments Operations teams to align on the vision and mission and coordinate tactics to deliver mutually aligned outcomes.