This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more.

Role Description

As the AI Automation Engineer in the AI CoE, you will be instrumental in standing up the initial automation framework for AI Quality. You’ll work closely with the AI Quality Lead and developers to build, run, and scale automated test suites for AI models and agents.

• Develop and maintain automated test frameworks for LLM, RAG, agentic, and classic ML systems
• Implement golden/adversarial test cases (functional, safety, bias, prompt injection, refusal correctness, tool use)
• Integrate evaluation tools (promptfoo, Ragas) into CI/CD pipelines and support batch quality assessments
• Author and validate OpenAPI/JSON Schema contracts for model inputs/outputs; enforce schema checks in pipelines
• Instrument observability metrics (latency, success rate, cost, drift) using OpenTelemetry and export to dashboards
• Support data QA, lineage tracking, and drift monitoring (Evidently, OpenLineage)
• Collaborate with AI developers, SDETs, and SRE to troubleshoot, optimize, and document automation processes
• Contribute to red-team playbooks and security testbooks mapped to OWASP/ATLAS risks

Qualifications

• Experience in test automation, software engineering, or quality assurance (AI/ML experience is a plus, but not required)
• Proficiency in Python, JavaScript/Node.js, or similar programming languages
• Familiarity with CI/CD, version control, and cloud-based deployment environments
• Experience with test automation tools (promptfoo, Ragas, Postman)
• Understanding of AI model evaluation metrics (faithfulness, answer relevancy, hallucination, refusal correctness)
• Basic knowledge of fairness, safety, and security testing for AI systems
• Experience with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
• Interest in Responsible AI, bias mitigation, and regulatory compliance
• Bachelor’s degree in Computer Science, Engineering, or related field, or equivalent practical experience

Requirements

• Flexible working arrangements available

Benefits

• Programs and plans for a healthy mind, body, wallet, and life
• Variety of health coverage options
• Wellbeing and support programs
• Retirement plans
• Vacation and sick leave
• Maternity, paternity & adoption leave
• Continuing education and training
• Voluntary benefit options