Upbound
Upbound delivers a single point of control to manage all your applications and infrastructure across teams and clouds.
Data Engineer – AI
Location
California
Posted
143 days ago
Salary
Not specified
10 yrs expEnglishAirflowCloudElastic SearchKubernetesSpark
Job Description
• Define and drive the technical vision for data platforms that support AI-powered features in Crossplane and Upbound Spaces
• Lead the design of data pipelines that transform infrastructure and data into training datasets for ML models
• Architect vector search and RAG systems that leverage Crossplane Control Planes & Upbound Marketplace as a knowledge store
• Build data infrastructure that processes resources, extensions, and compositions for semantic search
• Establish frameworks for collecting, processing, and analyzing infrastructure configuration data
• Design data pipelines that handle Crossplane-specific data
• Create infrastructure for indexing and searching Upbound Marketplace content, documentation, and community patterns
• Develop metrics and monitoring for AI features integrated with Upbound's control plane architecture
• Design data systems that power AI agents for infrastructure provisioning & operations, helping users generate and optimize Crossplane compositions
• Create feature engineering platforms that extract signals from control plane operations, resource status, and reconciliation patterns
• Implement data infrastructure for training models that predict infrastructure failures, optimize resource allocation, and suggest configuration improvements
• Drive the development of knowledge graph representations of infrastructure dependencies and relationships
Job Requirements
- 10+ years of software/data engineering experience with at least 4 years in technical leadership roles
- Proven track record building data platforms that support production systems at scale
- Deep expertise in both traditional data engineering (Spark, Airflow, data lakes) and ML-specific infrastructure (feature stores, model serving)
- Experience with vector databases (Pinecone, Weaviate, Qdrant, Milvus, pgvector, Opensearch, ElasticSearch)
- Demonstrated experience with LLM applications, including RAG architectures and semantic search implementations
- Understanding of Kubernetes, cloud-native architectures, and infrastructure-as-code principles
- Strong understanding of data requirements for AI/ML systems: training pipelines, feature stores, and inference infrastructure
- Hands-on experience building knowledge bases and semantic search systems for technical documentation and code
- Experience with embedding models for code and technical documentation
- Knowledge of time-series data processing for infrastructure metrics and events
- Understanding of graph databases and their application to infrastructure dependency modeling
Benefits
- Health insurance
- 401(k) matching
- Flexible work hours
- Paid time off
- Remote work options
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Data Engineer143 days ago
Full TimeRemoteTeam 51-200Since 1987H1B No Sponsor
Principal Technical Architect at Argano providing strategic direction for technical architecture.
Data Engineer Manager
SuperlanetAdvisory, Staffing, and Multi-State Employer of Record Solutions for Clinicians, by Clinicians.
Data Engineer144 days ago
Full TimeRemoteTeam 51-200Since 2017H1B No Sponsor
Data Engineer Manager leading healthcare data platform development remotely
ApacheAzureCloudETLKafkaPySparkScalaSparkSQLUnity
Data Engineer144 days ago
Full TimeRemoteTeam 501-1,000Since 2011H1B No Sponsor
Data Engineer building scalable data systems at Zapier.
AWSAzureCloudGoogle Cloud PlatformPythonSparkSQLTypeScript
Senior Data Engineer
InStridePartnering with businesses to create life-changing workforce education programs through a leading academic network.
Data Engineer147 days ago
Full TimeRemoteTeam 51-200H1B Sponsor
Sr. Data Engineer building scalable data infrastructure at InStride
CloudMongoDBPythonTypeScript
Arizona + 22 moreAll locations: Arizona, California, Colorado, Connecticut, Florida, Illinois, Kansas, Louisiana, Nevada, New Hampshire, New Jersey, New York, Ohio, Oregon, Maryland, Massachusetts, Michigan, Missouri, Pennsylvania, Texas, Virginia, Washington, Wisconsin
$150K - $165K / year