Senior DevOps & SRE

Current Tech Stack & Infrastructure  Cloud Infrastructure: AWS (ElastiCache, EKS, RDS Aurora MySQL)  Orchestration: Kubernetes on EKS with Karpenter for node management  Infrastructure as Code: Terraform (multiple repositories, experiencing drift and collaboration challenges)  Monitoring & Alerting: Datadog for monitoring, alerting, incident management, and runbooks  CI/CD: GitHub Actions with Atlantis for infrastructure PRs, Bitbucket Pipelines for Helm deployments, Rundeck for scripted operations  GitOps: ArgoCD for Kubernetes workloads  Data Infrastructure: Transitioning to segmented data stores with ClickHouse, MySQL Aurora, and Pulsar for event streaming  APM: Limited Datadog APM usage due to cost ($45 per host/month, ~10 hosts)  Additional Monitoring: Started Prometheus clusters within EKS for more verbose metrics alongside Datadog Scale & Workload  Handling hundreds of thousands of events per second via API (mix of synchronous and asynchronous processing)  EC2 instances: M5/M6 xlarge instances, scaling between 15-100 instances per environment depending on load  Currently transitioning workloads from EC2 to Kubernetes (K8s workload still smaller than core application)  Real-time workloads requiring minimal downtime (minutes not hours for maintenance windows) Key Operational Challenges  Infrastructure as Code: Terraform has become unmanageable due to drift reconciliation and multi-person collaboration issues  Legacy Environments: Some environments set up entirely manually, never in Terraform, with major operational challenges to migrate while keeping them online  Terraform Migration: Proven migration patterns in test environments, but moving to production challenging due to real-time workload requirements  Alert Management: Receiving too many alerts, need prioritization and structured approach to reduce noise and recategorize/adjust thresholds  Alert Distribution: Historically all alerts went to one tech ops team instead of being distributed to five different dev teams; working to shift left closer to developers  Service Catalog: Still defining service catalog and ownership model  Runbooks: Exist in Datadog but need more polish and structure  Database Scaling: Operational challenges with database scaling being addressed through data store segmentation  Monitoring Costs: Datadog is expensive; moved from CloudWatch ~8 years ago due to cost  Manual Changes: Infrastructure changes often implemented manually first, then imported to Terraform and rolled out to other environments Top 5 Required Skill Sets 1. Strong Terraform and CI/CD process expertise 2. Experience breaking down alerts to identify critical vs. non-critical and handling frequent alarms (Datadog-specific experience preferred) 3. SLA/SLI definition experience (team currently being asked to do this without prior experience) 4. AWS infrastructure expertise, DevOps-heavy background 5. Monitoring and alerting experience (Datadog preferred, though Prometheus/Grafana experience also valuable)