Databricks

We have an open headcount on my team at Databricks (Compute Lifecycle) for a senior or staff engineer! Typically this means 5+ years of experience. For this role, we're also looking for experience with distributed systems (and ideally experience running Kubernetes at scale). We're based out of San Francisco and Mountain View. Databricks has returned to office, so applicants must be able to commute to one of these two offices at least two days a week. Databricks started out building a platform based on Apache Spark. Over the last few years, we've built an all encompassing platform for enterprises to make use of all structured and unstructured data. This includes tools to allow all users (technical or not) to quickly query and visualise data. Lately, we've integrated machine learning and generative AI technologies into the product - acquiring MosaicML last year, a leader in LLM infrastructure. Our team maintains several systems that automate cluster provisioning at scale and allow us to manage hundreds of Kubernetes clusters that power Databricks' serverless products across all 3 major clouds. This includes heavy collaboration with MosaicML to support our new GenAI products. Tech stack wise, we primarily use Go (for Kubernetes facing components) and Scala for Databricks' internal services. We're one of the biggest users of Cluster API, EKS and AKS. This team is one of the strongest I've been fortunate to work with and the complexity of the environments we work in and the products we support means that there's a lot to learn! If you're interested and could be a fit, please send me a resume at sunil.shah@databricks.com. (Unfortunately I am not involved with university recruiting and cannot refer candidates for these roles - please keep an eye on https://www.databricks.com/company/careers/university-recruiting for those.)