Inside the Engine Room: Surviving the challenges of self-hosting Temporal
Abstract
At Datadog, we have been self-hosting Temporal for over four years, scaling from a handful of clusters to managing dozens. This growth has been accompanied by a series of complex and often unexpected challenges. From company-wide compute outages and cluster misconfigurations to the inherent difficulties of scaling a distributed system, our journey has been marked by hard lessons and high-stakes incidents. In this talk, we will spotlight the most significant issues we have encountered, offering a candid look at what self-hosting temporal looks like.
About the Presenters
Hardy is a software craftsman and Open Source enthusiast based in Sweden. He spent many years in the Java world, being, amongst other things, the project lead of Hibernate Search and Hibernate Validator. When Kubernetes became popular, he became interested in container development and joined the Red Hat Developers team to lead the Minishift development, providing developers a sandbox to work with OpenShift. After leaving Red Hat, he worked on Jenkins X and finally arrived at Datadog, where he shifted from containers and CI/CD to durable workflows powered by Temporal.
Marcos Cela is a Senior Software Engineer at Datadog with expertise in automation, infrastructure management, and observability. He has played a critical role in designing and scaling Datadog’s internal workflow platform, achieving significant growth in clusters and workflow volume. With experience spanning CI/CD pipelines, Kubernetes, and Temporal.io, Marcos is passionate about leveraging technology to optimize systems.
