José Valim is the creator of the Elixir programming language and member of the Rails Core Team. He graduated in Engineering in São Paulo University, Brazil and has a Master of Science by Politecnico di Torino, Italy. He is also the lead-developer of Plataformatec, a consultancy firm based in Brazil, and an active member of the Open Source community.
Broadway is a new open source project by Plataformatec that aims to streamline data processing pipelines. It allows developers to consume data efficiently and concurrently from many systems, such as Amazon SQS, RabbitMQ, Google Cloud PubSub and others.
In this talk, we will discuss how the Broadway architecture connects multiple stages and producers, how it leverages GenStage to provide back-pressure, how we rely on OTP for fault-tolerance, and other features such as batching, rate-limiting, partitioning and more.