José Valim is the creator of the Elixir programming language. He graduated in Engineering in São Paulo University, Brazil and has a Master of Science by Politecnico di Torino, Italy. He runs Dashbit alongside a team of Elixir experts to help companies and startups adopt and run Elixir in production.
Broadway is a new open source project by Plataformatec that aims to streamline data processing pipelines. It allows developers to consume data efficiently and concurrently from many systems, such as Amazon SQS, RabbitMQ, Google Cloud PubSub and others.
In this talk, we will discuss how the Broadway architecture connects multiple stages and producers, how it leverages GenStage to provide back-pressure, how we rely on OTP for fault-tolerance, and other features such as batching, rate-limiting, partitioning and more.