Async Building Blocks: A Streaming Data Drama in Three Acts

This talk, presented at RustConf 2023, delves into the complexities of building asynchronous streaming data pipelines. The presentation uses a narrative approach, introducing characters like Mary (a message), Pablo Pubsub, Beatrix Buffer, Dino Demuxer, Mitch Mutex, Greta Grouper, the Processor, and Padma Publisher, to illustrate key concepts. It covers challenges such as message synchronization, network issues leading to out-of-order or duplicate messages, and the implementation of solutions using Rust's asynchronous features like streams, futures, and Tokio. The talk explores buffering strategies, race condition handling with mutexes, parallel execution through tasks, and data extraction from asynchronous processes using channels. The narrative culminates in the metamorphosis of messages into predictions and their subsequent publication.