Advances in Stream Processing
The digital transformation is driven by gathering and analyzing ever new sources of data. In many cases, the value of analysis results decreases quickly. The faster decisions can be taken based on data observed, the better. To process data incrementally and in real time, an increasing amount of applications is leveraging stream processing systems. At the same time, new data sources become available and affordable, which means scalable solutions are required.
Current stream processing systems can scale out individual analysis tasks to tens to hundreds of nodes, processing millions of events per second. Today these systems can support the world's largest companies at peak hours, but they are severely limited in the number and dynamicity of analysis tasks making them inflexible and inefficient in setups with many users. In this presentation, we will show novel methods and system architectures that enable highly efficient stream data processing in these dynamic setups.