Large Scale Stream Processing with Blink SQL at Alibaba
The core business of Alibaba, such as search, recommendation, and advertisement, is driven by the data processing engine: Blink, a project we started about 2 years ago which improved Apache Flink to make it ready for large scale production use. In Alibaba, many data analysts and product managers do not have complex software programing background, but they can program SQL and use it on a regular basis to analyze data and power the business decisions. To enable our users to write SQL to process real-time data efficiently and reliably at Alibaba’s scale, we have developed a stream processing service platform built on top of Blink. In this talk, we will introduce our stream processing service platform, as well as share the experience of running large scale Flink SQL and TableAPI jobs at Alibaba.