Stream and Batch Processing in the Cloud with Data Microservices

Track: Data, Integration & IoT
Skill Level: Beginner
Room: Room A312
Time Slot: Wed 2/17, 1:00 PM
Tags: batch , mesos , data , stream processing , spring , microservices , spring cloud , iot , yarn , cloud , integration
Abstract

The future of scalable data processing is microservices! Building on the ease of development and deployment provided by Spring Boot and the cloud native capabilities of Spring Cloud, the Spring Cloud Stream and Spring Cloud Task projects provide a simple and powerful framework for creating microservices for stream and batch processing. They make it easy to develop data-processing Spring Boot applications that build upon the capabilities of Spring Integration and Spring Batch, respectively. At a higher level of abstraction, Spring Cloud Data Flow is an integrated orchestration layer that provides a highly productive experience for deploying and managing sophisticated data pipelines consisting of standalone microservices. Streams and tasks are defined using a DSL abstraction and can be managed via shell and a web UI. Furthermore, a pluggable runtime SPI allows Spring Cloud Data Flow to coordinate these applications across a variety of distributed runtime platforms such as Apache YARN, Cloud Foundry, or Apache Mesos. This session will provide an overview of these projects, including how they evolved out of Spring XD. Both streaming and batch-oriented applications will be deployed in live demos on different platforms ranging from local cluster to a remote Cloud to show the simplicity of the developer experience.

Marius Bogoevici

Marius Bogoevici is a software engineer at Pivotal, leading Spring Cloud Stream, and working on Spring Cloud Data Flow, Spring XD, and other Spring projects. Marius is co-author of “Spring Integration in Action” (Manning, 2012)

Mark Fisher

Mark has been a member of the Spring team for over a decade, contributing to the Spring Framework and several other projects. He founded Spring Integration in 2007 and is one of the authors of Spring Integration in Action, published by Manning in 2012. Currently he co-leads Spring Cloud Data Flow and contributes to other Spring Cloud projects.