Hands-on Workshop: Building Streaming Data Pipelines with Apache Kafka

Track: Cloud Technology

Abstract

Data is everywhere. It's being produced by almost everything around you. In order for businesses to be competitive, they need to build resilient, scalable systems that can efficiently tap into and react to this ever-increasing volume of data. Enter: streaming data pipelines. Apache Kafka is a distributed event streaming platform that allows you to move and transform data in real-time. Throughout this workshop, you’ll build up a solid foundation and understanding of Kafka, learning the basic building blocks of the technology, its components, as well as how to produce and consume data. From there, you’ll get hands-on experience ingesting data from external systems into Kafka using Kafka Connect, joining and transforming Kafka data in real-time with ksqlDB stream processing, and using Kafka Connect to move data to downstream systems. By the end of the workshop, you'll understand the value of real-time data and have everything you need to start building your own streaming data pipeline. Note: This workshop will utilize a free Confluent Cloud account for hosting Kafka clusters, stream processing, and schema registry. Participants require some programming knowledge.

Danica Fine

Danica began her career as a software engineer in data visualization and warehousing with a business intelligence team where she served as a point-person for standards and best practices in data visualization across her company. In 2018, Danica moved to San Francisco and pivoted to backend engineering with a derivatives data team which was responsible for building and maintaining the infrastructure that processes millions of financial market data per second in near real-time. Her first project on this team involved Kafka Streams – she never looked back. Danica now works as a Developer Advocate with Confluent where she helps others get the most out of their event-driven pipelines. Outside of work, Danica is passionate about sustainability, increasing diversity in the technical community, and keeping her many houseplants alive. She can be found on Twitter, tweeting about tech, plants, and baking @TheDanicaFine.

Amanda Gilbert

Amanda Gilbert is a Senior Solutions Engineer for Confluent and a Confluent Cloud subject matter expert. She has been working in the data engineering space since graduating college in 2014. She is passionate about building event driven architectures that utilize existing technologies, limit technical debt and create performant applications. Amanda lives in Charlotte, NC where she enjoys playing poker, hiking, traveling and reading in her free time.