For this week, we will learn together how to build a simple lambda architecture data pipeline also with my prefer language Go and the GCP stacks.

What’s the Lambda Architecture?

Lambda architecture is a data-processing architecture designed to handle massive quantities of data by taking advantage of both batch and stream-processing methods.

Three keywords, massive quantities of data (Stream data / Message Queue), Batch and Stream-processing.

Wikipedia — Lambda Architecture Diagram

What’s our use case and Architecture?

Everything builds on top of a use case, and a final deliverable end states, this is without question both for your learning project or your company project, without the clarification on this, most of the use case/projects will fail sooner or later. Here is the high-level architecture.

Here is the use case breakdown:

1. Build A WebSocket API with Topic request.
2. Consume the Twitter Realtime API based on the project request. (massive quantities of data)
3. Apply the Sentiment Analysis Models.
4. Return real-time Sentiment Analysis result via WebSocket. (Realtime processing)
5. Save the result to GCP and batch to the BigQuery for future analysis. (Batch processing)

#big-data #lambda-architecture #websocket #data-science