This article is a tutorial on how to use the Cloud-Native application in Real-Time Analytics for Continuous SQL on Stock Data.

I would like to track stocks from some companies frequently during the day using Apache NiFi to read the REST API. After that, I have some Streaming Analytics to perform with Apache Flink SQL, and I also want permanent fast storage in Apache Kudu queried with Apache Impala.

I will show you below how to build that in the cloud-native application in seconds.

Source Code: https://github.com/tspannhw/SmartStocks

To Script Loading Schemas, Tables, Alerts see scripts/setup.sh:

Source Code: https://github.com/tspannhw/ApacheConAtHome2020

• Kafka Topic
• Kafka Schema
• Kudu Table
• Flink Prep
• Flink SQL Client Run
• Flink SQL Client Configuration

Once our automated admin has built our cloud environment and populated it with the goodness of our app, we can begin our continuous SQL. If you know your data, build a schema, share it with the registry

One unique thing we added was a default value in our Avro schema and made it a logicalType for timestamp-millis. This is helpful for Flink SQL timestamp related queries.

#sql #apache kafka #apache nifi #apache flink