Kafka was not built for large messages. Period. Nevertheless, more and more projects send and process 1Mb, 10Mb, and even much bigger files and other large payloads via Kafka. One reason is that Kafka was designed for large volume/throughput - which is required for large messages. This article covers the use cases, architectures, and trade-offs for handling large messages with Kafka.
Use Cases for Large (Kafka) Message Payloads
Various use cases for large message payloads exist: Image recognition, video analytics, audio analytics, and file processing are widespread examples.
Image Recognition and Video Analytics
Image recognition and video analytics (also known as computer vision) is probably the number one use case. Many examples require the analysis of videos in real-time, including:
- Security and surveillance (access control, intrusion detection, motion detection)
- Transport monitoring system (vehicle traffic detection, incidence detection, pedestrian monitoring)
- Healthcare (health status monitoring, telemedicine, surgical video analysis)
- Manufacturing (machine vision for quality assurance, augmented support and training)
The usage of image and video processing via concepts such as Computer Vision (e.g., OpenCV) or Deep Learning / Neural Networks (e.g., TensorFlow) reduces time, cost, and human effort, plus this makes industries more secure, reliable, and consistent.
Audio Analytics
Audio analytics is an interesting use case, coming up more and more:
- In conjunction with video analytics: See the use cases above. Often video and audio need to be processed together.
- Consumer IoT (CIoT): Alerting, informing, advising people, e.g., using Audio Analytic.
- Industrial IoT (IIoT): Machine diagnostics and predictive maintenance using advanced sound analysis, e.g., using Neuron Soundware
- Natural Language Processing (NLP): Chatbots and other modern systems use text and speech translation, e.g., using the fully-managed services from the major cloud providers
#open source #big data #machine learning #internet of things #kafka #stream processing #event streaming #limitations #video processing #large file
