Real Time APIs in the Context of Apache Kafka. Kafka includes stream processing capabilities through the Kafka Streams API. ksqlDB is an event streaming database purpose-built for stream processing applications. It provides a SQL-based API for querying and processing data in Kafka.
One of the challenges that we have always faced in building applications, and systems as a whole, is how to exchange information between them efficiently whilst retaining the flexibility to modify the interfaces without undue impact elsewhere. The more specific and streamlined an interface, the likelihood that it is so bespoke that to change it would require a complete rewrite. The inverse also holds; generic integration patterns may be adaptable and widely supported, but at the cost of performance.
Events offer a Goldilocks-style approach in which real-time APIs can be used as the foundation for applications which is flexible yet performant; loosely-coupled yet efficient.
Events can be considered as the building blocks of most other data structures. Generally speaking, they record the fact that something has happened and the point in time at which it occurred. An event can capture this information at various levels of detail: from a simple notification to a rich event describing the full state of what has happened.
From events, we can aggregate up to create state—the kind of state that we know and love from its place in RDBMS and NoSQL stores. As well as being the basis for state, events can also be used to asynchronously trigger actions elsewhere when something happens - the whole basis for event-driven architectures. In this way, we can build consumers to match our requirements—both stateless, and stateful with fault tolerance. Producers can opt to maintain state, but are not required to since consumers can rebuild this themselves from the events that are received.
If you think about the business domain in which you work, you can probably think of many examples of events. They can be human-generated interactions, and they can be machine-generated. They may contain a rich payload, or they may be in essence a notification alone. For example:
zbeeblebroxlogged in at
2020-08-17 16:26:39 BST
CarParkedPayload: Car registration
A42 XYZparked at
2020-08-17 16:36:27in space
four tins of baked beanscosting a total of
2020-08-17 16:35:41 BST
These events can be used to directly trigger actions elsewhere (such as a service that processes orders in response to them being placed), and they can also be used in aggregate to provide information (such as the current number of occupied spaces in a car park and thus which car parks have availability).
So, if events are the bedrock on which we are going to build our applications and services, we need a technology that supports us in the best way to do this—and this is where Apache Kafka® comes in. Kafka is a scalable event streaming platform that provides
Kafka is built around the concept of the log. By taking this simple but powerful concept of a distributed, immutable, append-only log, we can capture and store the events that occur in our businesses and systems in a scalable and efficient way. These events can be made available to multiple users on a subscription basis, as well as processed and aggregated further, either for direct use or for storage in other systems such as RDBMS, data lakes, and NoSQL stores.
In the remainder of this article, I will explore the APIs available in Apache Kafka and demonstrate how it can be used in the systems and applications that you are building.
APIs can be as simple as 1 endpoint for use by 100s of users or as complex as the AWS APIs with 1000s of endpoints and 100s of thousands of users. Building them can mean spending a couple of hours using a low-code platform or months of work using a multitude of tools. Hosting them can be as simple as using one platform that does everything we need or as complex as setting up and managing ingress control, security, caching, failover, metrics, scaling.
Measuring website activity provides only half the story. See how to best track the developer's journey and what funnel stages makes sense for API-first products
Selling to developers is hard. How to market to developers efficiently using paid advertising leveraging inbound marketing techniques.
APIs are perceived as reliable—more than half of respondents stated that APIs do not break, stop working, or materially change specification often enough to matter.
The method used for this initial research was to obtain a list of the ASX100 (as of 18 September 2020). Then work through each company looking at the following