For any API-first company, implementing the right API analytics platform is critical for tracking the utilization of your APIs and to discover any performance or functional issues that may be impacting customers. There are a variety of open-source projects you can leverage to build a complete API analytics platform.
Before jumping into building an API analytics solution yourself, you should first list out your requirements and use cases. Not all tools support all use cases directly, and may require heavy investment in development and integration.
API analytics build requirements
Need to answer engineering or business questions
Engineers commonly turn to API logs and metrics to understand what’s happening over their APIs, inspect payloads, and root cause issues with their services that come up. Real-time API logging capabilities is a must for engineers that want to depend on their API analytics solution to put out the (hopefully far and few between) fires due to an API outage or reliability issue. Because real-time logging can raise the compute and storage cost, not all analytics infrastructure maintains a real-time pipeline. On the flip side, only the most recent data is needed to answer hair on fire engineering problems, so data can be retired after a short time period such as after 24 hours.
Engineering and product leaders make strategic decisions based on lines, not dots. Business questions are answered from historical trends in data, which may be over months or even years. This means your API analytics build should be capable of storing data for long retention periods such as for multiple years. This also requires infrastructure that can roll up and compress your data as storing and running aggregations on raw event logs will cripple your analytics infrastructure
Real-time alerting and monitoring?
While building dashboards and throwing them up on a monitor can be a great way to monitor your metrics, it still requires manually checking them periodically. To become more proactive, many API analytics builds also have some sort of monitoring and alerting features. These can vary from simple threshold-based alerts that sends an email when a metric reaches a certain value, to complex monitoring rules and workflows that can perform complex aggregations in real-time and route alerts to incident response platforms like PagerDuty and BigPanda.
Some customer-facing teams like developer relations and customer success may want to build automated workflows to notify customers or internal specific internal stakeholders based on complex user behavioral flows.
Access data anywhere vs controlled access
The post-COVID world has accelerated the shift towards work from home. This means internal users of your API analytics build may need to access metrics on home networks or on the go without a VPN. Placing unnecessary security restrictions to access the API data may limit the value your API analytics provides to your company or can even backfire creating bad habits such as password sharing or exporting large amounts of data to a personal device rather than using the API analytics build the way it was designed.
On the flip side, providing self-service API analytics from anywhere also means having strong security and access control. If you think typical enterprise access control like single sign on and role-based access control may be needed in the future, you should plan your API analytics build accordingly even if not needed immediately. Ripping out and changing authentication and authorization design is not an easy task and could force a complete rewrite. You don’t want to be caught off guard and be the engineer who failed at foreshadowing typical future enhancements.
Flexibility of visualizations
While most analytics platforms can display event data or plot basic single-value metrics over time, you platform may also be used for more advanced and specialized analysis like funnel analysis or cohort retention analysis. These are common queries used by business functions like marketing and growth teams but rarely used by developers themselves. Yet, building a funnel can be quote challenging if the right data model was not used.
Many times, you don’t know the types of queries that need to be displayed but should choose a project that enables flexibility in both the data model and the visualization layer.
#api #api-analytics #grafana #kibana
