Kibana and the rest of the ELK stack (Elasticsearch, Kibana, Logstash) is great for parsing and visualizing API logs for a variety of use cases. As an open-source project, it’s free to get started (you need to still factor in any compute and storage cost which is not cheap for analytics). One use case for Kibana that’s grown recently is providing analysis and forensics for API security, a growing concern for engineering leaders and CISO’s as companies expose more and more APIs to their customers, partners, and leveraged by Single Page Apps and mobile apps. This can be done by instrumenting applications to log all API traffic to Elasticsearch. However, a naive implementation would only store raw API logs and calls, which is not sufficient for API security use cases.

Why API logging is a naive approach to API security

Raw API logs only contain the information pertaining to execute a single action. Usually the HTTP headers, IP address, request body, and other information is logged for later analysis. Monitoring can be added by purchasing a license for Elasticsearch X-Pack. The issue is that security incidents cannot always be detected by looking at API calls in isolation. Instead, hackers are able to perform elaborate behavioral flows that exercise your API in an unintended way.

Let’s take a simple pagination attack as an example. A pagination attack is when a hacker is able to paginate through a resource like /items or /users to scrape your data without detection. Maybe the info is already public and low risk such as items listed in an e-commerce platform. However, the resource could also have PII or other sensitive information such as /users, but was not correctly protected. In this case, a hacker could write a simple script to dump all the users stored in your database like so:

skip = 0
while True:
    response = requests.post('https://api.acmeinc.com/users?take=10&skip=' + skip),headers={'Authorization': 'Bearer' + ' ' + sys.argv[1]})
    print("Fetched 10 users")
    sleep(randint(100,1000))
    skip += 10

Couple of things to note:

1. The hacker is waiting a random time between each call to not run into rate limits
2. Since the frontend app only fetches 10 users at a time, the hacker only fetches 10 at a time to not raise any suspicion

There is absolutely nothing in a single API call that can distinguish these bad requests vs real requests. Instead, your API security and monitoring solution needs to examine user behaviors holistically. This means examining all the API calls together made by a single user or API key which is called User Behavior Analytics or UBA.

How to implement User Behavior Analytics in Kibana and Elasticsearch

To implement User Behavior Analytics in Kibana and Elasticsearch, we need to flip our time-centric data model around to one that is user-centric Normally, API logs are stored as a time-series using the event time or request time as the date to organize data around. By doing so, older logs can easily be marked read only, moved to smaller infrastructure, or retired based on retention policies. In addition, it makes search fast when you’re only querying a limited time range.

#api #api security #api providers #security analytics #api security risks #api access #uba #user behavior analytics