It’s an old trope, but the companies of the world are sitting on a goldmine of insights locked away in data that no-one has ever looked at. This hidden data is nowhere more common than in server or machine event logs and over the past decade there has been an explosion in full-text search and log aggregation tools. Software like Elasticsearch and Splunk have internet-scale superpowers at ingesting and dashboarding real-time events.

For the humble analyst these new methods allow the indexing of literally everything. This means that counting the number of weird log-in attempts or the average temperature on one of 10,000 industrial sensors can be as easy as a Google search.

#logs #process #data-science #regex #power-bi