Be more effective with your data

In building Raygun we have to deal with a constant stream of exception data (100’s of millions per month) which we structure into errors and occurrences which have a simple 1:N relationship.

A relational database engine is great at set based operations and has a very familiar and powerful querying syntax but we are increasingly facing data challenges where leveraging a relational model doesn’t make as much sense. In building Raygun we have to deal with a constant stream of exception data (100’s of millions per month) which we structure into errors and occurrences which have a simple 1:N relationship.

At first, using a relational model seemed to be a natural fit for this but over time we ran into three general problems:

1. Querying over the occurrence data was performing poorly despite being strictly index based. E.g. To find all parents where they have a child occurrence matching a particular criteria.
2. Producing aggregations of occurrence data was too costly to query in real time. Using a pre-aggregated bucket approach would add a massive overhead to our database size.
3. Searching the data using FTS would require indexing most of the raw payload which is costly both in terms of indexing time and database size and our preference was to use Lucene.

Surely we must just need a more powerful database cluster? More RAM, more CPU? Sure, but what about when we add 10x our data volume?

Rather than just blindly scaling, I always like to look around and see what everyone else is doing incase there is a better way. It turns out these are the type of problems that are handled nicely by ElasticSearch.

big data

An extensively researched list of top microsoft big data analytics and solution with ratings & reviews to help find the best Microsoft big data solutions development companies around the world.

‘Data is the new science. Big Data holds the key answers’ - Pat Gelsinger The biggest advantage that the enhancement of modern technology has brought

We need no rocket science in understanding that every business, irrespective of their size in the modern-day business world, needs data insights for its expansion. Big data analytics is essential when it comes to understanding the needs and wants of a significant section of the audience.

In this article, see the role of big data in healthcare and look at the new healthcare dynamics. Big Data is creating a revolution in healthcare, providing better outcomes while eliminating fraud and abuse, which contributes to a large percentage of healthcare costs.

Big Data Analytics is the next big thing in business, and it is a reality that is slowly dawning amongst companies. With this article, we have tried to show you the importance of Big Data in business and urge you to take advantage of this immense...