Big Data is not a fad. In fact we’re living at the verge of a revolution that is touching every industry, business and life on this planet. With millions of tweets, iMessages, Live streams, Facebook and Instagram posts….terabytes and petabytes of data is being generated every second and getting “meaningful insight” from this data is quite a challenge since the traditional data bases and data warehouses are not able to handle the processing demands of these Big Data sets that need to be updated frequently or often in real time as in case of stocks, application performance monitoring or user’s online activities. In response to the growing demand for tools and technologies for Big Data Analytics, many organizations turned to NoSQL databases and Hadoop along with some its companions analytics tools including but not limited to YARN, MapReduce, Spark, Hive, Kafka etc.

All these tools and frameworks make up a huge Big Data ecosystem and cannot be covered in a single article. For the sake of this article, my focus is to give you a gentle introduction to Apache Spark and above all, the .Net library for Apache Spark which brings Apache Spark tools into .Net Ecosystem.

We will be covering following topics,
What is Apache Spark?
Apache Spark for .Net
Architecture
Configuring and testing Apache Spark on windows
Writing and Executing your first Apache Spark Program

#.net #.net core #apache spark #big data