Unleash the full potential of Spark and Graph Databases working hand in hand

With the recent release of the official Neo4j Connector for Apache Spark leveraging the Spark DataSource API, there has been a fundamental change in the way that Neo4j data can be queried from within an Apache Spark environment. Alongside this change, the previous Neo4j Spark Connector was marked as deprecated. In this article, I’d like to share an updated end-to-end workflow of setting up a fully interconnected pairing of Neo4j and Spark that makes use of the new connector’s capabilities.In the process, we will first set up a Neo4j cloud instance using an Azure virtual machine. Afterwards, we will set up an Azure Databricks instance running Spark before finally establishing a connection between both resources using the new Neo4j Connector for Apache Spark. If you already have an up-and-running instance of Neo4j or Databricks, you might of course want to skip the respective steps. However, please note the compatibility information at the top of each step.

#spark #neo4j #database #developer #pyspark