Combine 2 data frames with different columns in spark

Combine 2 data frames with different columns in spark

I have 2 dataframes:

I have 2 dataframes:

df1 :

Id    purchase_count   purchase_sim
12       100               1500
13       1020              1300
14       1010              1100
20       1090              1400
21       1300              1600

df2:

Id     click_count      click_sim
12       1030              2500
13       1020              1300
24       1010              1100
30       1090              1400
31       1300              1600

I need to get the combined data frame with results as :

Id     click_count      click_sim     purchase_count   purchase_sim
12       1030              2500            100               1500
13       1020              1300            1020              1300
14       null              null            1010              1100
24       1010              1100            null              null
30       1090              1400            null              null
31       1300              1600            null              null
20       null              null            1090              1400
21       null              null            1300              1600                                     

I can't use union because of different column names. Can some one suggest me a better way to do this ?

scala apache-spark

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Python Vs Scala For Apache Spark

Apache Spark is a popular open-source data processing framework. This widely-known big data platform provides several exciting features, such as graph processing, real-time processing, in-memory processing, batch processing and more quickly and easily.

Apache Spark Hands-on With Scala.

Welcome to some practical explanations to Apache Spark with Scala. There is even Python supported Spark is available which is PySpark. For the sake of this post, I am continuing with Scala with my windows Apache Spark installation.

Apache Spark Tutorial For Beginners - Apache Spark Full Course

This video on Apache Spark Tutorial For Beginners - Apache Spark Full Course will help you learn the basics of Big Data, what Apache Spark is, and the architecture of Apache Spark. You will understand how to install Apache Spark on Windows and Ubuntu. You will look at the important components of Spark, such as Spark Streaming, Spark MLlib, and Spark SQL. You will get an idea about implement Spark with Python in PySpark tutorial and look at some of the important Apache Spark interview questions

Apache Spark For Beginners In 3 Hours | Apache Spark Training

In this Apache Spark For Beginners, we will have an overview of Spark in Big Data. An introduction to Apache Spark Programming. The Spark History. We'll learn why Spark is needed and covers everything that an individual needed to master its skill in this field. In this Apache Spark tutorial, you will not only learn Spark from the basics but also through this Apache Spark tutorial, you will get to know the Spark architecture and its components such as Spark Core, Spark Programming, Spark SQL, Spark Streaming, and much more.

Apache Spark Tutorial | Spark Tutorial For Beginners

You will learn what apache spark is, the features of Apache Spark, and the architecture of Apache Spark. You will understand the various components of Apache Spark, such as Spark Core, Spark SQL, Spark Streaming, Spark MLlib, and Spark GraphX. You will look into a case study of Spark for OpenTable company. Finally, you will do a demo on linear regression and logistic regression using PySpark.