Combine 2 data frames with different columns in spark

I have 2 dataframes:

I have 2 dataframes:

df1 :

Id    purchase_count   purchase_sim
12       100               1500
13       1020              1300
14       1010              1100
20       1090              1400
21       1300              1600

df2:

Id     click_count      click_sim
12       1030              2500
13       1020              1300
24       1010              1100
30       1090              1400
31       1300              1600

I need to get the combined data frame with results as :

Id     click_count      click_sim     purchase_count   purchase_sim
12       1030              2500            100               1500
13       1020              1300            1020              1300
14       null              null            1010              1100
24       1010              1100            null              null
30       1090              1400            null              null
31       1300              1600            null              null
20       null              null            1090              1400
21       null              null            1300              1600                                     

I can't use union because of different column names. Can some one suggest me a better way to do this ?

Apache Spark Tutorial - Apache Spark Full Course - Learn Apache Spark

Apache Spark Tutorial - Apache Spark Full Course - Learn Apache Spark

This video will help you understand and learn Apache Spark in detail. This Spark tutorial is ideal for both beginners as well as professionals who want to master Apache Spark concepts.

This video will help you understand and learn Apache Spark in detail. This Spark tutorial is ideal for both beginners as well as professionals who want to master Apache Spark concepts. Below are the topics covered in this Spark tutorial for beginners:

2:44 Introduction to Apache Spark

3:49 What is Spark?

5:34 Spark Eco-System

7:44 Why RDD?

16:44 RDD Operations

18:59 Yahoo Use-Case

21:09 Apache Spark Architecture

24:24 RDD

26:59 Spark Architecture

31:09 Demo

39:54 Spark RDD

41:09 Spark Applications

41:59 Need For RDDs

43:34 What are RDDs?

44:24 Sources of RDDs

45:04 Features of RDDs

46:39 Creation of RDDs

50:19 Operations Performed On RDDs

50:49 Narrow Transformations

51:04 Wide Transformations

51:29 Actions

51:44 RDDs Using Spark Pokemon Use-Case

1:05:19 Spark DataFrame

1:06:54 What is a DataFrame?

1:08:24 Why Do We Need Dataframes?

1:09:54 Features of DataFrames

1:11:09 Sources Of DataFrames

1:11:34 Creation Of DataFrame

1:24:44 Spark SQL

1:25:14 Why Spark SQL?

1:27:09 Spark SQL Advantages Over Hive

1:31:54 Spark SQL Success Story

1:33:24 Spark SQL Features

1:37:15 Spark SQL Architecture

1:39:40 Spark SQL Libraries

1:42:15 Querying Using Spark SQL

1:45:50 Adding Schema To RDDs

1:55:05 Hive Tables

1:57:50 Use Case: Stock Market Analysis with Spark SQL

2:16:50 Spark Streaming

2:18:10 What is Streaming?

2:25:46 Spark Streaming Overview

2:27:56 Spark Streaming workflow

2:31:21 Streaming Fundamentals

2:33:36 DStream

2:38:56 Input DStreams

2:40:11 Transformations on DStreams

2:43:06 DStreams Window

2:47:11 Caching/Persistence

2:48:11 Accumulators

2:49:06 Broadcast Variables

2:49:56 Checkpoints

2:51:11 Use-Case Twitter Sentiment Analysis

3:00:26 Spark MLlib

3:00:31 MLlib Techniques

3:01:46 Demo

3:11:51 Use Case: Earthquake Detection Using Spark

3:24:01 Visualizing Result

3:25:11 Spark GraphX

3:26:01 Basics of Graph

3:27:56 Types of Graph

3:38:56 GraphX

3:40:42 Property Graph

3:48:37 Creating & Transforming Property Graph

3:56:17 Graph Builder

4:02:22 Vertex RDD

4:07:07 Edge RDD

4:11:37 Graph Operators

4:24:37 GraphX Demo

4:34:24 Graph Algorithms

4:34:40 PageRank

4:38:29 Connected Components

4:40:39 Triangle Counting

4:44:09 Spark GraphX Demo

4;57:54 MapReduce vs Spark

5:13:03 Kafka with Spark Streaming

5:23:38 Messaging System

5:21:15 Kafka Components

2:23:45 Kafka Cluster

5:24:15 Demo

5:48:56 Kafka Spark Streaming Demo

6:17:16 PySpark Tutorial

6:21:26 PySpark Installation

6:47:06 Spark Interview Questions

Hadoop vs Spark | Hadoop MapReduce vs Spark

Hadoop vs Spark | Hadoop MapReduce vs Spark

🔥Intellipaat Big Data Hadoop Course: https://intellipaat.com/big-data-hadoop-training/ In this video on Hadoop vs Spark you will understand about the top Big...

In this video on Hadoop vs Spark you will understand about the top Big Data solutions used in the IT industry, and which one should you use for better performance. So in this Hadoop MapReduce vs Spark comparison some important parameters have been taken into consideration to tell you the difference between Hadoop and Spark also which one is preferred over the other in certain aspects in detail.

Why Hadoop is important

Big data hadoop is one of the best technological advances that is finding increased applications for big data and in a lot of industry domains. Data is being generated hugely in each and every industry domain and to process and distribute effectively hadoop is being deployed everywhere and in every industry.

Spark MLlib tutorial | Machine Learning On Spark | Apache Spark Tutorial

Spark MLlib tutorial | Machine Learning On Spark | Apache Spark Tutorial

This video on Spark MLlib Tutorial will help you learn about Spark's machine learning library. You will understand the different types of machine learning algorithms - supervised, unsupervised, and reinforcement learning.

Spark MLlib tutorial | Machine Learning On Spark | Apache Spark Tutorial

This video on Spark MLlib Tutorial will help you learn about Spark's machine learning library. You will understand the different types of machine learning algorithms - supervised, unsupervised, and reinforcement learning.

Then, you will get an idea about the various tools that Spark's MLlib component provides. You will see the different data types and some fundamental statistical analysis that you can perform using MLlib.

Finally, you will understand about classification and regression algorithms and implement it using linear and logistic regression. Now, let's get started and learn Spark MLlib.

Below topics are explained in this Spark MLlib tutorial:

  1. What is Spark MLlib? 00:42
  2. What is Machine Learning? 02:27
  3. Machine Learning Algorithms 04:51
  4. Spark MLlib Tools 09:14
  5. Spark MLlib Data Types 09:55
  6. Machine Learning Pipelines 22:18
  7. Clasification & Regression 24:13
  8. Spark MLlib Use Case Demo 31:51