How to optimize loading 150GB data into hive table?

How to optimize loading 150GB data into hive table?

I have a 150 GB file in hive stage table which uses following table properties

I have a 150 GB file in hive stage table which uses following table properties

    ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
    WITH SERDEPROPERTIES (
       "separatorChar" = "|",
       "quoteChar"     = "'",
       "escapeChar"    = "\\"
    )  
    STORED AS TEXTFILE;

Now when I load this data in my main table, it fails with java heap error after running for one hour. I am using partitioned main table and there are about 12000 partitions in the data. For loading the data I am using simple hql:

    insert overwrite mainTable partition(date)
    select * from stage table;

I have also tried increasing the map memory to 15GB but sill it fails. Is there any way to optimize this ? Any solution which includes spark or hive would work.

Angular 9 Tutorial: Learn to Build a CRUD Angular App Quickly

What's new in Bootstrap 5 and when Bootstrap 5 release date?

Brave, Chrome, Firefox, Opera or Edge: Which is Better and Faster?

How to Build Progressive Web Apps (PWA) using Angular 9

What is new features in Javascript ES2020 ECMAScript 2020

Hadoop vs Spark | Hadoop MapReduce vs Spark

🔥Intellipaat Big Data Hadoop Course: https://intellipaat.com/big-data-hadoop-training/ In this video on Hadoop vs Spark you will understand about the top Big...

Apache Spark Tutorial For Beginners - Apache Spark Full Course

This video on Apache Spark Tutorial For Beginners - Apache Spark Full Course will help you learn the basics of Big Data, what Apache Spark is, and the architecture of Apache Spark. Yyou will understand how to install Apache Spark on Windows and Ubuntu. You will look at the important components of Spark, such as Spark Streaming, Spark MLlib, and Spark SQL. You will get an idea about implement Spark with Python in PySpark tutorial and look at some of the important Apache Spark interview questions

Why learn Apache Spark in 2020?

This video on "Apache Spark in 2020" will provide you with the detailed and comprehensive knowledge about the current IT Job trends based on Apache Spark and why learn Apache Spark in 2020? What is new in Apache Spark? What is Apache Spark? Top 5 Reasons to learn Spark. Salary trends of Spark Developer. Components of Spark. Skills required by Spark Developer. Companies using Apache Spark