When we travel to a foreign country, the different time systems can be a headache issue. Sometimes, after several hours of flight, we might arrive in a foreign country with local time even earlier than when we started the flight. It is not that we can magically travel back in time, this time difference is simply due to the different time zones across the world. We can deal with the time change by adjusting our watch to the new time zone. Similarly, in operating systems, there are also different ways to record time across the world. Some systems might be using unix time format, while others might be using UTC time format. Furthermore, even with the same UTC time format, companies with multinational business might also face the issue of different time zones, which complicates the time data.In this post, I hope to have an overview of different time formats. In addition, with PySpark as an example, I will show cases of how to transform among the different time formats.UTC and Unix TimeUTC (Coordinated Universal Time) is the primary time standard used across the world. With the British civil time Greenwich Mean Time (GMT) as UTC+0, areas across the world are divided into different time zones with reference to the Royal Observatory in Greenwich, London.Unix time is the seconds that have elapsed since 00:00:00 UTC on 1 January 1970. It is widely used in the operating systems.

#pyspark #python #spark #data-science #big-data