Processing Column DataΒΆ

As part of this module we will explore the functions available under org.apache.spark.sql.functions to derive new values from existing column values with in a Data Frame.

  • Pre-defined Functions

  • Create Dummy Data Frame

  • Categories of Functions

  • Special Functions - col and lit

  • String Manipulation Functions - 1

  • String Manipulation Functions - 2

  • Date and Time Overview

  • Date and Time Arithmetic

  • Date and Time - trunc and date_trunc

  • Date and Time - Extracting Information

  • Dealing with Unix Timestamp

  • Example - Word Count

  • Conclusion

Conclusion Pre-defined Functions