Writing a Scala/Spark UDF: Options to Consider
Join the DZone community and get the full member experience.Join For Free
A couple of weeks ago, at my work place, I wrote a metadata-driven data validation framework for Spark. After the initial euphoria of having created the framework in Scala/Spark and Python/Spark, I started reviewing the framework. During the review, I noted that the User Defined Functions (UDF) I had written were prone to throw an error in certain situations.
I then explored various options to make the UDFs fail-safe. Let us start by considering the data as below
Let us read the data into a dataframe, as below
For this data set, let us assume that we want to check if the name of the superhero is "kal el". Let us also assume that we are going to implement this check using a UDF.
The most obvious method of doing so is shown below:
When we apply the isAlienNameUDF method, it works for all cases where the column value is not null. If the value of the cell passed to the UDF is null, it throws an exception:
org.apache.spark.SparkException: Failed to execute user defined
This is because we are executing the method equalsIgnoreCase on a null value.
To overcome the problem of Option A, we can modify the UDF as follows
Instead of checking for null in the UDF or writing the UDF code to avoid a NullPointerException, Spark provides a method that allows us to perform a null check right at the place where the UDF is executed, as below
val df4 = df.withColumn("df4", isAlienNameUDF2(when(col("alien-name").
In this case, we check the value of the column. If the value is not null, we pass the value of the column. Otherwise, we pass a default value to the UDF.
In option C, irrespective of the value of the column, we are invoking the UDF. We can avoid this by changing the order of 'when' and 'otherwise', as follows:
val df5 = df.withColumn("df5", when(col("alien-name").isNotNull,
In this option, the UDF is invoked only if the column value is not null. If the column value is null, we use a default value.
At this point in time, I believe that option D should be the preferred option when writing a UDF.
Opinions expressed by DZone contributors are their own.