Writing a Scala/Spark UDF: Options to Consider

DZone 's Guide to

Writing a Scala/Spark UDF: Options to Consider

In this article, we discuss various ways that we can write User Defined Functions in Spark with Scala.

· Big Data Zone ·
Free Resource

A couple of weeks ago, at my work place, I wrote a metadata-driven data validation framework for Spark. After the initial euphoria of having created the framework in Scala/Spark and Python/Spark, I started reviewing the framework. During the review, I noted that the User Defined Functions (UDF) I had written were prone to throw an error in certain situations.

I then explored various options to make the UDFs fail-safe. Let us start by considering the data as below

Plain Text

Let us read the data into a dataframe, as below


For this data set, let us assume that we want to check if the name of the superhero is "kal el". Let us also assume that we are going to implement this check using a UDF.

Option A

The most obvious method of doing so is shown below:


When we apply the isAlienNameUDF method, it works for all cases where the column value is not null. If the value of the cell passed to the UDF is null, it throws an exception: org.apache.spark.SparkException: Failed to execute user defined 

This is because we are executing the method equalsIgnoreCase on a null value.

Option B

To overcome the problem of Option A, we can modify the UDF as follows


Option C

Instead of checking for null in the UDF or writing the UDF code to avoid a NullPointerException, Spark provides a method that allows us to perform a null check right at the place where the UDF is executed, as below

val df4 = df.withColumn("df4", isAlienNameUDF2(when(col("alien-name").
isNotNull,col("alien-name")).otherwise(lit("xyz")))) df4.show 

In this case, we check the value of the column. If the value is not null, we pass the value of the column. Otherwise, we pass a default value to the UDF.

Option D

In option C, irrespective of the value of the column, we are invoking the UDF. We can avoid this by changing the order of 'when' and 'otherwise', as follows:

val df5 = df.withColumn("df5", when(col("alien-name").isNotNull, 
isAlienNameUDF2(col("alien-name"))).otherwise(lit("xyz"))) df5.show 

In this option, the UDF is invoked only if the column value is not null. If the column value is null, we use a default value.


At this point in time, I believe that option D should be the preferred option when writing a UDF.

big data ,scala ,spark ,tutorial ,udf

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}