Playing With Pandas DataFrames (With Missing Values Table Example)
Sometimes, you may want to concat two dataframes by column base or row base. For this action, you can use the concat function.
Join the DZone community and get the full member experience.Join For Free
Sometimes, you may want to concat two dataframes by column base or row base. For this action, you can use the concat function. These can be done by the following code.
For a column base, you have to give axis=1 parameter. I want to explain this functionality by preparing a simple missing values table.
You may also like: PySpark DataFrame Tutorial: Introduction to DataFrames
For this, I will again use the data set I have referred to in my article Pandas Dataframe Functions. First, we have to load the data.
After loading the data we will calculate some information about the data frame and concat them in a dataframe at the end.
First, we can calculate the null count of each column by the following code and assign it to a variable.
It will give the following result. This will be our first column:
This calculation will give the percentage of the null values in the total values of the column.
Lastly, data types can be also added to our missing value table:
Three of the data sets give us one-dimensional series data. We will concat them to create our final dataframe:
As you can see it has all the values, since we want only missing values, we have to filter the result set. Moreover, there are no column names, to clarify what type of data the column has, we also have to rename all the columns. These can be done by the following codes.
And now it is time to filter the data frame just to list the missing values statistics for the loaded data frame. For filtering dataframe, iloc and loc can be used.
Selecting Rows and Columns By Loc and ILoc
Data can be selected from data frames by using loc and iloc options:
Loc is used for selecting rows and columns by index and value label, columns can be selected by column names,
Iloc is used for selecting rows and columns by their indexes.
Here are some examples before continuing our missing tables example.
Examples by loc: one of the ways is querying data by loc is using indexes on dataframe
The above query gives the data from df where the first_name column has the "zehra" values. or you can query by conditional expressions.
Gets all the rows for the "zehra" value with the first two columns in df.
Here is our final missing values table for our dataframe.
The above code first filters the data based on the Percentage column and then sort by on this column in descending order. The "missing value table" gives you a simple readable table for your data frame missing values.
Opinions expressed by DZone contributors are their own.