{{announcement.body}}
{{announcement.title}}

Playing With Pandas DataFrames (With Missing Values Table Example)

DZone 's Guide to

Playing With Pandas DataFrames (With Missing Values Table Example)

Sometimes, you may want to concat two dataframes by column base or row base. For this action, you can use the concat function.

· Big Data Zone ·
Free Resource

Sometimes, you may want to concat two dataframes by column base or row base. For this action, you can use the concat function. These can be done by the following code.

Python







For a column base, you have to give axis=1 parameter. I want to explain this functionality by preparing a simple missing values table.

You may also like: PySpark DataFrame Tutorial: Introduction to DataFrames

For this, I will again use the data set I have referred to in my article Pandas Dataframe Functions. First, we have to load the data.

Python







After loading the data we will calculate some information about the data frame and concat them in a dataframe at the end.

First, we can calculate the null count of each column by the following code and assign it to a variable.

Python







It will give the following result. This will be our first column:

missing_value code

Python







This calculation will give the percentage of the null values in the total values of the column.

percentage of null values

Lastly, data types can be also added to our missing value table:

Python







percentage dataframe

Three of the data sets give us one-dimensional series data. We will concat them to create our final dataframe:

Python







missing_values_table

As you can see it has all the values, since we want only missing values, we have to filter the result set. Moreover, there are no column names, to clarify what type of data the column has, we also have to rename all the columns. These can be done by the following codes.

Python







missing_values_table

And now it is time to filter the data frame just to list the missing values statistics for the loaded data frame. For filtering dataframe, iloc and loc can be used.

Selecting Rows and Columns By Loc and ILoc

Data can be selected from data frames by using loc and iloc options:

Loc is used for selecting rows and columns by index and value label, columns can be selected by column names,

Iloc is used for selecting rows and columns by their indexes.

Here are some examples before continuing our missing tables example.

Python







Examples by loc: one of the ways is querying data by loc is using indexes on dataframe

Python







The above query gives the data from df where the first_name column has the "zehra" values. or you can query by conditional expressions.

Python







Gets all the rows for the "zehra" value with the first two columns in df.

Here is our final missing values table for our dataframe.

Python







The above code first filters the data based on the Percentage column and then sort by on this column in descending order. The "missing value table" gives you a simple readable table for your data frame missing values.


Further Reading

A Brief Overview of Pandas DataFrames

Python Pandas Tutorial: Getting Started With DataFrames

PySpark Join Explained

Topics:
big data ,dataframes ,missing values ,pandas ,pandas data frames tutorial ,pandas dataframe ,python

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}