Data Normalization 101: MinMaxScalar

As a part of any data cleanup process you’re probably going to want to normalize your data.

Like in golf, if you’re playing against Tiger Woods, you’ll want to give him a handicap, just so that he won’t totally blow you away and that the game might still be fun.

Also, with data, you may want all things equal.  

Assume you’re trying to compare the height of a building and it’s internal temperature.  The two different units are fine, except when you're trying to teach a machine about the data you may want the units to have some sort of equal measure (300 meters vs. 26 degrees. To us that makes sense, but the actual value of the digits is clearly incongruous).

There are many different ways of doing this. Personally I like the MinMaxScalar function in the sklearn package. First off, it gives you pretty straight-forward results (everything ends up being between 0 and 1), and, secondly, it's the only one I know how to use so far.

So take it or leave it, my pretties...

Let’s make a numpy array. Here are 6 tuples of fictional height/temperature data:

buildingData = np.array([[300,24],[200,21],[126,18],[567,27],[420,19],[189,30]])
print (buildingData)

=

[[300  24]
 [200  21]
 [126  18]
 [567  27]
 [420  19]
 [189  30]]

As I said, we'll be using sklearn to do this stuff, so first you’ll need to import the MinMaxScalar function:

from sklearn.preprocessing import MinMaxScaler

Then we need to figure out the largest and smallest data point in your data set:

scaler_model = MinMaxScaler()
scaler_model.fit(buildingData)

Then scale the data appropriately:

scaled_data = scaler_model.transform(buildingData)

This will take the highest and lowest values in your data, turn them into 1 and 0 respectively, then stuff all the other values into relative numbers between 1 and zero.  So we get:

print (scaled_data)

=

[[ 0.39455782  0.5 ]
 [ 0.16780045  0.25 ]
 [ 0.          0. ]
 [ 1.          0.75 ]
 [ 0.66666667  0.08333333]
 [ 0.14285714  1. ]]

Voila, all of your data is (proportionately speaking) the same as it was before, but  reducing it into relative values between 0 and 1 will allow a machine to process it appropriately.