Data Normalization 101: MinMaxScalar
In this post, we take a look at how to better work with big data sets by using data normalization techniques in Python using the sklearn library.
Join the DZone community and get the full member experience.Join For Free
As a part of any data cleanup process you’re probably going to want to normalize your data.
Like in golf, if you’re playing against Tiger Woods, you’ll want to give him a handicap, just so that he won’t totally blow you away and that the game might still be fun.
Also, with data, you may want all things equal.
Assume you’re trying to compare the height of a building and it’s internal temperature. The two different units are fine, except when you're trying to teach a machine about the data you may want the units to have some sort of equal measure (300 meters vs. 26 degrees. To us that makes sense, but the actual value of the digits is clearly incongruous).
There are many different ways of doing this. Personally I like the MinMaxScalar function in the sklearn package. First off, it gives you pretty straight-forward results (everything ends up being between 0 and 1), and, secondly, it's the only one I know how to use so far.
So take it or leave it, my pretties...
Let’s make a numpy array. Here are 6 tuples of fictional height/temperature data:
buildingData = np.array([[300,24],[200,21],[126,18],[567,27],[420,19],[189,30]]) print (buildingData)
[[300 24] [200 21] [126 18] [567 27] [420 19] [189 30]]
As I said, we'll be using sklearn to do this stuff, so first you’ll need to import the MinMaxScalar function:
from sklearn.preprocessing import MinMaxScaler
Then we need to figure out the largest and smallest data point in your data set:
scaler_model = MinMaxScaler() scaler_model.fit(buildingData)
Then scale the data appropriately:
scaled_data = scaler_model.transform(buildingData)
This will take the highest and lowest values in your data, turn them into 1 and 0 respectively, then stuff all the other values into relative numbers between 1 and zero. So we get:
print (scaled_data) = [[ 0.39455782 0.5 ] [ 0.16780045 0.25 ] [ 0. 0. ] [ 1. 0.75 ] [ 0.66666667 0.08333333] [ 0.14285714 1. ]]
Voila, all of your data is (proportionately speaking) the same as it was before, but reducing it into relative values between 0 and 1 will allow a machine to process it appropriately.
Published at DZone with permission of Matt Hughes. See the original article here.
Opinions expressed by DZone contributors are their own.