Getting Insights From Survey Results Using Data Science in Python

prI'm neither a statistician nor a professional data scientist. But when I started talking with Jared Menard, a friend and fellow Chattanooga-based developer, about the salary survey he put together, I was intrigued by the insights that could be gotten from the data. The survey asked questions like:

• How much money do you make a year?
• How many years of experience do you have?
• How many books have you read in the last year?
• Do you have a degree? Is it related to your career?
• Name some things that you use at work.

Of the roughly 500 developers targeted 67 people responded to the survey, putting us short of the 80 respondents needed for a 10% margin of error per survey monkey. Also, the survey was not designed by a professional researcher / survey designer. So while I readily acknowledge that there may be some potential flaws in the findings, I was more interested in the process of using data science and machine learning techniques to get deeper insights from the data than what could be provided by the basic statistical tools of mean, mode, standard deviation, scatter plots and column charts.

Two of the questions I was hoping to answer were:

1. What kinds/groups of developers answered the survey and what are the general characteristics of those groupings?
2. Of all the features we have, what are the important features in this dataset when it comes to determining salary and given those features can we predict an expected salary?

Data Preparation

To get started, I had to do what every data scientist is forced to spend 80% of their time doing, which is data cleaning and preparation. Due to the small size of the dataset, it was easy enough to pull up the CSV file in a text editor or spreadsheet program and visually examine the data to see what needed fixing. For larger data sets, you would have to rely on tools to find missing values and outliers, generate histograms, etc. For example, the dataframe class from the Pandas library contains some methods like isnull() and fillna() that can be used to manage missing values.

When it comes to missing values, there are a number of strategies for handling them. You could drop the record with the missing value entirely, you could set the missing value to 0 / 'NA' / -1, or you could impute the missing value. Some imputation methods are; calculating the mean value, carrying forward the last observation, and using a predicted value from a regression analysis. For this survey, one record was dropped because the respondent failed to complete the survey, a couple of missing values in other records were set to 0 or NA, and a couple of other missing values were imputed based on the previous answers given by the respondent.

Another data preparation task that was done was taking a continuous variable (number of employees at your company) and turning it into a categorical variable (micro, small, medium, large, enterprise). This task could easily be accomplished with a few lines of Python code.

def companySizeCat(numEmployees):
    if numEmployees < 7: # micro
        return 1
    elif numEmployees < 250: # small
        return 2
    elif numEmployees < 500: # medium
        return 3
    elif numEmployees < 1000: # large
        return 4
    else: # enterprise
        return 5 survey_df['COMP_SIZE'] = survey_df['COMP_SIZE'].apply(lambda x: companySizeCat(x)) 

Finally, in the survey there was a free text field with the question; "Name some things that you use at work". In order to be able to accurately do some form of text mining with that field, I opted to perform a number of transformations: I lowercased all the words in the field, standardized/stemmed certain terms (e.g. 'osx' and 'os x' both get standardized to 'osx'), and deduced broader tools from specific ones.

Examining the Data

Once you get your data into a Pandas data frame, examining your data is quite trivial. Especially if you are working in a jupyter notebook environment. Pandas gives you lots of methods, such as describe, head, tail, hist, info, median, mean, mode, std, nlargest, nsmallest, that you can use to gain an understanding of what your data looks like.

For example, calling the describe() method will show you various summary statistics for every column in your dataset.

survey_df.describe()

Answering the question; "Amongst the survey respondents, what is the most common company size, years of experience, number of jobs held, number of books read and salary?" becomes as easy as:

sel_columns_df = survey_df[['COMP_SIZE','YEARS_EXP', 'WORK_HIST_COUNT', 'BOOKS_READ', 'SALARY']]
sel_columns_df.mode()

Note: For the work history count there is a tie: 3 & 4 are both the most recorded number of jobs held.

Another question that Python's data science tools (scikit-learn and Pandas) enable me to answer is the question: What are the most common tools used mentioned by the respondents?

To answer this, first I need to split up the raw entered text into units representing each item. This is called tokenization. Then I need to create a matrix of the count of the occurrence of each token for every entry. This matrix can then be used in other algorithms but I'm simply summing the counts and then displaying the top counts.

# Define a function to tokenize the raw text entered by the users 
def tokenizr(text):
    tokens = text.split(";")
    return [ token.strip() for token in tokens if token.isspace() == False ]

>>>[u'.net',
 u'airflow',
 u'android',
 u'anger',
 u'angular',
 u'ansible',
 u'apache',
 u'apex',
 u'asp .net',
 u'asp mvc',
 u'atom',
 u'aws',
 u'baby bottles & burping cloths.',
 u'bash',
 u'bootstrap',
 u'brainpower',
 u'c++',
 u'centos',

javascript    39
linux         23
node          22
ruby          22
rails         19
postgres      19
nginx         16
php           15
mysql         14
angular       12

The results show that many of the survey respondents are frontend developers, or at the very least full stack developers, as JavaScript, Ruby on Rails, Nginx, PHP and Angular dominate the list.

Clustering the Respondent Data

The first question that I had that requires the use of machine learning algorithms is: What kinds/groups of developers answered the survey and what were the general characteristics of those groupings?

Being able to cluster respondents into groups may be useful for understanding and expecting similar behavior from the members of a certain group. It allows us to make assumptions about certain populations that are founded in data. For instance, it may be that the data shows that infrastructure engineers tend to make certain salaries, work 40 hours a week, work for companies of a certain size and have X number of prior jobs. Which could be in contrast to front end engineers, who maybe have a different salary range, work slightly more hours, have X number of years of experience and have read Y number of books on average.

To get started with clustering I used the K-Means implementation provided by SciKit Learn. I also performed a version of the clustering using SciKit Learn's Agglomerative clustering implementation but since the results I got were similar to the K-Means solution, I chose to focus on iterating my K-Means solution. K-means clustering aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean. My first iteration of the solution was simply fed the dataframe and asked to come up with 5 clusters:

# Using a basic KMeans algorithm
from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=5, max_iter=500)

# Attempt 1: Just give it all the data in its raw, naked glory and see what it comes up with
kmeans.fit(survey_df)

# Taking a look at what records were assigned to what cluster, I notice a disturbing trend: Salary is outweighing all other features
survey_df_c1 = survey_df.copy()
survey_df_c1.insert(0, 'CLUSTER1_LABEL', kmeans.labels_)
survey_df_c1.head(10)

# To prove that out a bit more I group by the cluster label and run some stats
cl1_groups = survey_df_c1.groupby(['CLUSTER1_LABEL'])
cl1_groups.agg([np.mean, np.min, np.max, np.std])

This naive solution failed in my opinion. Because a naive K-Means implementation is sensitive to scale, the salary feature vastly outweighed the other features in determining the clusters. So my result was clusters that were dictated by salary ranges. Respondents with salaries in ranges from 20K - 65K were in cluster 3, 68K - 100K in cluster 0, 105K - 185K in cluster 2, etc.

In my 2nd iteration of the K-Means solution, I normalized (scaled) the data so that all the fields now contained values between 0 and 1.

def normalize(df, columns):
    result = df.copy()
    for feature_name in columns:
        max_value = df[feature_name].max()
        min_value = df[feature_name].min()
        result[feature_name] = (df[feature_name] - min_value) / (max_value - min_value)
    return resultsurvey_df_normalized = normalize(survey_df, ['YEARS_EXP', 'NUM_HOURS_WORK', 'DAYS_OFF', 'SALARY', 'CONF_BUDGET', 'BOOKS_READ', 'WORK_HIST_COUNT'])
survey_df_normalized.drop(['STACK', 'COMP_MEALS'], axis=1, inplace=True) 

kmeans.fit(survey_df_normalized)

survey_df_c2 = survey_df.copy()
survey_df_c2.insert(0, 'CLUSTER2_LABEL', kmeans.labels_)cl2_groups = survey_df_c2.groupby(['CLUSTER2_LABEL'])
cl2_groups.agg([np.mean, np.min, np.max, np.std]) 

Thanks to the feature scaling, my 2nd iteration produced good clusters. Or did it? While scanning the tables revealed no noticeable skew from over-weighted features, its very difficult for humans to understand and visualize clusters in 174 dimensions. So how do you validate clustering results without visualizing them? The answer to that question is beyond the scope of this article, but a succinct and easy-to-understand document that explores this topic further can be found here.

In order to sidestep the difficulties of visualizing cluster data in high dimensions, for my 3rd iteration, I chose 3 features to cluster on. This allowed me to produce a 3D scatter plot of the respondents and verify that the clustering was successful.

survey_df_selected = survey_df_normalized[['SALARY', 'DAYS_OFF', 'WORK_HIST_COUNT']]

kmeans = KMeans(n_clusters=5, max_iter=500)
kmeans.fit(survey_df_selected)

Principal Component Analysis is a dimension reduction method that could prove to be useful in reducing the

N dimension vector space to the 2 or 3 components that are the most significant (show the most variance). While I didn't implement PCA for this project, I was able to make small steps towards marginal success in clustering the survey respondents.

Predicting Salary

The second question I had that requires the use of machine learning algorithms is: What are the important features in this dataset when it comes to determining salary and given those features can we predict an expected salary?

This distils down to being a classification problem and there are many algorithms to solve these kinds of problems; logistic regression, naive bayes classifier, support vector machines, neural networks, etc.

I chose to use a decision tree algorithm to solve the problem because decision trees have built-in feature selection, are easy to understand and interpret, work on both categorical and continuous variables and require little data preparation. Therefore it was a perfect match for my use case because it automatically answers the question; "What are the most important features in this dataset when it comes to determining salary". And because, being a survey with an expected margin of error, it makes sense for me to optimize for usability and demo-ability over accuracy.

The decision tree algorithm works by recursively splitting a set of data observations into subsets based on the attribute that most effectively splits the data set into groups (subsets) of observations with similar outcomes. This results in a tree structure in which each internal (non-leaf) node denotes a test on an attribute, each branch represents the outcome of a test, and each leaf (or terminal) node holds a class label or regression value.

from sklearn.tree import DecisionTreeClassifier, export_graphviz

y = survey_df["SALARY"]
X = survey_df.drop(["SALARY", "STACK"], axis=1, inplace=False)
tree = DecisionTreeClassifier(min_samples_split=4)
tree.fit(X, y)

(click to see larger figure)

For my dataset, the decision tree algorithm found the test "does your company provide more than 8.5 meals per month" to provide the most effective 1st split. If the answer is no then the left branch is taken and the next test becomes "do you have more or less than 8.5 years of experience?". If the answer to that test is "less than 8.5 years" then the left branch is taken, otherwise, the right branch is taken. The tests answer the question of "What are the most important features in this dataset when it comes to determining salary". The leaf (terminal) nodes show a prediction based on the observations from the dataset. So for instance; if your company provides less than 8.5 meals per month, you have less than 8.5 years of experience, you've read less than 1.5 books, and your company does not provide a 401K you are likely to earn $70,000. This prediction is based on the fact that 3 observations in the dataset with those same attributes earn that salary.

A couple of things to note: like many other machine learning algorithms, decision trees are vulnerable to overfitting. Overfitting means that the model created is too closely fit to a limited set of observations and hence it describes the random error instead of the underlying relationship. There are many ways to combat the overfitting problem that are better covered in a machine learning course. One interesting approach is to use random forests in place of decision trees. For the decision tree I present in this post, I exclude the "STACK" feature (i.e. survey respondent's skills) from the model. This was done for simplicity sake. I have a version of the model that takes "skill" into account but it has significantly more tests and although it appears to be more accurate, it is also more susceptible to overfitting. The final thing to note is that confidence in a decision tree's classification prediction can be quantified by outputting class probabilities. For regression trees, the confidence measure is the error. This blog post has more information.

After you train your decision tree model in Python using SciKit-Learn, predicting outcomes is as easy as calling the predict method on the model object. If you would like to create a rules engine from the decision tree, or you simply want a textual representation of the tree, check out the last section of this blog post or this stackoverflow question.

Conclusion

Here's what I learned in the process:

• To ask specific and deeper questions from your data, custom coding is usually your best bet. Most tools generalize to meet a broader set of needs. Your custom Python code is the best way to meet your needs and handle the specific data that you have.
• The Pandas library makes it easy to perform your standard statistical analysis on data tables with methods like std, var, mean, quantile, etc. (See full list)
• Clustering high-dimensional data is easy enough to do with the SciKit-Learn clustering implementations but validating those clusters is a harder problem that requires more research and publication
• Decision trees are a very robust tool for both feature importance and prediction but care must be taken to ensure you don't overfit your model.

There's a lot more insights that could be deciphered from the survey data given adequate amounts of data and the right machine learning tools and techniques. If anyone is interested in this project, check out the GitHub repository: github.com/tim-ojo/chadev_survey.