Hypothesis Test Using Pearson’s Chisquared Test Algorithm
In this article, the author will show how to perform statistical hypothesis test using Spark machine learning API (aka Spark MLlib).
Join the DZone community and get the full member experience.
Join For FreeIn this article, I will show how to perform statistical hypothesis test using Spark machine learning API (aka Spark MLlib). The will be used to reveal the hypothesis testing with a stepbystep by example on the wellknown Wisconsin Breast Cancer Prognosis (WBCP) dataset.
In statistics, the Chisquared test can be used to test if two categorical variables are dependent, by means of a contingency table. However, before going into deeper, some background knowledge is mandated for them who are new to the area of statistical machine learning. Furthermore, a technical detail on the hypothesis testing using KolmogorovSmirnov (KS) test algorithm can be found in my previous article here.
What is Hypothesis Testing?
A statistical hypothesis [1] is a hypothesis that can be tested based on an observation modeled via a set of random variables. A statistical hypothesis test is, therefore, a method of statistical inference – where, more than one experimental datasets are compared. These datasets can be obtained by sampling and compared against a synthetic dataset from an idealized model [1].
In other words, a hypothesis is proposed and carried out for finding the statistical relationship between two datasets. More technically, the findings are compared as an alternative to an idealized null hypothesis that proposes no relationship between two datasets [1]. However, this comparison is deemed statistically insignificant if the relationship between the datasets would be an unlikely realization of the null hypothesis according to a threshold probability. The threshold probability is also called the significance level [1, 2].
The process of differentiating between the null hypothesis and alternative hypothesis [3] is aided by identifying two conceptual types of errors – also called type 1 & type 2 errors. Additionally, a parametric limit is specified on how many type 1 errors will be permitted[1, 3]. One of the most common selection techniques is based on either Akaike information criterion or Bayes factor [1] algorithm. Consequently, hypothesis testing is useful as well powerful tool in statistics to determine:
i) Whether a result is statistically significant
ii) Whether a result is occurred by chance or not
Because of the confirmatory nature, a statistical hypothesis test is also called confirmatory data analysis. Correspondingly, it can be contrasted with exploratory data analysis that might not have prespecified hypotheses [2, 3].
How Is it Performed?
According to the documentation provided by Wikipedia [1], the following two types of reasoning are considered for performing the hypothesis test:
Initial Hypothesisbased Approach
There is an initial research hypothesis for which the truth is always unknown. Now the below steps are typically iterated sequentially:
 The first step is to state the relevant null and alternative hypotheses
 The second step is to consider the statistical assumptions (i.e, statistical independence) being made about the sample in doing the test
 Decide which test is appropriate, and state the relevant test statistic say T
 From the assumptions, derive the distribution (i.e., most of the cases normal distribution) of the test statistic under the null hypothesis
 Select a significance level to say defined by αwhich is the probability threshold below for which the null hypothesis will be rejected (common values are 5% and 1%)
 Compute the observed value t_{obs} from the observations of the test statistic T in step 3
 Now based on the value, decide whether to reject the null hypothesis in favor of the alternate hypothesis or accept the hypothesis.
Note that, the assumption in step 2 is nontrivial since invalid assumptions will tend the results of the test invalid.
Alternate Hypothesisbased Approach
Here typically step 1 to 3 are iterated as an initial hypothesis as stated above. After that the following steps are performed:
 Compute the observed value t_{obs} from the observations of the test statistic T in step 4
 Calculate the pvalue [3], which is the probability under the null hypothesis. More technically, the value of sampling the test statistic at least as extreme as that which was observed
 Reject the null hypothesis, in favor of the alternative hypothesis, if and only if the pvalue is less than the significance level (the selected probability) threshold.
Now to compute the pvalue, here I describe two rules of thumbs based on several sources [13, 5]:
 If the pvalue is p > 0.05 (i.e., 5%), accept your hypothesis. Note that a deviation is small enough to take the hypothesis to an acceptance level. A pvalue of 0.6, for example, means that there is a 60% probability of any deviation from expected result. However, this is within the range of an acceptable deviation.
 If the pvalue is p < 0.05, reject your hypothesis by concluding that some factors other than by chance are operating for the deviation to perfect.
Similarly, a pvalue of 0.01 means that there is only a 1% chance that this deviation is due to chance alone, which means that other factors must be involved that need to be addressed [5]. Note that the pvalue may vary for your case depending upon the data quality, dimension, structure and types etc [2].
Hypothesis Testing and Spark
As already discussed that the current implementation of Spark (i.e., Spark 2.0.2) provides the hypothesis testing facility on the static as well as dynamic (i.e., streaming) data. Consequently, the Spark MLlib API currently supports Pearson’s chisquared (χ2) tests for goodness of fit and independence for batch or static dataset. Secondly,Spark MLlib
provides a 1sample, 2sided implementation of the KS test for equality of probability distributions. That to be discussed in the next section.
The third type of hypothesis testing that Spark provides support is using the streaming data. The Spark MLlib
provides online implementations of some tests to support use cases like the A/B testing [8]. Theis test may be performed on Spark Discrete Streaming that takes the streaming of type Boolean and double –i.e., DStream[(Boolean, Double)] [5]. Where, the first element of each tuple indicates control group (false) or treatment group (true) and the second element is the value of an observation.
However, in this article, I will show hypothesis testing using the Pearson’s chisquared (PCS) test algorithm implemented in Spark. However, the KolmogorovSmirnov (KS) test will be discussed in my next article.
Goodness of Fit Test Results Using the Pearson’s Chisquared Test
In this test, the input data types determine whether the goodness of fit or the independence test is conducted. The goodness of fit test requires an input type of Vector; whereas, the independence test requires a Matrix as input. Spark MLlib also supports the input type RDD [LabeledPoint] to enable feature selection via Chisquare independence tests [3]. The probability of obtaining a test statistic result at least as extreme as the one that was actually observed, assuming that the null hypothesis is true [3].
Here is how the pvalue is explained in Spark implementation (as Javalike pseudocode notation).
String pValueExplain = null;
if (pValue <= 0.01)
pValueExplain = "Very strong presumption against null hypothesis"+ $nullHypothesis;
else if (0.01 < pValue && pValue <= 0.05)
pValueExplain= "Strong presumption against null hypothesis"+ $nullHypothesis;
else if (0.05 < pValue && pValue <= 0.1)
pValueExplain= "Low presumption against null hypothesis"+ $nullHypothesis;
else
pValueExplain = "No presumption against null hypothesis" + $nullHypothesis;
Here we get to know the outcome of the assumption against the pvalue using the pValueExplain variable and its value changing over time and data objects.
Dataset Exploration
At first, we need to prepare dense vector from the categorical dataset like Wisconsin Breast Cancer Prognosis (WBCP) dataset. The details of the attributes found in WBCP dataset at [4] are:
 ID number
 Outcome (R = recurrent, N = nonrecurrent)
 Time (recurrence time if field 2 => R, diseasefree time if field 2 => N)
 4 to 33: Ten realvalued features are computed for each cell nucleus: Radius, Texture, Perimeter, Area, Smoothness, Compactness, Concavity, Concave points, Symmetry and Fractal dimension. The thirtyfour is Tumour size and the thirtyfive is the Lymph node status as follows:
 Tumour size  diameter of the excised tumor in centimeters
 Lymph node status  the number of positive axillary lymph nodes.
These values are observed at the time of surgery from the year 1988 to 1995 and out of the 198 instances, 151 are nonrecurring (N) and only 47 are recurring (R). Figure 1 shows a snapshot of the dataset. However, real cancer diagnosis and prognosis datasets like COSMIC, ICGC and TCGA nowadays contain many other features and fields in structured or unstructured ways [6].
Figure 1: Snapshot of the breast cancer prognosis data (partially shown)
Pearson’s Chisquared Test With Spark MLlib
In this example, I will show 3 tests to explain how the hypothesis testing works using the PCS test:
 Goodness of fit result on dense vector created from the breast cancer prognosis dataset,
 Independence tests for a randomly created matrix
 The independence test on a contingency table from the cancer dataset.
Step1: Load required packages
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.mllib.linalg.DenseVector;
import org.apache.spark.mllib.linalg.Matrices;
import org.apache.spark.mllib.linalg.Matrix;
import org.apache.spark.mllib.linalg.Vector;
import org.apache.spark.mllib.regression.LabeledPoint;
import org.apache.spark.mllib.stat.Statistics;
import org.apache.spark.mllib.stat.test.ChiSqTestResult;
import org.apache.spark.rdd.RDD;
import org.apache.spark.sql.SparkSession;
import com.example.SparkSession.UtilityForSparkSession;
Step2: Create a Spark session
static SparkSession spark = UtilityForSparkSession.mySession();
The implementation of the UtilityForSparkSession class is as follows:
public class UtilityForSparkSession {
public static SparkSession mySession() {
SparkSession spark = SparkSession
.builder()
.appName("JavaHypothesisTestingOnBreastCancerData ")
.master("local[*]")
.config("spark.sql.warehouse.dir", "E:/Exp/")
.getOrCreate();
return spark;
}
}
Step3: Perform the goodness of fit test
The following line of codes collects the vectors that we created using the myVector() method:
Vector v = myVector();
Where the implementation of the myVector() method goes as follows:
public static Vector myVector() throws NumberFormatException, IOException {
BufferedReader br = new BufferedReader(new FileReader(path));
String line = null;
Vector v = null;
while ((line = br.readLine()) != null) {
String[] tokens = line.split(",");
double[] features = new double[30];
for (int i = 2; i < features.length; i++) {
features[i2] =
Double.parseDouble(tokens[i]);
}
v = new DenseVector(features);
}
return v;
}
Now let’s compute the goodness of the fit. Note, if a second vector to test is not supplied as a parameter, the test run occurs against a uniform distribution automatically.
ChiSqTestResult goodnessOfFitTestResult = Statistics.chiSqTest(v);
Now let’s print the result of the goodness using:
System.out.println(goodnessOfFitTestResult + "\n");
Chisquared test summary:
method: Pearson
degrees of freedom = 29
statistic = 24818.028940912533
pValue = 0.0
Very strong presumption against null hypothesis: observed follows the same distribution as expected
Since the pvalue is very low (i.e., less than 5%) that is insignificance and consequently, we cannot accept the hypothesis based on the data.
Step4: Independence test on contingency matrix
Before performing this test, let’s create a contingency 4x3 matrix randomly. Here, the matrix goes as follows: ((1.0, 3.0, 5.0, 2.0), (4.0, 6.0, 1.0, 3.5), (6.9, 8.9, 10.5, 12.6))
Matrix mat = Matrices.dense(4, 3, new double[] { 1.0, 3.0, 5.0, 2.0, 4.0, 6.0, 1.0, 3.5, 6.9, 8.9, 10.5, 12.6});
Now let’s conduct the Pearson's independence test on the input contingency matrix:
ChiSqTestResult independenceTestResult = Statistics.chiSqTest(mat);
To evaluate the test result, let's print the summary of the test including the pvalue, degrees of freedom and the related statistic as follows:
System.out.println(independenceTestResult + "\n");
We got the following statistic as summarized follows:
Chisquared test summary:
method: Pearson
degrees of freedom = 6
statistic = 6.911459343085576
pValue = 0.3291131185252161
No presumption against null hypothesis: the occurrence of the outcomes is statistically independent.
Since the pvalue is significantly higher (i.e., 32%), we can now accept the hypothesis based on the contingency matrix. However, the size of the above matrix is very small which is clearly seen. Therefore, even though we have the required pvalue, we should not accept the hypothesis but let's try the hypothesis test using the independence test on contingency table instead with a larger dimension.
Step5: Independence test on contingency table
At first, let’s create a contingency table by means of RDDs from the cancer dataset as follows:
static String path = "input/wpbc.data";
RDD<String> lines = spark.sparkContext().textFile(path, 2);
JavaRDD<LabeledPoint> linesRDD = lines.toJavaRDD().map(new Function<String, LabeledPoint>() {
public LabeledPoint call(String lines) {
String[] tokens = lines.split(",");
double[] features = new double[30]; //30 features
for (int i = 2; i < features.length; i++) {
features[i  2] = Double.parseDouble(tokens[i]); //Exclude first two column
}
Vector v = new DenseVector(features);
if (tokens[1].equals("R")) {
return new LabeledPoint(1.0, v); // recurrent
} else {
return new LabeledPoint(0.0, v); // nonrecurrent
}
}
});
Well, we have constructed a contingency table from the raw (feature, label) pairs. Now let’s conduct the test as ChiSquaredTestResult for every feature against the label.
ChiSqTestResult[] featureTestResults = Statistics.chiSqTest(linesRDD.rdd());
Now let’s observe the test result against each column (i.e. for each 30 feature point) using the following code segment:
int i = 1;
for (ChiSqTestResult result : featureTestResults) {
System.out.println("Column " + i + ":");
System.out.println(result + "\n");
i++;
}
Column 1:
Chisquared test summary:
method: Pearson
degrees of freedom = 94
statistic = 85.96752752672163
pValue = 0.7103468748120438
No presumption against null hypothesis: the occurrence of the outcomes is statistically independent.
Column 2:
Chisquared test summary:
method: Pearson
degrees of freedom = 176
statistic = 180.50725658729024
pValue = 0.3921633980422585
No presumption against null hypothesis: the occurrence of the outcomes is statistically independent.
Column 3:
Chisquared test summary:
method: Pearson
degrees of freedom = 192
statistic = 197.99999999999957
pValue = 0.3680602522293952
No presumption against null hypothesis: the occurrence of the outcomes is statistically independent.
.
.
Column 30:
Chi squared test summary:
method: pearson
degrees of freedom = 0
statistic = 0.0
pValue = 1.0
No presumption against null hypothesis: the occurrence of the outcomes is statistically independent.
Conclusion
From the above result, we can see that for some feature point (i.e. column) we got larger pvalue compared to others. However, results vary on different columns. Readers are, therefore, suggested to select the proper dataset and do the hypothesis test prior applying the hyperparameter tuning. There is no concrete example in this regard since the result may vary against datasets you have. Readers are also suggested to find more on statistical learning with Spark MLlib at [3].
In my next article, I will show how to perform hypothesis testing using the KolmogorovSmirnov Test. The maven friendly pom.xml file, associated source codes and breast cancer prognosis dataset can be downloaded from my GitHub repository here.
Opinions expressed by DZone contributors are their own.
Trending

What ChatGPT Needs Is Context

Effortlessly Streamlining TestDriven Development and CI Testing for Kafka Developers

Is Podman a Dropin Replacement for Docker?

The SPACE Framework for Developer Productivity
Comments