Methods for comparing the responses from a Likert question, with paired observations and independent observations in each of two samples

Abstract

Researchers often encounter two samples of Likert data, which contain both independent observations and paired observations. Standard analyses in this scenario typically involve discarding the independent observations and performing the paired samples t-test, the Wilcoxon signed-rank test or the Pratt test. These naive approaches are examined alongside recently developed partially overlapping samples t-tests that make use of all of the available data in the two sample scenario. For two samples of observations from a Likert question with five categories or seven categories, test statistics are assessed for their Type I error robustness and power. A summary measure of Type I error robustness across the simulation design is quantified as that value of such that (1- )100 percent of Type I error rates are within 100 percent of the nominal significance level. Across a range of sample sizes and correlation coefficients, the partially overlapping samples t-tests are Type I error robust, and offer a more powerful alternative for the analysis of two samples including both paired observations and independent observations. In these scenarios, when the assumption of an underlying continuous distribution is not inappropriate, the partially overlapping samples t-test is recommended.