This report describes how sample design and measure reliability affect the power of hypothesis tests from LISREL's maximum Likelihood ratio. Using simulation data and data from the High School and Beyond survey (HSB), I show that test power decreases when sample design differs from an unrestricted random sample assumed for LISREL's maximum likelihood estimators (MLE).
The covariance model selected for the study includes three latent variables with two fallible indicators per variable. In simulation, the intracluster correlation among sample elements and element weights are manipulated. The measurement error variance for one variable is altered on different trials to change the composite reliability of one latent variable.
Power effects of the study's manipulations are combined into a single measure called "EFFN", a multivariate index of effective sample size derived from Wilks' likelihood ratio. EFFN may be used for planning covariance studies using data from complex sample surveys.
The results show that as cluster effects increase, the power to reject null hypotheses reduces at an increasing rate. No power effects from the sample weights were observed. Test power increases with improved reliability but as clustering effects increase, the power benefits from the more reliable measure are lost.
A second part of the study is a secondary analysis of two gender differences studies which used HSB data. The purpose of the analysis is to resolve the conflicting results reported from these studies. In a study by Ethington and Wolfle (1986), the researchers conclude there are gender differences in mathematics achievement while in another study by Marsh (1989), the researcher draws the opposite conclusion.
The analysis suggests that the opposing conclusions stem in part from the different ways that the researchers adjusted their LISREL based hypothesis tests for the HSB sample design. By adjusting both studies in the same way for sample design effects and by using a common significance criteria, the results from both studies tend to agree.