A Method to Increase the Power of Monte Carlo Method: Increasing the Number of Iteration
Duygu Koçak
PEDAGOGICAL RES, Volume 5, Issue 1, Article No: em0049
https://doi.org/10.29333/pr/6258
Research Article
[Abstract]
[PDF]
[References]
ABSTRACT
Iteration number in Monte Carlo simulation method used commonly in educational research has an effect on Item Response Theory test and item parameters. The related studies show that the number of iteration is at the discretion of the researcher. Similarly, there is no specific number suggested for the number of iteration in the related literature. The present study investigates the changes in test and item parameters resulting from the changes in MC simulation studies based on Item Response Theory. In this respect, the required number of iterations is determined through IRT three-parameter logistics model test and item parameters under different conditions regarding sample size, item number, and parameter restrictions. The results indicate that estimate error can be lowered to a specific point and the test information increases as the number of iterations is increased and that the required number of iterations decreases as the sample size gets larger. However, it is also observed that the required number of iterations increases when intervals that would restrict parameters during data generation process are defined. It is concluded that the number of iterations has a significant impact on estimate results in MC studies and that the required number of iterations depends on the number of conditions and their levels. The more complex and featured the conditions are, the higher number of iterations will be required to achieve estimates without errors.
Keywords: Monte Carlo, simulation, iteration numbers, item response theory
REFERENCES
- Aiken, L. R. (2000). Psychological testing and assessment. Boston: Allyn and Bacon.
- Andersen, E. B. (1973). A Goodness of Fit Test for the Rasch Model. Psychometrika. https://doi.org/10.1007/BF02291180
- Baker, F. B. (1998). An investigation of the item parameter recovery of a Gibbs sampling procedure. Applied Psychological Measurement, 22, 153-169. https://doi.org/10.1177/01466216980222005
- Baker, F. B., & Kim, S. (2004). Item response theory: Parameter estimation techniques (2nd ed.). New York: Marcel Dekker. 38, 123-140. https://doi.org/10.1201/9781482276725
- Binois M., Huang J., Gramacy R. B., & Ludkovski M. (2019). Replication or exploration? Sequential design for stochastic simulation experiments. https://doi.org/10.1080/00401706.2018.1469433
- Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443-459. https://doi.org/10.1007/BF02293801
- Brooks, C. (2002). Introductory econometrics for finance. Cambridge University Press.
- Brown, R. L. (1994). Efficacy of the indirect approach for estimating structural equation models with missing data: A Comparison of Methods. Structural Equation Modeling: A Multidisciplinary Journal, 1(4), 287-316. https://doi.org/10.1080/10705519409539983
- Dias, J. G. (2006). Latent class analysis and model selection. In M. R. Kruse, C. Borgelt, A. Nürberger, & W. Gaul, From data and information analysis to knowledge engineering. Berlin: Springer-Verlag.
- Fay D. S., & Gerow K. A (2013). Biologist’s guide to statistical thinking and analysis, WormBook, ed. The C. Elegans Research Community, WormBook. https://doi.org/10.1895/wormbook.1.159.1
- Gifford, J. A., & Swaminathan, H. (1990) Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361-370 https://doi.org/10.1111/j.1745-3984.1990.tb00754.x
- Glen Satten, A., Flanders, W. D., & Yang, Q. (2001). Accounting for unmeasured population substructure in case-control studies of genetic association using a novel latent-class model. Am. J. Hum. Genet., 68, 466-477. https://doi.org/10.1086/318195
- Goldman, S. H., & Raju, N. S. (1986). Recovery of one- and two-parameter logistic item parameters: An empirical study. Educational and Psychological Measurement, 46(1), 11-21. https://doi.org/10.1177/0013164486461002
- Hambleton, R. K., Jones R. W., & Rogers, H. J. (1993). Influence of item parameter estimation errors in test development. Journal of Educational Measurement, 30, 143-155. https://doi.org/10.1111/j.1745-3984.1993.tb01071.x
- Hammersley, J. M., & Handscomb, D. C. (1964). Monte-Carlo Methods. Springer Netherlands https://doi.org/10.1007/978-94-009-5819-7
- Harwell, M. R., & Janosky, J. E. (1991). An empirical study of the effects of small datasets and varying prior distribution variances on item parameter estimation in BILOG. Applied Psychological Measurement, 15, 279-291. https://doi.org/10.1177/014662169101500308
- Harwell, M. R., Rubinstein E., Hayes W. S., & Olds, C. (1992). Summarizing Monte Carlo results in methodological research: The fixed effects single- and two-factor ANOVA cases. Journal of Educational Statistic, 17, 315-339. https://doi.org/10.2307/1165127
- Harwell, M. R., Stone, C. A., Hsu, T. C., & Kirisci, L. (1996). Monte Carlo studies in item response theory. Applied Psychological Measurement, 20(2), 101-125. https://doi.org/10.1177/014662169602000201
- Hoaglin, D. C., & Andrews, D. F. (1975). The Reporting of computation-based results in statistics. The American Statistician, 29, 122-126. https://doi.org/10.1080/00031305.1975.10477393
- Hulin, C. L., Lissak, R. I., & Drasgow, F. (1982). Recovery of two and three parameter logistic item characteristic curves: A Monte Carlo study. Applied Psychological Measurement, 6, 249-260. https://doi.org/10.1177/014662168200600301
- Hutchinson, S. R., & Bandalos, D. L. (1997). A guide to Monte Carlo simulations for applied researchers. Journal of Vocational Education Research, 22(4), 233-245. https://doi.org/10.1142/2813
- Kannan, P., Sgammato, A., Tannenbaum, R. J., & Katz, I. R. (2015) Evaluating the consistency of Angoff-Based cut scores using subsets of items within a generalizability theory framework, Applied Measurement in Education, 28(3), 169-186, https://doi.org/10.1080/08957347.2015.1042156
- Kéry, M., & Royle, J. A. (2016). Applied hierarchical modeling in ecology Volume 1: Prelude and Static Models Book. ScienceDirect.
- Lewis, P. A. W., & Orav, E. J. (1989). Simulation methodology for statisticians, operations analysts, and engineers. Volume 1. Wadsworth & Brooks/Cole, CA, USA: Pacific Grove.
- Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA, USA: Addison-Wesley.
- Mundform, D. J., Schaffer, J., Myoung-Jin, K., Dale, S., Ampai, T., & Pornsin, S. (2011). Number of replications required in Monte Carlo simulation studies: A synthesis of four studies. Journal of Modern Applied Statistical Methods: 10(1), Article 4. https://doi.org/10.22237/jmasm/1304222580
- Murie C., & Nadon R. (2018). A correction for the LPE statistical test. Bioconductor. Available at: https://www.bioconductor.org/packages/devel/bioc/vignettes/LPEadj/inst/doc/LPEadj.pdf
- Naylor, T. H., Blantify J., Burdick D. S., & Chu K. (1968). Computer simulation techniques. John Wiley and Sons, New York.
- Psychometric Society. (1979). Publication policy regarding Monte Carlo studies. Psychometrika, 44, 133-134. https://doi.org/10.1007/BF02293964
- Qualls, A. L., & Ansley, T. N. (1985, April). A comparison of item and ability parameter estimates derived from LOGIST and BILOG. Paper presented at the meeting of the National Council on Measurement in Education, Chicago.
- R Development Core Team (2011), R: A language and environment for statistical computing, a foundation for statistical computing. Vienna, Austria. Available at: http://www.R-project.org
- Rubinstein, R. Y. (1981). Simulation and the Monte Carlo method. New York: John Wiley and Sons. https://doi.org/10.1002/9780470316511
- Saeki, H., & Tango, T. (2014) Statistical inference for non-inferiority of a diagnostic procedure compared to an alternative procedure, based on the difference in correlated proportions from multiple raters. In: K. van Montfort, J. Oud, & W. Ghidey (eds) Developments in Statistical Evaluation of Clinical Trials. Berlin, Heidelberg: Springer. https://doi.org/10.1007/978-3-642-55345-5_7
- Sobol, I. M. (1971). The Monthe Carlo method. Moscow, Russian. https://doi.org/10.1007/978-3-642-55345-5_7
- Stone, C. A. (1993). The use of multiple replications in IRT based Monte Carlo research. Paper presented at the European Meeting of the Psychometric Society, Barcelona.
- Svetina, D., Ling Liaw, Y., & Rutkowski, L. (2019). Routing Strategies and Optimizing Design for Multistage Testing in International Large-Scale Assessments. National Council on Measurement in Education Annual Meeting in Toronto, Ontorio Canada. https://doi.org/10.1111/jedm.12206
- Wind, S. A., & Jones, E. (2019). The Effects of Incomplete Rating Designs in Combination with Rater Effects. Journal of Educational Measurement, 56(2), 76-100. https://doi.org/10.1111/jedm.12201
- Wolkowitz, A., & Wright, K. (2019). Effectiveness of equating at the passing score for exams with small sample sizes. Journal of Educational Measurement 56(2), 361-390. https://doi.org/10.1111/jedm.12212
- Yaşa, F. (1996). Rasgele değişen bazı fiziksel olayların 3 boyutlu Monte Carlo yöntemi ile modellenmesi (Unpublished Master Thesis). Kahramanmaraş Sütçü İmam University, Graduate School of Natural and Applied Sciences. Kahramanmaras, Turkey.
- Yen, W. M. (1987). A comparison of the efficiency and accuracy of BILOG and LOGIST. Psychometrika, 52, 275-291. https://doi.org/10.1007/BF02294241
- Zhang, X., Tao, J., Wang, C., & Shi, Z. H. (2019). Bayesian model selection methods for multilevel IRT models: A comparison of five DIC‐Based indices. Journal of Educational Measurement 56(1). https://doi.org/10.1111/jedm.12197