Missing value imputation in a data matrix using the regularised singular value decomposition

Sergio Arciniegas-Alarcón, Marisol García-Peña, Wojtek J. Krzanowski, Camilo Rengifo

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

Some statistical analysis techniques may require complete data matrices, but a frequent problem in the construction of databases is the incomplete collection of information for different reasons. One option to tackle the problem is to estimate and impute the missing data. This paper describes a form of imputation that mixes regression with lower rank approximations. To improve the quality of the imputations, a generalisation is proposed that replaces the singular value decomposition (SVD) of the matrix with a regularised SVD in which the regularisation parameter is estimated by cross-validation. To evaluate the performance of the proposal, ten sets of real data from multienvironment trials were used. Missing values were created in each set at four percentages of missing not at random, and three criteria were then considered to investigate the effectiveness of the proposal. The results show that the regularised method proves very competitive when compared to the original method, beating it in several of the considered scenarios. As it is a very general system, its application can be extended to all multivariate data matrices. • The imputation method is modified through the inclusion of a stable and efficient computational algorithm that replaces the classical SVD least squares criterion by a penalised criterion. This penalty produces smoothed eigenvectors and eigenvalues that avoid overfitting problems, improving the performance of the method when the penalty is necessary. The size of the penalty can be determined by minimising one of the following criteria: the prediction errors, the Procrustes similarity statistic or the critical angles between subspaces of principal components.

Original languageEnglish
Article number102289
JournalMethodsX
Volume11
DOIs
StatePublished - Dec 2023

Keywords

  • Cross-validation
  • Eigenvalues
  • Eigenvectors
  • Genotype-by-environment interaction
  • Iterative computational scheme
  • Overfitting

Fingerprint

Dive into the research topics of 'Missing value imputation in a data matrix using the regularised singular value decomposition'. Together they form a unique fingerprint.

Cite this