Methods and Theory on Model Selection and Model Averaging

Description

Model selection and its diagnosis are foundational elements in modern statistical and machine learning applications that serve the purpose of obtaining reliable information and reproducible results. In this short course, we introduce the principles and theories on model selection and model averaging and their applications in high-dimensional regression. Model selection methods include information criteria (AIC, BIC etc), cross validation, penalized regression (LASSO, SCAD, MCP) and more. We will learn to understand their differences, connections, performances, limitations, proper uses, and approaches to achieving the best performance without knowing which method is the best for the data at hand. In addition, we will study new tools to characterize model selection reliability. When model selection uncertainty is high, model averaging/combining typically offers more accurate prediction and more reliable conclusions. Theoretical results covered include model selection consistency, consistent cross validation, adaptive minimax optimal regression learning in high-dimensional regression, and optimalities of model averaging methods.