Shall I use AIC or BIC?

A common question which we usually face in applied econometrics problems is which model is the "best". Several criterias have been suggested such as the Akaike's information criteria (AIC) and Bayesian information criteria (BIC). The minimum the values of these criterias, the better the model. Nonetheless, quite often you select different models using BIC or AIC. So a question that stands out is: Which criteria is better?

In model selection literature, it is know that BIC is consistent in selection when the "true" model is finite dimensional, and the AIC is asymptotic efficient when the "true" model is infinite dimensional. Then the choice would be easy if we know the nature of the "true" model. 

However, in practice, we have no idea what kind of animal we are dealing with: finite or infinite dimensional, and we are still unable to decide which criteria to use.

And here is where a new paper by Liu and Yang, "Parametric or Nonparametric? A Parametricness index (PI) for model selection", published at The Annals of Statistics in 2011, comes to help us.

Liu and Yang (2011) develop a measure, called parametricness index, to check if the model selected by a consistent procedure can be treated as the "true" model.  

In order to understand better what is in fact this Parametricness Index, let's cite Citing Liu and Yang (2011):

"While there are many different performance measures that we can use to assess if one model stands out, following our results on distinguishing between parametric and nonparametric scenarios, we focus on an estimation accuracy measure. We call it parametricness index (PI), which is relative to the list of candidate models and the sample size. Our theoretical results show that this index converges to infinity for a parametric scenario and converges to 1 for a typical nonparametric scenario. Our suggestion is that when the index is significantly larger than 1, we can treat the selected model as a stably standing out model from the estimation perspective. Otherwise, the selected model is just among a few or more equally well-performing candidates. We call the former case practically parametric and the latter practically nonparametric".

So, when the PI is close to 1, several models shares the same properties and it is hard to pic one of them, hence we cannot treat the selected model as the "true" one. Liu and Yang (2011) called this situation as "practically nonparametric". When the PI is significant larger than 1, the selected model is expected to perform better than the others (within the given sample size), and we could treat it as the "true" model - this situation is called "practically parametric". By their numerical exercises, it is suggested that the fact of the true model being parametric or nonparametric do not matter (in finite samples), but what matter is the if we are in a "practically parametric or nonparametric" framework!

Using this idea, Liu and Yang (2011) proposes that whenever the PI is significant bigger than 1 (they use 1.2 as a suggestion), one should use BIC. Otherwise, one should use AIC. 

Ahá!! That's what we were looking for! A date-driven way to choose between AIC and BIC! 

Now the work that has to be done is to code the suggested procedure, so we could use it often! If anyone knows if the code is available somewhere, let me know, please!

Obs: The criteria is developed under linear Gaussian models only, but the results seems promising.