Background

The algorithm described by Rubio-Ramírez, Waggoner and Zha (2010) remain one of the most widely used tools of decomposing the reduced form errors of a vector autoregression (VAR) into economically meaningful shocks. The algorithm draws random orthogonal matrices which are then multiplied with a continuous function (usually Cholesky decomposition) of the reduced form covariance matrix to produce candidates for the structural impact matrix corresponding to the shocks of interest. Then the algorithm either accepts or rejects the candidate depending on whether its elements are in line with the assigned sign restrictions (see the paper for an actual detailed account).

The problem

This particular type of algorithm has garnered some concerns however. As is well known, increasing the number of restrictions increases the rejection rate rapidly, which in turn increases the number of draws required for a given amount of accepted draws. The practical implications for scientific work were highlighted by Baumeister and Hamilton (2020) while discussing the model of Kilian and Murphy (2014). In the latter, the reported model was estimated by 5 million draws out of which 16 were accepted. As demonstrated by the former, changing the random number seed (and thus the random draws and candidates) changes the core results significantly.

Uhlig (2017) argues that when many draws are rejected, the identification is sharp. That is, the data has informative opinions on the matter. This begs the question: how should we think about the aforementioned situation, where 0.00032 % of the draws are accepted but different random generations demonstratively give totally different model implications?

The first obvious (or rather apparently not so obvious) question to ask is; what does this statement even mean? In my view, there are at least two interpretations. First, one could interpret that retaining a small number of draws indicates sharp identification, which is a good thing. Second, it could also be taken to mean a small fraction of draws indicates a sharp identification. I shall argue, that the first one is potentially very mistaken and can lead to dangerous mistakes, while the second is totally reasonable. (To be clear, I have no idea which interpretation was in Harald’s mind.)

The whole point of sign restricting (a Gaussian model) is to narrow down a non-unique set of impact matrices that are observationally equivalent for a given reduced form point estimate. For arguments sake, let’s say one attains only one accepted draw for a given number of total draws. Should we conclude that the model is point estimated? Should we conclude that there exists no possible arbitrary neighborhood $\varepsilon>0$ for the draw that is also observationally equivalent? With the type of model we are discussing (that is Gaussian VARs) the answer is unequivocally no; the model simply lacks the information (or rather the ability to utilise such information from the data) to make such inference. Again, the point of these kinds of accept/reject algorithms is to approximate the whole observationally equivalent set of impact matrices.

Takeaway

I think it is fairly clear that to get a reasonable idea of the identified set one should not think of a number of total draws but of the number of accepted draws. This way one is guaranteed to get a reasonable estimate at least in light of the law of large numbers. Instead of running 5 million draws, run a loop that stops when 1000 draws have been accepted. Or if you are running an explicitly Bayesian model, for every reduced form posterior draw run a loop until one draw (or how ever many draws) is accepted. If using a toolbox instead of coding the sampler from scratch, it is worth checking what the toolbox does. This of course comes with immense computational hurdles for models with more than a few restrictions. For example, getting a 1000 accepted draws in the aforementioned Kilian and Murphy (2014) context could potentially mean something like 500 million draws. In practice, many problems are simply not doable with accept/reject algorithms. In such cases, one should revert to alternative algorithms.

Or alternatively: take advantage of other sources of identification information, such as fat tailed, skewed or heteroskedastic shocks – which almost always exist in macroeconomic data -, and strive for at least plausible point identification.

References:

Baumeister and Hamilton (2020) Structural Interpretation of Vector Autoregressions with Incomplete Identification: Setting the Record Straight

Kilian and Murphy (2014) THE ROLE OF INVENTORIES AND SPECULATIVE TRADING IN THE GLOBAL MARKET FOR CRUDE OIL

Rubio-Ramírez, Waggoner and Zha (2010) Structural Vector Autoregressions: Theory of Identification and Algorithms for Inference

Uhlig (2017) Shocks, Sign Restrictions, and Identification