See the discussion in #480 for details. We need to see when and how we want pointers of matrices (including empty matrices) to be aligned.