Consider using `Vec` directly in builders

[alamb]

          I think many of these instances could be changed to use `Vec` directly (and use optimized extend, etc. from them)

Originally posted by @Dandandan in #7309 (comment)