We shouldn't modify CFLAGS

[real-or-random]

Originally posted by @theuni in #700 (comment):

I'll follow-up with a PR to move away from modifying CFLAGS itself. @jonasnick is exactly right, that's nasty.

[real-or-random]

I think that has been solved by b924e1e.