Seems to be a problem on riot-web 0.16.3. It wasn't a problem on 0.16.1, so it must be some regression that occurred in 0.16.2 or 0.16.3.
For direct chats with only 1 other person, the default room avatar is wrong when the other person doesn't have an avatar image. Instead of using an avatar showing their display name initial, riot-web would instead show my own avatar for the room.
For direct chats where the other person has an avatar image, the room avatar would default to that person's avatar, as expected.
In addition to this, rooms with just myself (self chat) display an avatar with my initial instead of my actual avatar, like one would expect.