Indeed. If it were me, I would probably smile to acknowledge them, go about my business, and only say something if they talk to me first. If the other person doesn't say anything, they don't really have the right to think negatively, because it takes two to be silent.I've done it before.
Citizen, they aren't going to think of anything negative about the encounter. Usually, people will say something quick since it is the bathroom. Maybe something like, "Well, back to work. See you around."