I know how you feel. Whenever that happens to me, if I try to smile, I look way more frightened rather than actually happy. I can't even keep my mind on what the person is saying. So by the time they finish talking and expect a response.. I'm just like.. "huh?" or "yeahhh, i know right?" *twitch* *queue freaky smile*
What helps me deal with it is practicing in the mirror. Just pretending to have a conversation with myself. Weird, I know but I'm pretty sure some people here do it.

Or sometimes, I webcam with friends and I still get nervous, so I use that as a way to observe myself talk. (that's how I found out I look frightened when I try to smile) So when I see that happening, I just calm down and make an actual smile. I think I've improved quite a bit from doing those things often.