I think I read your bones stop growing by your early 20s, so after that your skull is pretty much fully grown I would think. The fleshy parts of your head can get bigger, but I don't know if that affects hat sizing.
Those "one size fits all" hats never adjust to any size that fits me. I know my head isn't growing though since they haven't fit me in the last 20 years. Being mounted on a large frame, my head doesn't look abnormally large though. Similar to how size 13 shoes look normal on me, yet would look gigantic if you just had them sitting out on display (size 9 are the typical display models).