I've given up caffeine for short periods (by tapering to avoid the withdrawal headaches/triedness). I find my tolerance decreases and even a small amount of caffeine/coffee gives me a much more pronounced effect than when I'm regularly taking it. For instance I find it difficult to nap.
Anyway, the story is that when I was off the caffeine I once had three double espressos in a row. I ended up nearly having a panic attack in morning church. Full of anxious thoughts of imminent death and possible damnation.
Anyway, it passed. Then for the rest of the day until 3am I couldn't stop talking. I found myself really 'sparky' and even quite witty, a first for me. It was an amazing experience. I've so often lacked that spark and found myself collapsing in to myself at parties. This was the total opposite. However, it wasn't a party just friends together having. I really enjoyed myself. I've since found this to be true generally. If I can just get myself through the anxiety panic phase then it gives me social energy and I'm much less shy and even have fun. I think this is also the reason for the rampant use of amphetamines in performers. Imagine having to turn on all that wit and spark and high energy at exactly 6pm every evening. Better to be Norah Jones than Judy Garland, and so not need them at all. It's made me rethink high-energy, high-passion song composition. Imagine performing those every freaking day.
The downside is the panic attack problem. I've since realised that when I'm back on the coffee that two or three cups of strong coffee still makes me experience some measure of uncomfortable anxiety even despite full tolerance. I wasn't aware of this before. My brother, non caffeine drinker, also experienced a full blown panic attack when he injudiciously drank two liters of coke (about to die etc). Fortunately I was able to reassure him afterwards that he wasn't going mad.
I wouldn't be surprised if panic attacks are being caused when people have one or two strong coffees only in the morning and so never build up a tolerance.
Ultimately I'm going to give up caffeine for the long term. I might dip in to it for social reasons now and then, but there is another downside which is more important to me. It's linked to procrastination. According to a study a few years ago, the underlying cause of procrastination is impatience. Counter-intuitive, I know. I'll let you work it out. It then occurred to me that maybe the adrenaline effect of caffeine would feed impatience. Searching on the web I found a guy saying that when off caffeine he had no problem doing his todo lists, but when on caffeine he would end up doing low priority stimulating stuff instead of the high priority humdrum stuff. I remember as a child being very happy to create 25 mini passports for my teddy bears. But on caffeine I simply don't have the patience for humdrum enjoyment. Or slow movies. Or music. etc etc. I need to get my high priorities done, so the caffeine has to go.
One benefit of being off caffeine: I've almost always been a good sleeper. But morning wake-up is pain. I've made it easy by having an espresso by my bedside. Also painful with longer naps (up to 25 mins, with an alarm to avoid deep sleep of 30+mins). However, off the caffeine I wake up almost effortlessly. Also with naps. Just like when I was a child. And far less energy/sluggishness issues. But my sleep is much, much lighter and I need to use ear plugs.