Why Early Retirement Terrifies Me

When everything goes wrong

Note: This post was first published in October 2020 and updated in April 2022

Ahh, the sweet promise of early retirement.

The date is January 31, 2020. The news outlets are slowly picking up coverage of the strange pandemic that’s shutting down Chinese cities and seems to be spreading beyond the country’s borders.

I have zero time to pay attention – because it’s 9 pm on a Friday night and I am just now leaving the office. To say the week has been brutal would be an understatement.

On Monday morning, I unexpectedly had to jump on an 11-hour flight and fly halfway around the world (remember those days when people used to travel?) to get a deal back on track.

Sadly, I already had a full calendar and despite my PA’s valiant efforts to clear it, I regularly find myself up at four in the morning, walking yet another client through the strategic merits of a potential deal.

I’ve come back to London late on Thursday, jetlagged and exhausted.

Friday was spent in a futile attempt to make a dent in a mountain of work that piled up while I was away.

I need to be back on the road in less than 36 hours, which means a big chunk of my Saturday will also be spent in front of my iPad, reviewing and commenting on documents.

At this point in time, I basically resent anyone who has time to sleep 7 hours a day and doesn’t have to work over the weekend.

And if you also happen to hit the gym for more than 5 minutes a week then I don’t even want to know your name – because I am beyond jealous.

I secretly long to join that exclusive club of people with a life.  Unsurprisingly, the thought of packing it all in crosses my mind… and instantly leaves me terrified.

The Early Retirement Rabbit Hole

According to the financial independence gospel, including the one I preach here, it shouldn’t be that way.

You work hard, make sacrifices, reset your lifestyle – and get to reap the rewards when you finally leave the corporate world behind.  Say hello to lie-ins, a healthy way of living, and all the free time in the world!

Sure, you may keep working in some shape or form.  Perhaps it’s a part-time job – or a volunteer position, helping advance a charitable cause.

Looking after your children – or parents.  Learning a new language or picking up a hobby.

All highly engaging and worthwhile endeavours, no doubt.  But as I close my eyes and visualize life after work, I simply can’t shake the nagging question:

Once the office door shuts behind me on the way out, once I catch up on my sleep and give my family and health the attention they deserve, will I be happier?  Or will I miss the days gone by?

The reality is that cashing in the chips in the form of early retirement isn’t a free option.  It comes at a price – and the loss of many things we may not be ready to part with just yet.

And as the boisterous (and maskless) crowd around me gets ready to kick off the weekend, I reflect on the things I will miss once I pull the trigger.

Status Games

Might as well start with the elephant in the room.

Being brutally honest with myself, the loss of status after retirement will be the biggest mental challenge to overcome.

Despite taking a significant image beating post-crisis, being a senior investment banker still carries a certain cachet – and I will sure miss it once it’s gone.  And no, it has nothing to do with money (more on that later).

Rather, it’s the social capital I’m worried about.

Put simply, social capital is a critical enabler of trust, reciprocity, and collaboration with the people around you.

Whether you are a university professor, a doctor, a police officer – or a CEO – the job that you do likely means people relate to you in a different way than they would otherwise.

It’s their willingness to take your phone calls – and the amount of time they give you if they do answer. The desire to plug you into their network.

To extend a favour – because they know that you’d be willing and able to reciprocate.

And yes, that also applies to investment bankers.

There is no 4% rule for social capital.  It doesn’t compound and you will struggle to top it up once you’ve run out.

Is it a deal-breaker?  Depends on the person.

But regardless of how you feel about it, you better be prepared to be treated differently once you retire – and no degree of financial independence can change that.

Less Money, More Problems?

Walking away from a well-paid career is always a challenge.

There’s the natural desire to get our two children off to a great start in life.  The 16 or so years before they are ready to live independently will likely present unforeseen expenditures.

We can either account for these things by increasing our FIRE number – or by staying in employment for longer.  At the end of the day, the two things are equivalent.

We’ve also got two sets of aging parents.  Given everything they’ve done for us, we would like to be able to step up and take care of them if need be.  Top up the FIRE number yet again…

And while everyone talks about spending less in retirement, what if I actually end up spending more?

Over the years, I have found that being an investment banker actually helps me live well below my means.

I don’t feel the pressure to own a car.  I rocked an iPhone 6 for so long that it literally died on me one day.  You won’t find any monogrammed shirts in my closet.

Partially, it might be because I am a pretty low-key person.

But what if it also has something to do with the fact that people know that I can easily afford all of these things.  I just choose not to.

We often underestimate how much we change throughout our lives.  For all I know, I may well be pressured to spend more money once I’m out of the game – just to continue fitting into my current social circle.

Great Expectations?

My grandfather left his little village behind and used sheer willpower and hard work to build a new life for himself in a big city.  He enjoyed life, but he also worked well into his 60s and passed on not long thereafter.

My parents, who led a relatively comfortable lifestyle (partially thanks to my grandfather), took massive risks and left everything behind to emigrate to a new country.

Despite long being able to retire, they keep working to this day.

How will my decision to retire around the age of 40 fit into the family narrative?  How will I feel being the person who stopped early, trading our our family’s progress for early retirement?

The beauty of not coming from a wealthy family is the ability to live your own life.  Unlike some people I know, I don’t have to worry about being cut out of the inheritance if I don’t toe the party line.

And yet, as many first-generation immigrants will attest, I am not a free man.  The immigrant ethos of hard work, sacrifice, and parental approval hangs above my head to this day.

Given it’s not gone away in four decades, I doubt it’s going away anytime soon.

Intellectual Stimulation

Despite what some media articles would like you to believe, investment bankers aren’t a bunch of scammers walking around looking for their next victim.

On the contrary, the fundamental premise of the job is connecting capital with the most productive investment opportunities – whether in the form of debt/equity financing or strategic M&A.

Over the years, I’ve built up great relationships helping clients work through some of the most fundamental challenges facing their business.

It may take some round-the-clock effort, but it sure feels rewarding.  More importantly, it also keeps me mentally sharp.

Not Easy Street by a stretch, but as Pete Barton says in one of my favourite books:

Staying on a track can kill, one easy day at a time.

There’s no earth-shattering revelation at the end of this post. I got off the tube, walked home, had dinner with my wife, spent the weekend mixing up work and play.

Come Sunday, I was back on the road.  As it turned out, it was my last business trip for a long time.

It’s now been more than two years since that introspective tube ride of mine.  We live in a totally different world, and I’m happy to say I no longer do nearly as much travel as I used to.

As far as the other questions?

Money is slowly being solved, though I do have a pretty ambitious “number”.  I’m also confident of my ability to find intellectual stimulation once I leave banking behind.

Shaking off the chains of social status and breaking the family narrative may take a bit more work still.

In other words, the journey continues – but at least I’m being honest with myself.  And that just might be the most important thing of all.


About Banker On Fire

Enjoyed this post?

Then you may want to sign up for our exclusive updates, delivered straight to your inbox.

You can also follow me on Twitter or Facebook, or share the post using the buttons above.

Banker On FIRE is an M&A (mergers and acquisitions) investment banker. I am passionate about capital markets, behavioural economics, financial independence, and living the best life possible.

Find out more about me and this blog here.

If you are new to investing, here is a good place to start.

For advertising opportunities, please send an email to bankeronfire at gmail dot com

43 thoughts on “Why Early Retirement Terrifies Me”

  1. Ha – well, you know my take on this one!

    I worried about a lot of the similar things but two years in, can now see they were unfounded, for me.

    But you do have to work at it to make it happen that way – and absolutely, it does start with being painfully honest with yourself.

    I think half the problem is that those of us who do the work hard/succeed path – it’s literally all we know, all we’ve done – until we stop…

    So for me early retirement was a bit of an experiment too, as to how I could have a life outside work. And that it could be as challenging and meaningful.

    The result so far, as you know, is that’s it’s all that plus more. A lot more. But if you don’t go into it with the right mindset, then yes, I can see why a lot of people get scared back to work quickly.

    I think you are self-aware enough to figure this one out, look forwards to hearing you do so.

    1. I think you are right. Like most things in life, the “you won’t know until you try” concept applies here.

      But, as you say, a lot of work to be done still. Reason I came back to this post 9 months later is that I vividly recall how intensely I felt that packing it in might be the wrong / suboptimal move.

      Good thing there are people like you who have walked the path before and can share the experience!

  2. High performance elite athletes often struggle once “retiring” from the game. Driven and motivated people will always do more than the average expected.

  3. With regards to status, I definitely recognize the importance of social capital even though I just started out my career in banking (although not within the traditional IB). However, while it’s true that you MAY lose your status as an IB, I think you are already building a pretty solid status as a blogger. Once you leave the financial industry and you decide to reveal your true identity, I believe this “new” status could potentially bring even more fulfillment and a more solid base of social capital.

    With respect to the family narrative, in my opinion even if you decide to stop working within IB, it doesn’t mean that you actually stop working. Even if you only continue to write your blog (which takes a lot of work and that it seems you enjoy), to me that still counts as work. Maybe you become a full-time blogger, maybe you end up building a business around your blog, maybe you write a book (which I would definitely want to read) etc etc.

    I think this would allow you to live a lifestyle that FIRE preaches, have a solid status (which I think you already have within the blogging sphere) and continue “working” (thus fitting into the family narrative). Additionally, you’d behind a legacy by building generational wealth (if you plan to do that) and influencing a lot of people’s life positively (either through your blog or a potential book). Obviously, I realize that these may not be important in your culture.

    Thanks for another great article and sharing a bit more about yourself!

      1. Thanks to both of you for the vote of confidence in my writing ability!

        Jon – I think your points are spot on. Many different ways to skin the cat. At the end of the day, it’s more about rising above the environment that occupied your life over the past decade to get a glimpse of everything else is out there.

        At the moment, becoming a full-time blogger / managing our rental properties just seems like a stretch post IB, but it could well change.

        I’ll keep you guys posted on how my thinking evolves!

  4. You show a lot of self-awareness pondering these issues so honestly.

    For what it is worth, here are a couple of thoughts based on my own experience.

    The familial monkey on the back is all mental. We choose to carry it around. Much like giving up drinking or stopping biting our finger nails, setting it down is also a simple choice. Simple, but not easy! Some folks think that parental mortality will one day cure it, failing to realise that perceived expectations live on, and it is hard to win an argument with a ghost.

    Social circles inevitably change after you pull the pin, as most of the work centric relationships fall away. This can be brutal if you have lived and breathed work every waking moment for years.

    You rapidly go from an insider to bystander. No longer an ally or a threat. No skin in the game or role to play. You lose interest in the day-to-day institutional politics, and your knowledge of the current deal quickly becomes dated. Reminiscing about past glories is fine occasionally, but your time-poor former colleagues quickly move on to the next deal.

    I would liken it to what happens to school friendships after graduation, the majority revealed to be acquaintance of convenience and circumstance rather than genuine friendships that persist beyond.

    But here is the thing. After a while, we start to realise that the conferred status was an illusion, and most of the ego gratification stuff was bullshit. Six months after we leave who will remember any of the deals we did or projects we delivered? After a year, how many will remember we were ever there? Few, if any.

    We do the work because we loved the hunt, the thrill of the chase. Too old to be athletes. Too slow to drive racing cars. Too poor to race maxi-yachts. This was our means of showing the world how good we were.

    Perceiving ourselves to be superheroes in suits, saving the day as we strut amongst the glass towers of Canary Wharf, or jet around the world doing our very busy work.

    Meanwhile, the slick estate agents and slimy car dealers watch us rushing between meetings along Upper Bank Street. They don’t see masters of the universe, they see us for what we really were: prey in expensive suits.

    It is instructive to have a chat with them one lunch time over a burrito or a roti roll. Humbling too. Provides an interesting sense of perspective, as they have seen generations of players rise and fall. Succeed. Fail. Or just age out.

    Which may help to figure out what is really important, both during our working life and beyond.

    1. I second that. Indeedably – your comments can give most full-fledged blog posts a run for their money.

      A couple of reactions.

      On familial expectations – perhaps there are none in the first place and this is something we make up ourselves. The reality is I never had a heart-to-heart with my parents on this one, but the conversations I did have never explicitly suggested the expectation to keep working. Could well be that now being in their 60s, they could well encourage me to pull the plug earlier!

      As far as work associations – well put. I’ve seen many folks cycle out of the industry. A leaving-do and a cursory lunch / coffee a few months later is where it ends. Most of them disappear off the horizon, no longer part of the relevant ecosystem.

      And on your last point, I often feel like I have a bulls-eye on my back when dealing with anyone who tries to sell me something. This is when my immigrant background comes in handy. Amazing how people like that start treating you in a very different way you if you don’t mind telling them off.

  5. it’s not binary – work or not work. I’ve been doing 2-3 days a week for a while which gives me the advantages of work that you raise above but also gives me precious time as well. I also think it eases any future transition, either back to full time work or to full FIRE!

  6. Accountantonfire

    Fantastic post as always, BOF! I am in the Transaction Services, doing FDD (I’m sure you know what I’m talking about). So my working lifestyle is broadly similar (albeit may not be as intense as an IBer like yourself). A lot of food for thoughts here as I have been always struggling between taking more time for family vs. doing a good job in an intense work environment.

    1. Of course 🙂 As you can imagine, I spend quite a bit of time reading FDD reports, especially the QoE sections.

      Frankly I’ve always felt you guys weren’t far off IB in terms of work intensity. I think key difference is you have less competition for new business (given really just a few players in the space), unlike banks that cap out at a 10% market share.

      As a result, we often need to pitch for new business while being on multiple live deals at the time. Always a good time…

  7. Love the post and comments!
    My take: unless you die while still working, you’ll have to face the loss of social capital at some point, so are you just delaying the inevitable? And does changing career/activity sooner rather than later give you a better chance of dealing with it as you’ll be younger and therefore more adaptable and have more time to reinvent yourself?

    1. Thank you. Very well put, I hadn’t thought about it this way.

      All goes back to the question of framing retirement as leaving to do something else as opposed to simply leaving something behind.

  8. I find that having things to do in retirement helps such as volunteer work and hobbies. suddenly leaving the workforce after many years comes as a bit of a shock.

    1. Yeah. Problem exacerbated by the fact that in intense jobs, you don’t often have the time for extracurriculars like that. Once you pull the plug, you are essentially starting from scratch.

      Ultimately, that’s one of the reasons I started this blog – to give me a creative outlet but also something to do if / when I leave IB.

  9. Everything is new is scary! I said keep on working until you don’t like it anymore.

    I was an executive director at worked at two bulge bracket firms. It was sad to lose my I didn’t eat for a little bit, maybe six months. But then it was a slow reinvention everything turned out fine.

    The status thing really is overrated, especially after having a passive income to cover your living expenses. I think that’s more status than any type of job title.

    My parents didn’t give a crap what I did either, so long I was healthy and happy. And I was and I am.

    Sam

  10. I think I would have challenges in a forced retirement. I’m sure that if I dug deep I would find the anchors that tie me and they would likely be different than yours. For me the FIRE movement is mostly about the FI part, I haven’t really given the RE part enough thought and for me and I think for a lot of people having a purpose is essential.

    Great post as always.

    1. Thanks Matt. For me the FI part has always been more important. In a way, my idea of retirement is working 40 hours a week instead of 80 🙂 – as well as finding a meaningful pursuit to occupy those 40 hours with.

  11. This is a really great summary of many of my fears as I approach financial independence and consider walking away. Like you, I worry less about the money and intellectual stimulation part, but definitely think about social capital and great expectations. As an introvert, the social part will not be rebuilt easily and you’re exactly right – it doesn’t just keep growing. As for the expectations, while not an immigrant the rise from poverty into status does matter to my family. They, more than I, will struggle with my choice to walk away from a prominent community position. This was a great read that helped clarify some of my thoughts – thanks!

  12. The social capital at work is all based on power, status, and money. That’s not a solid base for a real friendship. If you enjoy that kind of relationship, then you shouldn’t retire early.
    After early retirement, I left all of that behind. I had to start over and build social capital with friends, neighbors, and other people I like.
    If I’m treated differently because I retired, then those people aren’t worth my time.

    1. You are right, the social capital that’s based on work status is very transactional.

      That being said, I’m not viewing it through the prism of friendship. Rather, it’s the ability to extend (and call in) favours that make our lives easier.

      I won’t necessarily miss it, but equally something to be cognizant of.

      As you say, the way to replenish it is to build up relationships that aren’t based on power and status.

  13. An interesting one- as you seem to define yourself based on your job/work, rather than who you are? I think that’s how many people see themselves- since that’s what they are doing for most of their waking hours (and it sounds like even more so for you!). And most people will do this, for all of their working lives, so it’s going to be pretty hard to let go. However, if you aren’t driven to do this forever, then naturally there is going to be a change- which should be planned to ensure that it doesn’t happen overnight. I think everybody has heard about somebody that retired at 65, and within a year either died, or got so bored they went back to work. It feels like FIRE could end up being the same without this planning for what’s next.

    1. It’s the whole “retire to” vs “retire from” conversation.

      I wouldn’t say I define myself based on my work, but you are 100% right in saying that when work occupies the vast majority of my waking hours, it quickly comes to dominate the “self”.

      So while planning is important, I suspect that in my situation I need to pencil in a “rediscovery” period of one-two years in order to find the right way to spend the next decade or so.

  14. Tolerable post and comments ?

    One thing not really touched on is the company one can keep and do things with during this new infinity of empty days. I’ve previously been “between jobs” twice, once for 6 months. It was great at first but you soon realise that everyone else your age is still working full time. Sounds obvious but going to the pub or playing tennis ? by yourself is quite hard work!

    I’ve just taken redundancy and can afford, just about, to retire at 55. I’ve 6 months of renovating my house but after that I’m convinced I’ll be looking for a van to drive or something! I couldn’t give a fig for social capital, I just don’t want to be bored sh!tless !!!

    1. Great point. Given we’ve got two kids at home, getting bored is the last of my worries but I agree that being “off” isn’t nearly as much fun if everyone in your social circle is still grinding away.

  15. Great post and it’s something that I am thinking about carefully.
    My 15 years work experience is in an industry that is doubly impacted in 202 and it will probably result in the loss of maybe a third of the jobs (and half the wages) permanently.
    It’s complex of course but my position is that I am basically FI and don’t really need to work anymore but I am also not working and there’s no great prospects ahead.
    So what to do? Sit on the career scrapheap? Retrain for a new industry? Hold on on the hope that more work comes along?

    The difference between retirement and being redundant is a fine one – helped by having savings but not solved

    1. Interesting perspective. As you say, money is an important component but not the only factor at play.

      Fundamentally, should boil down to what you really enjoy, shouldn’t it? You can be the master that maintains the light of a declining craft…

  16. BOF, would you consider launching a podcast and delivering your fantastic content verbally? These short, sharp, posts would be perfect to consume audibly on my morning walk (which I now do to replicate my commute). Thoughts?

    1. Thanks a lot Josh, appreciate the compliment.

      I’d love to do more here – both in terms of content and delivery. Unfortunately, between investment banking for a day job and two young children at home, doing anything more than two blog posts a week is already a challenge.

      Perhaps in the future, once I leave my current job behind or if this blog actually generates enough income to allow me to hire some help and outsource the more menial tasks 🙂

  17. Interesting post and honest reflections/fears even.
    The word status is interesting – I don’t see or understand it’s importance when you are considering sacrificing quality time with growing children. You are currently young, fit and healthy, – all of which we take for granted until they are taken away and then it’s too late. What does status matter then?
    I am a Firefighter, in 6 months I will be FI.
    I currently have status having risen through the ranks to a senior position.
    I have seen the fragility of life, professionally and personally.
    Life is for living, while you can, whenever you can and at every opportunity.
    Status is something that I am happy to walk away from.
    I do understand that for others this is not the case.
    Remaining in the workplace to secure your families future and security I absolutely understand and applaud you for, but not for status.
    But I’m not sure that is what you were saying?

    1. Thank you for a thought-provoking and poignant comment – and for what you do day in and day out, which has far more societal value than most jobs (including mine).

      Your points are spot on, though you are also right in saying that this isn’t quite the point I am making. Status for the sake of status doesn’t bother me, though I am sure it matters for some.

      What worries me is losing the ability to make life easier for my family (especially my kids). Perhaps help get my kids a foot in the door with a summer internship, or the ability to connect them to someone in a grad program they want to apply to. That sort of thing. I have no misconceptions that many people in my social circle won’t be willing to extend favours once they know I cannot reciprocate.

      Then again, it’s not something I ever had, so I’m sure they’ll be just fine. Everyone has to start fending for themselves at some point 🙂

  18. Hi BOF,

    You raise many interesting questions that I think are central, not to achieving FI, but to making the most of FI once you’ve achieved it.

    My main question is: what kind of life do you want to lead?

    Does working 80 hour weeks for the next decade or two, in the hope of reaping a few favours because you’re an investment banker, fit with your values and what you want from life?

    What is the opportunity cost of those 80-hour weeks?

    What values underpin the future you imagine? What does a life well-lived mean for you?

    Do you wish to move in a social circle where you are valued because you’re an investment banker? Or a social circle in which you are valued for who you actually are?

    I’ve enjoyed the posts in which you reveal your strategies for climbing the corporate ladder. I wonder if the next stage is to devote some of your strategic talent towards plotting a similar path for the world beyond the corporate.

    1. Great points – and very thought-provoking questions.

      You are absolutely right. The biggest concern, as you may have seen in one of my replies above, isn’t the loss of status. It’s the opportunity to give my children a leg up by calling in some of those favours, cashing in on some of the social capital built up over the years.

      Framing it in the context of opportunity cost provides a helpful perspective. As my capital markets colleagues would say, it’s not a good trade!

  19. Agree with Indeedably on this one. At some point the importance of status (retired lawyer here) starts to fade and you realise its all pretty meaningless. Then comes a period of adjustment as you start looking at things in a new light. Its not something you can rush – it will happen and new horizons will open up.

  20. Agree with Indeedably on this one – status and the importance of job titles etc are an illusion. Once that realisation truly hits, whether suddenly or gradually (and the epiphany comes sooner or later for most Im sure), there’s no going back! Adjusting to that and working out what will provde more meaning going forward, takes time. This doesnt detract in any way from previous achievements (and of course the financial benefits) gained from working in a high status job. It just becomes time to move on.

    1. Banker On FIRE

      Appreciate the perspective – and I do agree with you (and indeedably as well).

      Easy to get caught up in the day to day and imagine your work / contribution to be more impactful and important to people than it really is.

      Leaving the industry tends to clarify these things pretty quickly as folks drop off your radar almost instantly.

  21. Great post as always BOF!

    Quick question from me: “You won’t find any monogrammed shirts in my closet.” – Where do you buy your suits from?

    1. Banker On FIRE

      Shirts from Brooks Brothers.

      Suits from Suit Supply – relatively inexpensive and fit me better than $5k suits ever did!

Leave a Reply