What's your strategy for solving problems? If you're not familiar with the concepts of strategic optimism and defensive pessimism, it's worth a quick read but I'll summarize briefly here.
They are contrasting cognitive strategies for problem solving. Strategic optimists visualize what success looks like and determine the steps needed to emulate that visualization. Defensive pessimists, on the other hand, visualize what failure looks and determine the steps needed to avoid those failure scenarios.
It's important to recognize that they are equally valid and successful strategies, and highly personalized it turns out. Research shows that people tend to center themselves on one approach or the other and have roughly equivalent performance outcomes, but when asked to switch strategies their performance decreases.
Personally I am an unabashed defensive pessimist, 💯. I found this out in college when a good friend of mine blurted, "Neil, why do you always have to be the g------ voice of doom?" Not necessarily the feedback I was prepared for or looking for, but a worthwhile nudge toward self awareness to be sure. I've spent decades of my life learning how not to lead with negativity, or at least warn people that I'm about lead with negativity, and why that's a sign I'm engaged and interested. I'm still working on it.
I think defensive pessimism gets short shrift. First off, it's a terrible name - the summation of two qualities generally considered negative - "Boy, I really like the sound of that defensive pessimism!" Second, who promotes this way of thinking? Imagine listening to a famous athlete say, "well, the first thing I do is obsess about all the ways in which I can screw up." I'm hard-pressed to think of anyone who publicly advocates for the defensive pessimism strategy.
That used to bug me, but I've learned to harness it as my engineering secret weapon.
Inexperienced engineers tend to build things the way they are taught, by going step-by-step from the starting point to the desired outcome. This is what is known as the "happy path" and it's a recipe for pain. What experienced engineers (especially ones that care greatly about user experience) learn is that handling exceptional circumstances gracefully takes a ton of effort and preparation. If you start by assuming failure and build up to success, you get a system that is robust and impregnable. Slightly longer initial development, but significantly decreased defect rates in testing and production.
Moreover, it's way easier to build fault-tolerance in from the start than to bolt it on as a "feature" at the end of the development cycle. Fault tolerance requires synchronized handoff across every layer of the application stack. It works well when baked in, and leads to spagetti code when not.
And the kicker is that fault-tolerance is almost never listed as a requirement in the spec, but is always demanded as a capability prior to (or in dire cases, after) release. If the release was built using a "happy path" strategy that will inevitably lead to surprise delays at the end of the release cycle.
I applied the defensive pessimism mindset when building Pandora's music streaming engine on the iPhone. We were competing with terrestrial radio in the car. I figured that in order to compete we'd need the music to start playing as quickly as possible and never stop. The mobile environment was different - people were going to have their phones in their pockets while walking down the street, so don't try to show them an error message. To use the trite phrase, it needed to just work. Of course there are going to be times when things go wrong - servers are down, there's a problem with the audio transcoding, and of course losing Internet coverage. I couldn't control that from the app but what I could do is make sure that the music starts playing again once conditions return to normal. And that's what the engine did, and still does to this day closing in on 20 years later.
On the flip side defensive pessimism also keeps me aware of my own limitations. For example, I once spent roughly an hour looking into a stock trading automation platform in consideration of automating trades in my personal portfolio. Difficult questions immediately started flooding my mind. What happens if the servers loose connectivity during a trade? What happens if I'm trying to execute two trades simultaneously and one fails? What happens if there's a delay in processing and prices change significantly? What happens if trading is halted mid-trade? How does the platform handle these situations? What kind of error codes will I receive? Will I be notified if my node is taken down for maintenance? Would my system be brought up on a new node? It was as if I'd pulled off the lid and discovered an endless void inside.
As you can see, my anxiety went through the roof before writing a single line of code. I realized I'd be building something that worked well most of the time but could lose a lot of money rapidly in rare circumstances. I needed a statistics-based mitigation plan and I didn't yet know the necessary math. Applying newly learned math to an anxiety-inducing problem space is the sad path - no thank you. Reading the experiences of others who'd followed this path confirmed my intuition - this was not a good fit for me. I bailed and have never looked back, no regrets.
Defensive pessimism is what made Bill Belichick run practices outdoors in bad weather and has olympic athletes training at high altitude. If you can perform well in the worst of situations, then you'll do even better in prime conditions.
Defensive pessimism is a great and underrated tool in my humble opinion. And to my fellow secret DPs out there I say, it doesn't work every time so consider your options carefully.
I have thoughts on how best to apply DP when adopting AI, but unfortunately that's going to have to wait for another week. Until then!