I preach against this, but got caught myself doing this. Taking a set of facts and developing a conclusion then using confirmation bias to make sure that’s the conclusion that wins.

Here’s the deal. We have a system that runs in the background on a server and pulls up customer records over time to send them reminder emails. Our system also encrypts a lot of the sensitive user info as well.

So things are going all honky-dory until one week when things go amiss.

  • We change up the client credentials (best practice to rotate these things occasionally)
  • We notice some errors from this system, specifically around encryption
  • Somehow the metrics associated with processing change

Ok, obviously we screwed something up around the encryption. So we futz around and make sure everything is up-to-date on that server.

  • Metrics go up again

At this point we’re spiking the ball in the end-zone. Problem solved, lets just backfill the stuff that errored out.

But the errors from the system don’t stop.

Huh?

Between fighting other fires (crazy week) we try to fix the obvious on-again-off-again encryption problem to no avail. No matter what we do the errors don’t stop.

Yesterday I finally had a chance to dig a bit deeper into the problem. Up until then I hadn’t had a chance to come up for air. While I was occupied another coworker that was in deeper on the encryption end had been chasing the problem when he had time.

What I found was interesting. The error that we were getting, wasn’t really an error; it was more of a warning when I looked at the code that was generating it. But since everything fit what we were thinking we just ran with it.

It’s a classic case of “ready, fire, aim.”

And it’s not that I didn’t know it before. This is a good reminder to validate your assumptions. Just because what you are seeing fits with your theory doesn’t mean you’re right. It’s always worth taking a step back to assess the situation a bit more even if you think you know what the problem is.

Now to figure out what the problem actually is.

<sigh/>

Live and learn.

…and then forget… and learn again.