The 4 "REs" of Systems Debt

I like to joke that the perfect code is the code you've not yet written. It's optimized, bug-free, fully capable, and carries zero debt. Because, the unfortunate truth is, as soon as we're writing code, we're likely accruing some debt - and that's ok provided we're consistently paying it down. 

A common suggestion is to allocate 20% of the teams' time towards paying down technical debt. As a reminder, Technical Debt is when teams take shortcuts to deliver a project sooner, knowing they will later refactor. Not paying it down leads to a product with a brittle code base that is tough to maintain, scale and support - and teams are often bad about actively paying it down, which leads to Engineers having a strong preference for green field work over a debt-ridden brown field.

So - what if we allocated 20% of our time towards paying down debt? In a typical 5-day work week, this could mean dedicating Fridays to paying down debt you accrued over the course of the week. With code, this is straightforward and a great practice - especially when you have a good sense of the team's story-point capacity or lead & cycle times: The code, and all its shortcuts, I wrote and delivered on a Tuesday is refactored on Friday. Refactoring might suggest a large undertaking but if we're diligently paying down debt, it won't be - and even if it were, we've still delivered the working solution earlier in the week. We'll just need to be mindful of the debt we carry week over week. 

Now - let's consider the same approach for Systems Debt. If you're new to my theory on Systems Debt, it's building off of Systems Theory and categorizes anything that slowly builds up within a system that impedes flow as debt. Just like the perfect code is the code we've yet to write, the perfect team, the perfect process, the perfect business is the one that does not yet exist. To exist is to accrue debt.

A single day is rife with inefficiencies: from the extra steps it takes to get someone's attention, to the inefficient steps that arise from a suboptimal policy, to time lost looking for the resource you need. That's just the obvious stuff - but now consider the more implicit and nefarious stuff: The wrong assumptions, the things we only partially understand, the tribal knowledge, the things we've learned about our customers, how our vision has evolved (but may not have fully disseminated across the team).

Imagine aggregating all the things that slowed us down from Monday to Thursday and systematically and surgically addressing them on a Friday. It feels impossible of course - but not if we're persistent and intentional.

That's where the 4 REs come in: 

  • Refactoring for scale
  • Realigning to an evolving vision
  • Restructuring to optimize flow
  • Refreshing basic principles

Refactoring for Scale

Refactoring goes beyond code, though the same idea holds: improve operation without altering functionality.

Scaling is a continuous challenge - whether it's scaling software, teams, customer base, whatever. We see this in practice every time we turn back to old habits:
  • turning to the Senior person to resolve what a Junior person should be empowered to solve
  • hacking a methodology to support our bad habits
  • avoiding change because it's uncomfortable
I'm not against reusing old solutions - but I think an old solution should always come with reevaluating the problem's framing.

If, on a Monday, I'm asking a Sr. Engineer to field an ad hoc request because "it's just faster when you do it...", come Friday I should invest in a faster alternative.

Realigning to an Evolving Vision

Vision is a big word - but it applies to many different levels: Team visions, Product visions, Process visions. OKRs have a vision, NPS measure an implicit vision. Even our meetings have a Vision! All of our activities are grounded in producing an outcome or impact.

But sometimes - oftentimes - we learn the outcome we delivered is not the best outcome. And of course, that's OK. There are learnings to be had! 

As the vision evolves, are we evolving with it?

Imagine taking Fridays to reevaluate the past week of meetings, our focus, our backlog, our OKRs. More importantly - imagine taking the time to make sure everyone is aligned to the incremental adjustments. Weekly adjustments may be too rapid and frequent for a larger scope - but trickling that down to our day-to-day operations and evaluating how we might adjust is still a valuable exercise.

Restructuring to Optimize Flow

As someone who has built their career writing code, the most impact I've ever had is when I've avoided writing it. Code is only one possible solution to a problem: in the automation space, there's a common refrain that automating a bad process doesn't make it better, it just makes a bad process faster. Coincidentally, the great John Carmack recently shared the same sentiment: "Software is a just a tool to help accomplish something for people - many programmers never understood that. Keep your eyes on the delivered value, and don't over focus on the specifics of the tools".


The same applies to our business workflows (formal or informal).

It's natural to challenge things that don't immediately make sense - maybe we don't have a full picture, maybe the step is unnecessarily redundant, maybe it's no longer relevant, maybe it's good in theory but horrible in practice.

As an example: PR reviews are an important practice - but bad habits and a busy team can make a PR review pointless. The review isn't pointless though - and simply adopting new practices like adding an Estimated Review Time can really improve the quality of the review.

Our Inboxes & DMs can also be a cluttered mess - and separating signal from noise is a challenge. While there are plenty of investments happening to leverage AI to help us optimize, we can also optimize how we communicate with one another. Creating guidelines for prioritizing communication streams (when to DM, when to Email, how to structure messages to optimize around the action needed) can be valuable. 

Imagine taking a Friday to look at all the ways where we were unnecessarily slow - and then fixing them.

Refreshing Basic Principles

I still remember my high-school teacher teaching us about the pitfalls of "common sense." The subjectivity of what is "common" can lead to miscommunications, problems, and waste. 

Similarly believing our basic principles and understandings are shared is dangerous. I learned a lot from a mentor/manager after he'd conducted a Stupid Questions Only meeting. Not only did I realize how incorrect we were in assuming common knowledge to be common, but we'd also uncovered knowledge gaps based on who was an onboarding buddy, how much our reality had evolved and how little that evolution was known. It was single-handedly the most impactful meetings I'd ever been a part of, because it reestablished our foundation. 

Moreover, there are many implied contracts in our interactions - between individuals, between teams, across hierarchies, across time-zones, across cultures. Taking the time to collectively agree to how we work is yet another way we can create more efficiency.

The Hidden Bonus 5th "RE"

Retrospectives are the most important Scrum ceremony - but I'm a big believer in not waiting for the retro to solve a problem (it's probably why I prefer Kanban).

There are a couple of challenges with retros coming after a Sprint: Even with a 2 weeklong Sprint, we can still fall victim to Recency bias. But, more problematically, I've seen retros that get overloaded with problems that need resolution, or retros that don't have any problems. Usually, the latter follows the former and is a sign that the team is getting burnt out, and no longer has faith that the problems they raise will actually be resolved.

The best retro I'd ever run, as a manager, was when I challenged my team to challenge me with their issues. I wanted to hear all the problems they didn't believe could ever be changed - the ones that they'd just accepted as "that's just how it is...".

That exercise was critical to not only improving our work lives, but also in improving our overall relationship and happiness.

The preceding 4 REs are a recipe for running a mini retro at any scale: individually, an immediate project team, a hierarchical team, a business unit, or more. They also frame some important questions we can cover in more formal retros:
  • How do we need to scale?
  • How has our vision evolved?
  • How far is the outcome we produced from the outcome we need?
  • What steps are slowing us down?
  • What activities could speed us up?
  • What is a bad assumption we may be have?
  • What are we too afraid to ask?

Not All Debt is Created Equal

Paying down 20% of our debt is a big task. Every Friday is an even bigger ask - and, honestly, I think any fixed process is dangerous. Everything should be adapted to the reality.

Part of that reality is that we can find Systems Debt everywhere. We'll never be able to eliminate all of our debt because we're constantly acquiring more. Prioritizing what debt, we pay and when is important.

When it comes to the debt we prioritize, I think taking a page from any playbook about managing personal finances is a strong framework:
  • Prioritize debts with the highest interest rates (i.e., the ones that cause the most pain.)
  • Pay off one debt in its entirety before focusing on another.
  • Even replacing the debt with different debt with a lower rate of interest helps.

Wrapping It All Up

Technical Debt, Process Debt, Architectural Debt, Systems Debt - none of it is bad. Shortcuts help us get to a destination faster. It also provides us with a safety net - prioritizing value over perceived perfection.

The highly iterative, highly incremental world of continuous improvement requires us to continuously evaluate where we are carrying the most debt. Just like too much Technical Debt will lead to an unmaintainable codebase, too much Systems Debt leads to an unmaintainable system.

Popular

Let's Clear Up The Ambiguity!

FAQs for a Software Engineering Hiring Manager

7 Steps to Writing an Amazing Resume

7 Steps to Building your Portfolio MVP

Work Experience vs Professional Experience