Technical Debt is Soul-crushing

Technical debt is incurred when the software or system designers take shortcuts to ship a feature faster, increasing the overall complexity of the system. The goal is to optimize the present rather than the future. In other words, it’s the easier path that takes us to the end-goal faster, but the resulting code (or design) is messy and complicated. It will require extra time in the future to add new features or to fix bugs.

The most common reason why companies take on technical debt is to meet the time to market demands. “We must release this feature by February, or our revenue will take a big hit. Just hack it for now, and we’ll fix it later.” Other reasons for incurring tech debt include lousy design choices, poor programming, changing requirements, or the presence of outdated libraries or frameworks that made sense in the past but have become a liability now.

Technical debt is not always a bad thing. It can help companies ship a critical feature fast and acquire users more quickly than its competition. My first job was at a startup. We intentionally took on tech debt because a) what we were doing was risky, b) we had a tight deadline to meet or the company would run out of money - no point in writing perfect code if it wasn’t going to be ever released. Our tech debt wasn’t the opposite of over-engineering. It was an intentional compromise to get the product out of the door on time. We understood that we’d have to pay the debt off or it will make it difficult to maintain and grow the system in the future.

Tech Debt is Demoralizing for Software Developers

The problem starts when companies forget to pay off the debt and let it creep and pile up for an extended period. The past comes to haunt the present. For good software developers, it is totally demoralizing to work on products that have high tech debt. This aspect isn’t often talked about, but its effects are very real. Simple things like changing a title tag of a webpage page take up a whole day because the logic was scattered in five different files. At the end of the day, it’s not a great feeling that it took so much time for a small task. It’s even more upsetting when they have to explain it to their managers, colleagues, or the product team why it took so long. Troubleshooting a bug is not just difficult but also painful. Jeff Atwood called it a major disincentive to work on a project:

Beyond what Steve describes here, I’d also argue that accumulated technical debt becomes a major disincentive to work on a project. It’s a collection of small but annoying things that you have to deal with every time you sit down to write code. But it’s exactly these small annoyances, this sand grinding away in the gears of your workday, that eventually causes you to stop enjoying the project.

I’ll add that it is even more frustrating if it is a large project with many other developers working on it. Imagine finally finishing a feature only to find that it is not working on the QA or Stage environment because another developer from a different team changed something which somehow broke the feature you just completed and you both have no idea how to get out of the mess. It can also fester and promote intra-team conflicts and dissatisfaction.

In addition to just being frustrating, good software developers have a burning desire to keep their skills up-to-date. Systems with high tech debt are very difficult to work with, much less keep up to date. As a result, developers are stuck with libraries and frameworks that are many years old. They fear they will break something, so they avoid upgrading it altogether.

Cost of Tech Debt: Employee Turnover

The real cost of all this is turnover. Good developers leave when they believe it is going nowhere. There is no sense of accomplishment, just frustration. There is worrying about the future since they aren’t learning anything new or acquiring new skills. They leave when they find a better opportunity, leaving behind those who are at peace with taking shortcuts or stuck working on the crummy project.

The people who are left behind do not voice their concerns as vocally as those who left and they continue increasing the tech debt. They have learned to survive with the beast and are at peace, taking the quick-and-dirty path all the time. One could also argue that the messy codebase encourages further poor practices and software rot owing to the broken window theory.

I’d argue that for organizations with high tech debt, it is more expensive to replace software developers than those with a lower amount of debt. When someone leave, they end up taking a chunk of tribal knowledge of code and processes with them. New hires take a long time to understand and get up to speed. It takes them a long time before they become productive.

How to fix it?

Good leaders and managers must evaluate tech debt regularly and take proactive steps to keep it under control. The worst thing senior technical leadership can do is deny the existence or effects of tech debt if it is present. The second worst thing to do is to ignore it and take no action if an action is needed. Those who do this are being short-sighted because they will soon be drowning in problems (if they stay.)

Let’s look at some strategies to paying off the debt. This is in no way a comprehensive list and based on my personal experiences.

A common strategy that I have seen work is setting aside a percentage of time each quarter or sprint to gradually pay off tech debt along with new feature development. Anywhere from 15% to 30% is reasonable, depending on how bad it is. This strategy works great if the tech debt isn’t out of control.

If the debt is out of control, then it must be attacked head-on and prioritized. A good strategy is to acknowledge the problem and let the software team know that it will be addressed. For large or medium organizations, it is important that all senior leaders are on board. For example, if the product or the marketing team don’t see the value or believe in it, they’ll continue pushing for new features that will result in a poor dynamic and make it fail. The debt metaphor works really well on non-technical people and helps them understand the problem. After the buy-in across the board, invite senior engineers to come up with a solution.

1. Start by identifying tech debt in your system or code. Resist the urge to define solutions at this stage. Create a shared spreadsheet identifying classes, methods, entire modules or services, use cases, dead feature flags or A/B tests, etc. in your code.
2. Go through the items that are identified above and extract common themes. Prioritize these themes such that most painful ones are taken care of first. Define clear goals. The goal shouldn’t be to pay off all tech debt but to achieve a reasonable ratio. You can choose to leave some of it in there or until the next phase.
3. Discuss steps but avoid the urge to rewrite from scratch. Inexperienced developers love total rewrites. Resist the urge and instead identify incremental improvements.
4. Be open to retiring systems that aren’t useful or needed anymore. When a system is retired, all of its technical debt is retired with it. At my previous (mobile gaming) company, we had a few very old central services (backend) that had a ton of tech debt and troubleshooting them every now and then was very time-consuming. These services were used by older game clients and had become irrelevant for business purposes. In theory, we could shut these down without any impact. But it wasn’t technically possible because doing so would break the API calls and impact the game and the user (forced updates of game clients wasn’t an option), so we came up with a creative way. We retired these services and replaced them with a new ‘mock’ service that always returned a mock (successful) response.
5. Assign action items to engineers and teams. Make sure they are empowered and that there is a clear execution plan.
6. Changing existing habits. This might be a little harder, especially with engineers who have been with the company for a long time and are used to just hacking it in even when it is not required. Coaching helps. Re-iterate the debt minimization vision and promote best practices like writing unit tests etc. for all new features.
7. Devise a strategy to track new tech debt. An easy and common technique is placing TODO comments with a description e.g. TODO - move this logic to XYZ or TODO: setup and use internationalized string instead of hardcoded
8. Make it visible and a priority. Follow up regularly in check-ins and re-adjust as required.

This process is an art rather than science. The most important thing is to identify a path that allows tech debt to be paid off gradually, and also allows new feature development or bug fixes to happen in parallel (albeit with some compromises.) Even the most complex systems can be gradually retired. I’ve found reverse-proxy’ing to be an effective technique in which some requests or parts of requests are redirected to newer services and older services are retired slowly.

If you’re a leader, you must assess the tech debt situation in your organization. Early detection and prevention will go a long way. Don’t ignore it if it has got too big and your developers are complaining and stressed out. It the situation isn’t too bad, schedule time to pay it off periodically. If it is out of control, make it an immediate priority. You might get away with it in the short term, but it will come back to bite you and feature development will slow down to snail’s pace.

I ran a survey to understand the impact of tech debt on employee retention. Please visit this link to see the results.