Technical Debt is Soul-crushing

Umer Mansoor Umer Mansoor Follow Jan 25, 2020 · 9 mins read
Technical Debt is Soul-crushing

Technical debt is incurred when the software or system designers take short-cuts to ship a feature faster, increasing the overall complexity of the system. The goal is to optimize for the present rather than the future. In other words, it’s the easier path that takes us to the end-goal faster but the resulting code (or design) is messy and complicated. It will require extra time in the future to add new features or to fix bugs.

The most common reason why companies take on technical debt is to meet the time to market demands. “We must release this feature by February, or our revenue will take a big hit. Just hack it for now and we’ll fix it later.” Other reasons for incuring tech debt includes lousy design choices, poor programming, changing requirements or the presence of outdated libraries or frameworks that made sense in the past but have become a liability now.

Technical debt is not always a bad thing. It can help companies ship a critical feature fast and acquire users more quickly than its competition. My first job was at a startup. We intentionally took on tech debt because: a) what we were doing was risky, b) we had a tight deadline to meet or the company would run out of money. No point in writing perfect code if it wasn’t going to be ever released. Our tech debt wasn’t the opposite of over-engineering. It was an intentional compromise to get the product out of the door on time. We understood that we’d have to pay the debt off or it will make it difficult to maintain and grow the system in the future.

Tech Debt is Demoralizing for Software Developers

The problem starts when companies forget to pay off the debt and let it creep and pile up for an extended period of time. The past comes to haunt the present. For good software developers, it is totally demoralizing to work on products that have high tech debt. This aspect isn’t often talked about, but its effects are very real. Simple things like changing a title tag of a webpage page take up a whole day because the logic was scattered in five different files. At the end of the day, it’s not a great feeling that it took so much time for a small task. It’s even more upsetting when they have to explain it to their managers, colleagues, or the product team why it took so long. Troubleshooting a bug is not just difficult but also painful. Jeff Atwood called it a major disincentive to work on a project:

Beyond what Steve describes here, I’d also argue that accumulated technical debt becomes a major disincentive to work on a project. It’s a collection of small but annoying things that you have to deal with every time you sit down to write code. But it’s exactly these small annoyances, this sand grinding away in the gears of your workday, that eventually causes you to stop enjoying the project.

tech_debt_human

I’ll add that it is even more frustrating if it is a large project with many other developers working on it. Imagine finally finishing a feature only to find that it is not working on the QA or Stage environment because another developer from a different team changed something which somehow broke the feature you just completed and you both have no idea how to get out of the mess. It can also fester and promote intra-team conflicts and dissatisfaction.

In addition to just being frustrating, good software developers have a burning desire to keep their skills up-to-date. Systems with high tech debt are very difficult to work with, much less keep up to date. As a result, developers are stuck with libraries and frameworks that are many years old. They fear they will break something, so they avoid upgrading it altogether.

Cost of Tech Debt: Employee Turnover

The real cost of all this is turn-over. Good developers leave when they believe it is going nowhere. There is no sense of accomplishment, just frustration. There is worrying about the future since they aren’t learning anything new or acquiring new skills. They leave when they find a better opportunity, leaving behind those who are at peace with taking shortcuts or stuck working on the crummy project.

The people who are left behind do not voice their concerns as vocally as those who left and they continue increasing the tech debt. They have learned to survive with the beast and are at peace taking the quick-and-dirty path all the time. One could also argue that the messy codebase encourages further poor practices and software rot owing to the broken window theory.

I’d argue that for organizations with high tech debt, it is more expensive to replace software developers than those with a lower amount of debt. When someone leave, they end up taking a chunk of tribal knowlede of code and processes with them. New hires take a long time to understand and get up to speed. It takes them a long time before they become productive.

How to fix it?

Good leaders and managers must evaluate tech debt regularly and take proactive steps to keep it under control. The worst thing senior technical leadership can do is deny the existence or effects of tech debt if it is present. The second worst thing to do is to ignore it take no action if an action is needed. Those who do this are being short-sighted because they will soon be drowning in problems (if they stay.)

Let’s look at some strategies to paying off the debt. This is in no way a comprehensive list and based on my personal experiences.

A common strategy that I have seen work is setting aside a percentage of time each quarter or sprint to gradually pay off tech debt along with new feature development. Anywhere from 15% to 30% is reasonable depending on how bad it is. This strategy works great if the tech debt isn’t out of control.

If the debt is out of control, then it must be attacked head-on and prioritized. A good strategy is to acknowledge the problem and let the software team know that it will be addressed. For large or medium organizations, it is important that all senior leaders are on board. For example, if the product or the marketing team don’t see the value or believe in it, they’ll continue pushing for new features that will result in a poor dynamic and make it fail. The debt metaphor works really well on non-technical people and helps them understand the problem. After the buy-in across the board, invite senior engineers to come up with a solution.

  1. Start by identifying tech debt in your system or code. Resist the urge to define solutions at this stage. Create a shared spreadsheet identifying classes, methods, entire modules or services, use cases, dead feature flags or A/B tests, etc. in your code.
  2. Go through the items that are identified above and extract common themes. Prioritize these themes such that most painful ones are taken care of first. Define clear goals. The goal shouldn’t be to pay off all tech debt but to achieve a reasonable ratio. You can choose to leave some of it in there or until the next phase.
  3. Discuss steps but avoid the urge to rewrite from scratch. Developers, especially those who don’t have a lot of experience love total rewrites but the goal is to identify incremental improvements. (Incremental rewrites are fine if they make sense: e.g. taking a theme and incrementally moving it to a new service or microservice if your company is already using SOA or microservices.) Be open to retiring systems that aren’t useful or needed anymore. When a system is retired, all of its technical debt is retired with it. At my previous (mobile gaming) company, we had a few very old central services (backend) that had a ton of tech debt and troubleshooting them every now and then was very time consuming. These services were used by older game clients and had become irrelevant for business purposes. In theory, we could shut these down without any impact. But it wasn’t technically possible because doing so would break the API calls and impact the game and the user (forced updates of game clients wasn’t an option) So we came up with a creative way. We retired these services and replaced them with a new ‘mock’ service that always returned a mock (successful) response. This took care of a massive pain point the team had to deal with.
  4. Assign action items to engineers and teams. Make sure they are empowered and that there is a clear execution plan.
  5. Changing existing habits. This might be a little harder especially with engineers who have been with the company for a long time and are used to just hacking it in even when it is not required. Coaching helps. Re-iterate the debt minimization vision and promote best practices like writing unit tests etc. for all new features.
  6. Devise a strategy to track new tech debt. An easy and common technique is placing TODO comments with a description e.g. TODO - move this logic to XYZ or TODO: setup and use internationalized string instead of hardcoded
  7. Make it visible and a priority. Follow up regularly in check-ins and re-adjust as required.

This process is an art rather than science. I was part of a large project any my team successfully paid it off. It was possible in large part because we had good, strong technical leadership and their full support. There was a solid execution strategy and calculated risk taking was encouraged.

If you’re a leader, you must assess the tech debt situation in your organization. Early detection and prevention will go a long way. Don’t ignore it if it has got too big and your developers are complaining and stressed out. It the situation isn’t too bad, schedule time to pay it off periodically. If it is out of control, make it an immediate priority. You might get away with it in the short term, but it will come back to bite you and feature development will slow down to snail’s pace.

I ran a survey to understand the impact of tech debt on employee retention. Please visit this link to see the results.



Umer Mansoor
Written by Umer Mansoor Follow
Hi, I am Umer. I live in the San Francisco Bay Area and it is where I work. Since 2011, I have worked at startups, mobile gaming and internet companies. If you have any questions, feel free to message me.

Comments (3)


Johann Gotthilf Unlieb

I have been there. I recently quit a job, not because of the immense technical debt in the decades old naturally grown massive monolith of an awful project (it could have been solved incrementally even though it was made of five or six lengthy god objects that had rancid spaghetti with sql&orm sauce oozing out of every opening, of which there were many), but because middle management simply chose to ignore everything about it and simply leave the team alone without support or resources. Everybody despised working on it. It as a mystery bag full of nasty surprises. Everyone knew about this. When addressing the issue and possible solutions, nobody was surprised about the state of the software, but reactions were resigned rather than hopeful. Soon after I started there I found out why.

To me, it seems more often to be a structural issue in the hierarchy of an organization, at least from what I've seen or heard from people in similar positions I've talked to about this. (As Conways's Law states humorously: Any piece of software reflects the organizational structure that produced it.) Does assigning resources to fixing technical debt account for admitting to having made a mistake as a manager? It seems to me the cost of technical debt is maybe to well hidden, to unclear or to accepted as normal to do anything about it. It is difficult to give it a price tag, and even more, to put a price on the promise of increased efficiency in future development. Maybe it boils down to a simple issue of trust in some cases, or the willingness to make a decision of vague payoff for ones own career. It seems hard to understand. Isn't "doing nothing beyond the bare minimum maintenance" the same risk as "total rewrite from the ground up in current hyped technology"?


Matt S.

It's bewildering how many many managers just let it build and not do anything about it. They treat it like payday loans and wonder why they are drowning a year later.


Philippe Bourgau

I love your rewrite mock technique :-)

I've seen another problem with rewrites. After developers have tried a few failed rewrite initiatives, product people lose trust and don't want to hear about paying down tech debt and refactoring anymore. In this case, you becomes almost impossible to get them onboard. That's one reason why incremental refactoring techniques are so important. With them, you can make the refactoring steps so small that you don't have to ask for permission! I've write a series of post on the topic

Thanks a lot!


Speak Your Mind

Join Newsletter
Enter your email address to receive a very occasional newsletter. I'll never spam or sell your email.