Engineering management · Web Development

How do you handle technical debt?

October 23rd, 2015

Like many (all) startups we have technical debt and it's always a tension between fixing it and working on new or existing features. Obviously, if it is perfecting performance we prioritize it but curious to hear how other companies think through it and prioritize fixing it?

Benjamin Olding Co-founder, Board Member at Jana

October 24th, 2015

Zero technical debt is a bad sign... It likely means you're not moving fast enough. Here are some clear signs you have too much though:

1) Debugging often takes as much time or more than writing a new feature.

2) A developer is as likely to introduce a new bug when fixing an existing one as not.

3) No one can explain "random" moments of high latency on the system (hint: they aren't random).

4) New hires with multiple years of experience seem to have still not fully ramped up on the system after 6 weeks.

5) Your QA procedure has 8+ different ways to classify bug priority/status.

How do you fix these problems? Unfortunately this usually isn't a technical problem but an organizational one. I say "unfortunately" not because it's bad to have organizational problems (it's a startup!), but just because you have to work a little harder to solve them than just writing new code.

If zero technical debt is a sign you're not moving fast enough, too much technical debt is a sign you've been moving faster than your current team can handle. To pay down existing debt, you need to start reserving time to refactor/add test frameworks/change technology (ugh, that one sucks - sometimes it has to happen though). Don't make things worse by rushing through this process: you need to specify ahead of time how you're going to test the new parts to make sure they are functionally equivalent to what you're replacing.

Doing this means you won't be adding new features at the rate you had been - this requires buy-in from more than just your technical team. Don't ever unilaterally do this - you'll just shift technical debt into further organizational problems. Communication matters: it's not enough to do the right thing. You need to do the right thing the right way and work as a team with the non-technical part of the organization. This may feel like a purely "technical" problem that is a waste of time to discuss in any kind of real detail with non-technical people, but it's not: it took more than the technical team to get you to this point, and it will take more than just them to get you out.

To keep from accumulating debt at the rate you had been, you need to commit to more emphasis on planning, testing and refactoring than you have been doing. This isn't necessarily a bad thing: it's a sign of growth. You may need to grow the team to do this and keep up with the company's business requirements. Again, don't grow it without the whole organization buying into the idea that you'll be growing the team without (initially at least) growing the feature rate. There's a good chance they're sick of the bugs and performance lags too, so if you actually have too much technical debt, buy-in shouldn't be that hard.

If you're early stage & it's not clear you really have clients yet that are buying a "product" instead of a "service" from you, some good advice may just be to lighten up a little bit. It's really depressing to delete excellent code. It feels liberating to delete a pile of garbage... When you're early and the business requirements are still murky, run fast & keep a positive attitude. Reserve being a technical curmudgeon for when you're actually making money in a repeatable way.

Tom Maiaroto Full Stack Consultant

October 23rd, 2015

I think too many people underestimate the impact of technical debt. I hear some people say, "That's not why startups fail." I agree, it's not the "sole reason" -- but it is almost always a factor. Whether you realize it or not.

For example: You might say, "Well there was no product/market fit." Ok...But if your technical debt was smaller, you might have been able to find that fit or pivot easier.

It has a profound affect on a company and can also make the difference of having a healthy profit margin. Technical debt quite often leads to increased overhead and if that plus the CTA is not lower than the LTV of customers...You're toast. Your business can't scale without fixing something.

So at that point, yes. You must fix technical debt because it may not be an option to fix something else. However, if you can lower your CTA, you may wish to focus on that before the technical debt because it may be easier and cheaper to address.

Fixing technical debt is expensive. Perhaps one of the most expensive work efforts out there.

Here's an approach I've seen have quite a bit of success: Get your core team (which is going to be the more expensive resources) to define the problems first. Get a good handle on what needs to change, why, and put together a plan. Then utilize the cheaper resources you may already have or outsource new resources to execute on that plan and remove some of the debt.

This allows you to bucket the work and do it in phases. Which helps fit any budget. Especially if contracting out the work...Because otherwise you're paying salaried employees to do it and you can't just stop when you want. Pssst, this helps prevent layoffs.

That's how I'd repay technical debt on a budget to be frank. I'd simply outsource it. I do believe it's a problem that can be outsourced too...But only after properly identifying it and coming up with a game plan. You need absolutely clear instructions when outsourcing.

For example: "Refactor this code to do x, y, and z. Removing a, b, and c which are no longer needed. Then update some test cases and write new ones for the new stuff." It could be as simple as that.

This way, your core team is able to focus on new features and maintaining the current product and UX.

Hiring new resources requires a ramp up period. They need to learn the product, the roadmap, etc. So the best resources to further the product is those who are already intimately familiar with it.

This means the cost of adding a new feature is (usually, but not always) lower than it would be if you had existing resources clean up technical debt and hired new resources to build new features.

Using new resources to build new features is most likely only going to increase your technical debt.

New resources need to learn the product and one of the best ways to do that is by cleaning up that technical debt. This is because it can be done in pieces and it gives someone a good overview of the product and window into the past. It clearly shows developers what was tried and what didn't work and what now needs to happen.

THEN if you want those new resources to work on the future of the product, you could do so without accruing as much technical debt.

Though one thing to keep in mind is that you always accrue technical debt. There is no such thing as a technical debt free product. I think the most important thing of all is to understand when you need to repay it. Even if you aren't efficient in repaying it.

Kevin Tureski VP Products at Fabric Software Inc

October 23rd, 2015

I provided a fair bit of feedback on the Technical Debt chapter of Kenny Rubin’s “Essential Scrum” and am currently with a startup. My views are pretty much unchanged: Flag it and set aside time to address it or else. If you need a prototype to demonstrate your unique value, accrue as much technical debt as is required BUT realize that you will need later need to refactor large parts of it before putting that code into production or a paying customer’s hands AND reserve that time as part of the next phase / milestone etc. Fail to do that, and you’ll soon be on your way to your next startup. If you are building a framework that will become the basis of your product, then near-zero tolerance is a good place to start. I say near-zero since there are exceptions such as adding this feature will land / keep this client allowing us to meet payroll and refactor soon after. Kevin

Sean Ganser CTO / Co-founder of InMoment Software

October 23rd, 2015

Tricky territory.

Firstly, the payoff long-term is worth taking care of the technical debt asap. It can be arduous, and your devs might hate you, but you need to do it.

Also, moving forward, a well-engineered codebase that is testable and maintainable will help keep technical debt at bay for future endeavors.

We've been in many situations where we had to just 'leave things be' or code freeze because we've got paying customers, and we can't rock the boat. It's painful, but it's part of the process.

Technology is a journey, there is no perfect, and everything takes time to develop and mature. Your systems will never be ideal or perfect (this goes for all companies, big and small).

Top priorities should be:

Solidify current systems for paying customers.

Consider a refactor.

Test different systems before implementation.

These are expensive and time-consuming, but it's necessary. Sometimes as founders/C-levels we want to move things forward and can't justify our devs testing new systems that won't have a payoff, but it's part of the process.

Be prepared to spend 1-3 months having your team experiment with new things and processes. It's natural.

Unfortunately, in my experience, it's a matter of letting go...

John Tobin CTO Limelight Mobile Inc. Owner

October 25th, 2015

Technical debt is a fact of life, so you need to address it in your application life cycle.

I would suggest before transitioning to development of next version of your app, to set a target, say 50% of all bugs, to be eliminated before proceeding. This way you start the new app with a lower technical debt overhead and more importantly becomes part of the DNA of the development process and scheduled.

I sometimes ask all devs to target the least complicated bugs first, this is a numbers game. Reaching the target is your objective. Starting with the easiest first ensures you do not get you or your team overwhelmed. Sure you may have some "must fix" high priority items, but use simplicity as your friend.

You can also include, group bug hunts, bounties, running totals, finding anecdotes, group outing, show and tell to make more that the team is not weighed down in the drudgery of fixing bugs and actually looks forward to this phase. There are lessons to be learnt from fixing bugs, optimizations to be found, make it as encouraging as possible.

From a CTO/Dev Manager perspective you have the authority to sit down with devs and help them fix their bugs, provide a different viewpoint, suggest strategies to resolve and encourage group assistance.

From the team's view, you have turned a least wanted task, into something exciting and rewarding.

Rob Mitchell Senior Java Software Engineer at Direct Commerce

October 23rd, 2015

There's a slew of both business and technical variables when talking the dreaded "tech debt" challenge. The fact that you're talking about it is a good indication that everyone's on-board with its realizations.

My $0.02 as a long-time engineer and now software manager:

* keep the talking, diagramming, and/or some level of measurement going

* tech debt is not "all" bad

* based on your company's age and revenue's, the younger/smaller you are, the more you should focus on customer and revenue generation; otherwise always to to whittle-away at some tech debt in the context of feature development

* if you can, devote 2-solid days for the entire business and/or tech teams to work collaboratively and solve tech debt (or, conversely, add to the tech debt if its too low and you can get a huge ROI but adding to it)

Got specifics? Feel free to share.

Thx

-Rob

Jason DeMorrow Senior Software Engineer at iSpot.tv, Inc.

October 25th, 2015

If you've got good senior devs, "managing" technical debt should be a bottom-up process. An experienced engineer, if encouraged to do so, will gladly let you know when some portion of the codebase has gotten out of hand and is slowing progress on new features. The best managers I've known simply encourage their devs to speak up about these things and are always open to devoting sprint time to them.

Zach

October 24th, 2015

There's a lot of good stuff here already but I would phrase it this way:

1. Analyze the debt in terms of what is causing the most headache and rank it. Put a time/money value to each headache in a spreadsheet and sort by cost.

2. Fix them in descending order until your time/money headaches are within a desired threshold. As a few above have said, some debt is ok.

Zach

Ivo Jansch CEO / Co-founder at Egeniq and Buildozer

October 28th, 2015

It amazes me nobody has used the term interest yet.

If you have debt, you first pay interest. This is the money you pay to keep the debt at an equal level. It's the extra time it takes developing anything because this debt is in place. To reduce the weight of this interest you have to start working on repaying the principal. This is, on top of your interest, the time you invest in reducing the debt.

If you pay nothing at all then the accumulated interest will bankrupt you. This is no different from any other debt.

So: if you don't have the means to clear your debt right away, check what the maximum is you can spend per month to reduce the principle. Live with the extra interest until you clear the debt.

Be wary of 'let's just start over'. Usually this just means refinancing and accepting new technical debt at different interest rates, because corners might be cut to get to the current feature level.

Managers often don't allow developers to work on reducing the principal debt, while complaining about why everything takes so long. Explaining the essence of technical debt to them in financial terms will help them understand how just paying interest and not repaying the principal will not improve their situation.

Mike Web Developer

October 23rd, 2015

Can someone define technical debt? I think that it often relies on the technical lead to identify the code that is determined to be technical debt, which can also yield varying opinions.

I'd say first and foremost, if you want to start with paying down technical debt, and you have no test suite, start writing writing tests.

Question stats