Project management · Agile

How has technical debt affected your company?

Joseph Foster Fostercode Senior Software Engineer

January 8th, 2015

Source: https://www.linkedin.com/pulse/technical-debt-can-kill-your-startup-joseph-foster

Has anyone else experienced gross accumulation of technical debt? I'm curious how wide-spread this problem really is.

It's helped my career a lot, cleaning those items up, but I've also seen a good number of start-ups fail or large companies hemorrhage millions of dollars due to adding up technical debt without regard....

...what are some thoughts on experiences with it?
A great idea is 1% of the work. Execution is the other 99%. In this course, we’ll teach you how to conduct market analysis, create an MVP and pivot (if needed), launch your business, survey customers, iterate your product/service based on feedback, and gain traction quickly.

Anonymous

January 8th, 2015

Almost by definition you will accumulate tech debt when moving forward with a startup, as you adapt your product to market demand.

Focus first on product market fit, worry about tech debt later.

Michael Barnathan

January 8th, 2015

I've seen companies (startups and midsize) fail, or more often, have to rewrite their product and miss opportunities as a result, due to tech debt, often of the sort that wouldn't have taken much longer to do the right way in the first place.

This is in part due to the popularity of feature-driven methods such as the Agile family, which emphasize continually delivering user-visible changes - refactoring and other "grungy" work systematically falls by the wayside unless you're very disciplined. I've seen many nontechnical founders who don't realize that their user-visible features aren't truly "done" yet, or worse, drift from one MVP to another without producing any sort of experience that can truly hit it out of the park for users. (The opposite extreme of trying to absolutely perfect everything before moving on is harmful as well, and will result in software that's always "three weeks away from done" but which never makes it out the door).

I've seen some companies grow to midsize with tech debt - if such a company makes it to that stage, it ends up placing a significant burden not only on tech people, but on support! The harder it is to deliver a great product and fix bugs, the more you'll need to focus on keeping your users happy when they inevitably encounter problems. Better by far to have your product delight users in the first place.

The tech companies I've seen founded by engineers tend to do better in this respect (they do worse in some other respects, so pick your poison).

Tim Cullen Principal Software Engineer

January 8th, 2015

This definition of "Technical Debt" reminds of what I've seen several startups do. Both ones I've worked for and ones I've known other people have worked for. They start out with a prototype, demo it around, get some funding, then try and go forward and make a product out of it. More often than not these prototypes were quick hacks thrown together just to get something to demo. What they should have done was take what they learned from what worked and what didn't with their prototypes and in their demo sessions and started over applying what they learned to a new code base. The usual excuse is "That would take to much time and we need to get something to deliver now!". Then they spend much more time and effort working around problems that arise that could have been fixed correctly from the start.

Michael Barnathan

January 8th, 2015

Accumulation of significant tech debt isn't an inevitable aspect of building complex software. It's a product decision, continually made - do we spend time managing the increased complexity of a new feature, or build features at a more rapid rate and let it grow? Sometimes either is the correct choice.

Think of it as your product's entropy. You can decrease the entropy of an open system, but it takes an input of energy from outside of the system to do it.

Rob G

January 9th, 2015

Joseph, This is not the place to market your consulting firm...

Steven Schkolne Computer Scientist on a Mission

January 8th, 2015

That article is dumb, it says tech debt is about putting off building technology. When in reality tech debt is the complexity that accumulates as you grow software.

Good engineers refactor as they grow a product (and sometimes decide to leave krufty code in, carefully weighing the tradeoffs based on their experience). Tech debt accompanies all software products, it is not just prevalent but omnipresent. Just one of those things any good software engineer manages.

And yeah - I dunno what this has to do with government. Whoever wrote that should probably read this: https://hbr.org/1996/01/a-country-is-not-a-company


Robert Gezelter Principal, Robert Gezelter Software Consultant

January 9th, 2015

In many ways, I agree with Tim and Michael.

I have seen many projects, in startups and in established firms, where a "quick and dirty" prototype is done and then an attempt is made to "salvage the existing investment" no matter how crufty the proof of concept is. As Tim has noted, this almost always leads to large amounts of time renovating the existing code, rather than designing a solid implementation from scratch.

In my opinion, Michael's process based perspective also has much to say for it.  Sometimes, one does need to hack something temporarily so that progress can continue. The challenge is not losing sight of the fact that the workaround was temporary, and needs to be fixed. This is not only a question of programming, many organizations commit this sin in ways large and small (I remember my first job as a systems programmer, the Computer Center was housed in a "temp" building, for over a decade).

The real danger of technical debt is a side effect of the synergy of both of these causes: one never knows when the debt will be called. A problem which needs correction involves one or more items of technical debt (incompletely thought out features/code). The problem must be corrected quickly, but it also requires that all of the outstanding issues be completely understood. To continue the debt metaphor, it is a margin call or repayment demand. Haste is the enemy of care, and the resulting consequences prove a burden to the organization.

It may be desirable to work quickly, but working too quickly leads to hard to undo defects.

Steven Schkolne Computer Scientist on a Mission

January 20th, 2015

@JohnShiple .... you're telling me the CTO wrote the code without a framework. Then couldn't fix it.... in 2 days!? Yes there are tradeoffs to writing from scratch but typically it is *easier* to find bugs. All the code is under your control, after all. These "omg" bugs in my experience result from using piles of framework code (where crucial elements/side effects can be obfuscated). Documentation is not needed if the person who wrote the code is sitting right there (and has structured their code well).

There's no accounting for taste and, apparently, no accounting for one's preferred approach to architecting systems either. However to me this seems like something else going on. Either the CTO is inept, or there is some other factor we're not hearing about.

Dan Rubenfield Chief Technical Officer at VREAL

January 20th, 2015

This is the age old debate. Engineering as a known construct, as a fixed solution to a desired product, or as a dynamic and fluid element that's as fungible as the creative aspect. 

I've seen many many more companies and products fail due to over engineering than technical debt. 

Tech Debt has turned into a symbol of a dysfunctional team/company nowadays. It's described in terms of "You made me do this, and now we have to clean it up". And that's not healthy.  It further reinforces the manufactured divide between engineers and "product". 

To say that it's the difference between Agile and something more fixed, such as Waterfall, is also disingenuous. Typically, we know where we generally want to go in terms of deliverable. But we don't know the exact destination. 

A good team can work together to get to the destination even if they need to discover the path. They can work together day to day to make the correct choices technologically to get there as quickly, safely, and cleanly as possible. 

Complexity happens. Refactoring happens. And I'd argue that a system that's created, used and refactored, is an order of magnitude better than a system that's planned up front and never changed.

Empirical usage data is far better than paper planning work. 

If technical debt brings down your company or product, then you hired the wrong team. That's not a failure of planning or engineering, it's institutional. 


****CAVEAT: This assumes you are not working on life or death software or hardware products. 

Steven Schkolne Computer Scientist on a Mission

January 21st, 2015

@JohnShiple I do agree that frameworks are great. Writing a game from scratch when Unity is available? Coding raw JS animations with jQuery? These things are hard to imagine these days. It seems we've both seen the bad use of frameworks. I think what we're both saying, and I see throughout this thread many saying that it depends on the quality of your tech team.

In business, it is hard to quantify technical debt and in many cases hard to know why to pay top $ for engineering talent. Let's say your building an MVP. For $100k you can hire the A-team, who will complete it in 2 months with 0.9 robustness. For $50k, the B-team will complete it in 3 months at 0.9 robustness. So who do you hire?

Seems obvious to hire the B-team right? I'll wait a month to save $50k. For you, the result is identical - equivalent features, with equivalent # of bugs.

But what you don't see is what we could call the mutability index of your MVP. The A-team has delivered something that is far cleaner, better structured, with all the right engineering constructs in place to robustly modify the behavior. While the B-team's solution is patched together, or written in the wrong technology, or something like that. So, when you need to add the next set of features, the A-team suddenly gets relatively less expensive. At some point, the A-team is actually cheaper, feature-by-feature, than the B-team.

This is what I think matters - the ability for you to quickly grow, or repurpose the tech. Seems necessary if you are going to grow and compete in the fast digital world. This lack of mutability is my understanding of the term "technical debt". However, it could also be looked at as a lower asset value and - as a creative - that's how I more tend to think of it;).

This debt can be accumulated for a number of reasons - many mentioned above. It would be great if non-technical managers could chose, say Agile over Waterfall, or a certain process of documentation, Ruby on Rails or whatever to avoid it. But the truth is that, for all these decisions, I've seen cases where each choice is the best for maximizing mutability of the final result at minimal cost. So, you are kind of helpless to avoid it, and have to trust in your tech team. And choose a team with an eye towards how much that future mutability is worth to you. You can ask your A-team to cut corners to save cost, at the risk of future mutability. But in my experience, even given infinite time, weak technologists cannot develop highly mutable versions of complex systems.