262 - Avoid Intellectual Debt In The New Year
We all accrue intellectual debt: notebook records instead of detailed logs, updated spreadsheets rather than new databases, four different systems instead of a single streamlined one. We justify these little shortcuts in the name of time and money, hoping they won’t snowball into bigger problems.
But they often do, affecting our employees and customers who then must find workarounds for our cheap solutions and must pay the hefty interests on our ballooning debts.
The Origin And Evolution Of Intellectual Debt: Technical Debt
To understand how to avoid intellectual debt, we must first look to its older sibling, technical debt, a software development term coined in the early 1990s by Ward Cunningham, the founder of Wikis: “A little debt speeds development, so long as it is paid back promptly with a rewrite.”
The analogy has had legs because, ultimately, it’s applicable to all walks of business. As Ben Horowitz wrote in his 2014 book The Hard Thing About Hard Things: “Like technical debt, management debt is incurred when you make an expedient, short-term management decision with an expensive, long-term consequence. Like technical debt, the trade-off sometimes makes sense, but often does not.”
As a fundamental part of software development, technical debt must be precisely monitored. Burning holes in an operating system to show proof of value can save time and money by quickly getting a product into customer hands, where bugs can be discovered and functionality can be refined.
But the speed is the price. As Cunningham first wrote, the irony of technical debt is that the product that initially made the debt tolerable eventually makes paying the debt with a prompt rewriting intolerable.
How Data Management Creates Intellectual Debt
Intellectual debt was actually identified three decades ago, but its relationship to software is largely due to the massive amount of data management required by today’s businesses.
Still, the concept is similar. Suppose you have an idea kicking around in your head: an innovative way to collect and analyze a certain slice of your company’s data. You draw it up on a spreadsheet, see that it works and send it to all potential stakeholders. Once they evaluate your proof of concept and confirm its viability, they’ll begin using it, updating it as necessary.
Here’s the problem: Deep down, you know the spreadsheet’s not really the right solution. There are just too many aspects you didn’t consider -- and too many you couldn’t have considered. But just like with technical debt, you’ve arrived at the same paradox: Your currently successful shortcut obviates the need for the inevitable fix. The longer you sit on your debt, the more the interest grows, making it all harder to achieve that ultimate, better solution.
Your company is drowning in intellectual debt. Every time a decision is made to shortcut a good system design, your customers and your employees are forced to work around your shortcut. These shortcuts are caused by a “missing of the minds” -- a myopic understanding of what you were trying to solve in the first place. This is often the case when a challenge is described as “have a list of all our products” versus a more rounded understanding of “have a complete view of our product line.” The difference may seem subtle. Down the line, a 1,000-row spreadsheet that has 12 similar products spread all over it, coupled with duplicates and errors caused by manual entry, is the root of what holds your whole product division back from taking advantage of stock-ordering, e-commerce and logistics systems in a meaningful way.
Unruly Data And The Need To Economize Operations
Today, many businesses seek to economize operations. For these companies, the real problem is not just that their intellectual debts are big but that they are unruly.
As the business world rapidly changes -- e-commerce, for instance, is only about two decades old but already influences 56% of in-store purchases and accounts for 10% of all retail sales -- IT departments have piled solutions atop solutions. This has led to countless data redundancies, trivialities and inconsistencies -- headaches that stem from growing piles of intellectual debt and impede business practices.
Take onboarding customers, for instance. While it’s imperative to onboard as quickly as possible, how do you balance fast transactions with duplicate entries? How do you assure customer identity without adding friction to the process? How do you deal with 500,000 unique customers spread across 800,000 accounts, each identified by overlapping components? In a world where customer satisfaction is important, this is troublesome. In a world where GDPR is a regulatory requirement, it’s a disaster.
Avoiding Intellectual Debt
Building trusted data in a single view -- across multiple systems and owned by different teams or even outside organizations -- is a notoriously difficult undertaking that lends itself to intellectual debt. Here are three ways to avoid heading down that path.
1. Stop spreadsheet sharing: Solutions that require multiple people emailing multiple spreadsheets are falsely comforting. And just because the spreadsheets are shared in Dropbox or Google for real-time updating doesn’t wish away the problem -- it just shifts the dynamics of the problems to security and version overlap.
2. Think globally: Small problems signify larger problems. If you keep all your products in spreadsheets, you might be missing out not just on easy inventory updating but on customer service capabilities and product implementation analytics and logistics.
3. Resist the over-engineering impulse: Most employees are comfortable using software and systems. Data solutions shouldn’t involve everyone’s input, take a year to build, cost a million dollars and involve mega-training sessions.
Follow these simple steps and you’ll not only have clean data, you'll be able to fuse different business aspects -- a customer master, a product master -- and roll them out across the organization, seamlessly dropping them into tool stacks that serve as a hub for data stewards and business analysts alike, all without a pile of debt.
Source: Forbes
