chasing ROI

1. 1

   ML on June 28, 2002 at 10:58 am

   ROI will always be an important resource for understanding value of efforts. I see this concept creeping up again and again in all my product management and marketing class, books, and notes. Even if it is a mirage, there are folks who need to find new ways to measure and instrument their work in order to get some kind of # that would in turn provide more $$$ for future projects.

2. 2

   jz on June 28, 2002 at 5:15 pm

   While I don’t disagree with the article’s suggestion that usability engineers could benefit from training in ROI calculation, I agree with you, Christina, that ROI is in large part, a mirage. In my experience, most important business decisions are not based on ROI.

   For example, what is the ROI of Prada’s decision to hire architect Rem Koolhaas to design their flagship store in NYC? Is there a quantitative method for evaluating the return of hiring Koolhaas versus a less expensive architect? What if they didn’t hire an architect at all, but rather relied solely on a construction firm to design and build the store? Or what if they simply purchased an existing building and converted that for their use? Economically speaking, which offers the greatest value? I could be wrong, but I’ll bet that Prada never bothered to rigorously compare those options.

   Similarly, what’s the ROI of Prada having a web site? They don’t sell products on the site, so you can’t track direct revenue from the site. Even if you could, you still couldn’t calculate ROI as a simple cost vs. revenue equation. You’d have to factor in influenced sales, natural sales rate, fulfillment costs…probably a whole bunch of stuff that people don’t really have a very good picture of.

   Attempts of projecting ROI have, in my experience, done far more harm than good. Usually what happens is people present a calculation, and immediately everyone’s attention turns to underminding the numbers. ROI projections always include a degree of assumption — they’re projections afterall. So, instead of people focusing on the common sense explanation of why investing in usability is going to help make a product better, instead people pay attention to why your calculation is wrong.

   Besides this, there are a score of other problems associated with trying to project ROI of usability, not the least of which is the fact that usability cannot be objectively stated. Usability does not exist as a yes/no quality of a web site, just as it does not exist as a yes/no quality of physical products. It exists as a relative quality. ROI projections, therefore, have to be projections of the value of incremental improvements in usability, i.e. the value of having something be relatively more usable. You can only measure such incremental improvements with a discrete set of parameters, knowing what the current degree of usability is versus the predicted state. It’s possible, perhaps, to do that for relatively basic and minor changes, say the effect on productivity of moving the button on a machine used in an assembly plant. ROI is irrelevant, though, when trying to predict the value of usability in product innovation. If you’re designing something new, to what degree does usability contribute to its marketplace value? How do you know? Whatever the relative usability of Windows when it was released, to what degree did that usability influence its bottom line? What’s the difference between its usability and its usefulness? In essence, to the extent that usability is an attribute of a design, it cannot be measured unless there exists a design of relatively more or less usability, with all else being equal.

   There’s the old quote about not being able to say what obscene means, but knowing it when you see it. I think most people think that way. They’re not really sure how to define good design, but they know it when they see it. So it’s far better to focus on telling a good story to potential clients. When it comes down to it, they’re just going to go with their gut anyway, since they know you can’t really count anything that counts.

3. 3

   Lawrence Krubner on June 29, 2002 at 12:35 pm

   People who have to come up with numbers to justify business decisions are going to become good at coming up with numbers. Therefore, it stands to reason that ROI numbers are at least as solid and reliable as the financial numbers that publicly traded companies like Enron report to the SEC. And, of course, we all know how reliable those numbers are.

Comments are closed.