Design

Statistics class

Now, god knows I can barely multiply and divide without resorting to toes or a calculator, but even […]

Now, god knows I can barely multiply and divide without resorting to toes or a calculator, but even I know that this train of logic is poor. From Improving Usability Guideline Compliance (Alertbox June 2002), Nielsen says that he looked at 20 sites last year, 15 this year, and there is 4% better compliance with usability guidelines and therefore the internet will be fixed in 2017?!?!?!

Jumping Jesus on a pogostick.

First of all… 15 sites??? 15 sites??? I spent the other day looking at a bunch of websites that sell music looking for a good one to take screenshots for That Damn Book(tm) and most of them flaunted not only the usability guidelines, but those of good taste and common sense.

I don’t know much about the infamous guidelines he uses, but I do know that the internet is too much in flux right now to hope for stable standards. Are the guidelines changing also? And honestly, do guidelines really make usable sites? Wouldn’t it be better to combine testing with heuristic evaluations, if you are making grand pronouncements like “the average e-commerce site complies with 49% of established usability guidelines.” The average e-comerce site…he hasn’t even seen enough sites to even know what the average music e-commenrce site looks like (I do, and I’m 49% offended, and 87% despondant over the state of them)

The whole thing makes me deeply uneasy. And “International Websites lag behind”… he looked at six! Six sites? What if he looked at the wrong six sites? It’s like saying I looked at 15 American women and 6 foreign women and American woman are much better looking. If someone said that to you, you’d wonder what women he was looking at… and what women he missed.

This article gives me the screaming willies. maybe I am way off base, but this seems like the most irresponsible thing that the Dane has put out. God forbid anyone read it and believe the conclusions.

Am I on drugs here? Any statisticians out there who can back me up? or prove me wrong. Hey, either way.

14 Comments

Add Yours
  1. 1
    Eric Scheid

    I agree with your sentiment. How the blue jeezus could anyone extrapolate 15 years into the future based on only two datapoints 1.5 years apart?

    As to the crudely scant number of sites he studied, well I’m far more impressed with a study of over 1,800 sites. Although JIm Ho’s study wasn’t on “usability” but instead “functionality”, it still manually examined hundreds of sites, a similar degree of difficulty.

    It’s an interesting study btw on the question of comparing how “advanced” USA websites are vs international websites.

  2. 2
    PeterV

    Jakob must be doing bad drugs. I see no explanation for this (and I even understood why he did the “Flash=99%bad” thing, this is worse).

    Believability down 98.7%.

  3. 3
    Eric Scheid

    Creating a debate about the stats and/or methodology assumes that the stick he’s measuring by is appropriate. The more people assume that the “Usability Standards” he harps on about (which are in truth Jakob’s proprietary collection of usability recommendations, available only by fee), the happier he (and his bank balance) will be.

    What kind of “standards” are these which are not public, and not ratified by any standards organisation?

  4. 4
    DonnaM

    Wasn’t this incredible. He stated this last week at the NNg conference and I nearly fell out of my chair.

    We are having enough trouble selling usability/IA or whatever, without the ‘guru’ making up amazingly rubbery figures.

    But look at this (from the article in question):

    “It’s annoying to spend so much time documenting 207 principles that are known to make online shopping easier for users only to discover several months later that most of these best practices are still not in place on most sites.”

    How egotistical. So first he has to publish them, then we have to fork out a load of $$ to buy them, then we have to implement them, all in a few months! Give it a few years for any of the larger sites I’ve worked on. No way can you redesign a site to be perfect in a few months!

    And overseas sites lagging behind US sites. I’ve seen good and bad on both sides of the ocean. How dare he put the rest of the world behind based on a sample of 6 sites.

    Grrrr…can you see the steam coming out my ears.

    Donna (in Australia)

  6. 6
    Derek Brown

    The boy J has done it again! If he were not such an arrogant, egotistical, self righteous and pious pontificator (swalloed a dictionary – sorry), perhaps people would appreciate the work he is trying to do. Instead of this he alienates, insults and degrades anything which does not match HIS self appointed (and self opinionated) standards.

    This guy defies belief!

  7. 7
    Stephen Downes

    I agree with your observations. Nielsen’s conclusions extend far beyond the depth of his research. More disturbing is the requirement that you purchase his report in order to determine what the criteria for evaluation were. There is no way to meet Nielsen’s standards without paying Nielsen money. Not good.

  8. 8
    Mick McAllister

    Read the Alertbox in question, expecting to be aghast. The data is what statisticians call, in their arcane jargon, “silly.”

    One of many minor details that are missing:

    How long before the first sample did JN bring the tablets down from Sinai? If the guidelines were published within a few years of the first sample, then dissemination of the information (some it almost common sense, some of it by word of mouth) accounts for some of the 4% leap in compliance.

    Extrapolating from two samples across a gap eight times the size of the gap in the samples? Makes sense to me. Let’s see, my apple tree produced ten pounds of apples in its first productive year, and thirty pounds of apples in its third productive year. So sixteen years from now, I can expect around 250 pounds of apples, and in 32 years, a ton. Yowee!

    Seriously, the “international sites lag behind” data ticked me off. I’ve been in the localization trade for nearly twenty years, and I thought we had finally outgrown the notion that “doing it right” means “doing it our way.” Usability needs and preferences are culture-dependent, Mr. N. Duh.

  9. 9
    Sebastian Fiedler

    Let me just throw in some more questions…

    What qualifications has got Mr Nielson and his team to evaluate international Websites?

    Do they speak other languages?

    Have they ever worked and lived in parts of the world where English is not the first language?

    Whatever happened to modesty? 😉

  12. 12
    Andy Yadegar

    What kills me about JN is that his ignorance and “I, me, my, mine” attitude is making him even more famous out there than before. As a User Experience Specialist, I have never thought of JN as a credible individual. Granted, he has some great “ideas” and a few books, but that’s just about it. It is very easy to sit back and spit out rules and regulations, but we all know very well that usability isn’t an out-of-the-box – or even out-of-the-rulebook – practice. He is nothing but an elaborate self-marketing machine, who hasn’t achieved any substantial and applied accomplishments as of yet. All he now needs is his sidekicks and marching song and he’ll be the next guest celebrity at your local wrestling arenas.
    Every usability practitioner has proud moments and projects, but that does not grant god-like privileges. What scares me now is that he is going to put his claws into Flash as it is crossing its chasm. Macromedia is going to use his status to, in my opinion, market Flash to a skeptical customer-base. That’s it.
    IMHO, Macromedia should have never come to terms with JN: in fact they should have hired someone like Scott McCloud who truly represents applied experience.
    Ok… I’m going to stop before I REALLY get annoyed! 🙂

  13. 13
    Melody Callahan

    I must admit I have read alot of JN stuff and really liked much of it. I enjoy the fact that
    he admits upfront that there is no possible way for anyone to meet all the usability ‘rules’
    AND have a site that function. I also recently read and interview with him where he admits
    to being outspoken and not ashamed about it, or things he may have said in the past (his
    bashing of Flash article).

    I wonder if what he’s really doing hasn’t been amazingly successful. He’s managed to make
    all of us wonder, questions, dig deeper, discuss and THINK about out sites and how they
    work. Do they work, can we make them work better, can we collaborate, what else is out
    there….etc.etc.etc. Maybe his ‘flip’ statistics were just a way to cause this thinking
    outside the square. Which means we will all have better sites in the end (whether we follow
    his formula or not)…we will be driven to have great UI.

  14. 14
    Hey OOMTA

    Andy or should I say Ontario Web Design, you talk about ignorance. Everyone at CIBC knows that you and Taylor stole the TMS code from Nestley. You know very little about User Experience with your lack of education. If you think you got away with stealing the code you are wrong. You have a lot of enemies that are making sure this goes to more people. People now know that Michele knew before that Taylor stole the code and told you guys about the security sweeps of your computers. It is all coming down soon and we will love every minute of it. Remember there is another person who was there that saw everything and can prove it all. Also, you won’t be getting any interviews at RBC. They know about you.

Comments are closed.