Why Does Vista Suck? Blame Automated Testing

At least, that's the opinion of Joel Spolsky...

...this search for the holy grail of program quality is leading a lot of people to a lot of dead ends. The Windows Vista team at Microsoft is a case in point. Apparently -- and this is all based on blog rumors and innuendo -- Microsoft has had a long term policy of eliminating all software testers who don’t know how to write code, replacing them with what they call SDETs, Software Development Engineers in Test, programmers who write automated testing scripts.

The old testers at Microsoft checked lots of things: they checked if fonts were consistent and legible, they checked that the location of controls on dialog boxes was reasonable and neatly aligned, they checked whether the screen flickered when you did things, they looked at how the UI flowed, they considered how easy the software was to use, how consistent the wording was, they worried about performance, they checked the spelling and grammar of all the error messages, and they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one.

None of those things could be checked by automated scripts. And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished.

Interesting...

This isn't to say that automated tests are useless... far from it. However, if your requirements for "tester" are the same as your requirements for "programmer," then you're not really doing testing. All programmers test their applications... that is, programmers test applications in the way the application is supposed to be used. They are not bothered with inconsistencies, usability, or how the product "feels." Your target market cares... but your developers usually don't.

For that, you absolutely positively need testers who are non-programmers. You need feedback from people like your target market, to give you information on how to make the product better. Even if your target market is nothing but developers -- say, you're making a new programming language -- you still need non developers to test your products... there's simply no better way to test intuitiveness.

Most importantly, you need to teach your programmers that such feedback should be taken very seriously... although always with a grain of salt. Requests for better usability should always be followed... whereas requests for new functionality need more scrutiny. In my experience, highly technical feature requests frequently mask the true needs of the client. You should understand the goals of your clients, and give your developers the freedom to help achieve those goals.

As my database professor John Carlis always said... the client doesn't always know what they want, but they are very fast learners!

UPDATE: Lots of great quotes in the comments... my favorite: "Automated tests ensure the software is working as designed. Human tests ensure the design is working."

comments

Or put another

Or put another way.

Automated tests ensure the software is working as designed. Human tests ensure the design is working.

I was wondering what new

I was wondering what new interesting information was about to pop up on my screen and then I see "At least, that's the opinion of Joel Spolsky...". Yeah, I'd already read that on his site.

Not that I think he's bsing, but he does have a mean streak (no pun plz) for thinking he's always right, so I would like to get a second opinion on some of his statements, including this one.

You really think Microsoft

You really think Microsoft doesn't understand the importance of usability testing? Really?

The problem isn't automated tested or a lack of usability testing, the problem is politics.

Internally, MS is all over the place. There's no good sense of direction or leadership, management is a mess, communication and cooperation are lacking.

Re: You really think Microsoft

You really think Microsoft doesn't understand the importance of usability testing? Really? The problem isn't automated tested or a lack of usability testing, the problem is politics.

I agree, politics is probably the main problem here. Microsoft has plenty of smart people who know what needs to be done... it just appears that the wrong ones were at the helm when Vista was being put together.

Hagrin's approach is interesting... a public beta would have helped out a lot... just like OSX 10.0 helped Apple avoid some doozies... but, as they say in Dont Make Me Think, you want usability feedback as early on in the process as possible. Otherwise, it will always get cut from the feature list. Then you're left with a feature rich product that feels clunky and unusable.

RE: blame automated testing...

Automated testing isn't designed to evaluate usability or workflow design. Automated testing isn't the reason Vista "sucks," it's misunderstanding where it can be applied, and if Joel is right the blind reliance on it being the quality silver bullet. When it comes to testing usability there is a whole other set of practices and disciplines, apart from TDD or automated testing. It usually requires real customers, prototypes, scenarios, profiling, etc. There's no way to automate usability testing.

I don't quite follow your line of reasoning on how this is a failure on the part of automated testing and not using "real" non-programmer QA people. I see this as a failing with the project's management and execution, especially around design.

Vista was a huge project, I can't even imagine how many LOC it involved. It was years late. Am I to understand your argument that QA people should evaluate the usability of the product after it's completed? It seems like you want to figure that out earlier, possibly as your developing the product. By the time it reaches the end and you realize the workflow makes little sense or is not consistent with the other parts of the application it will cost too much money and time to fix.

I think this is why Agile teams approach software development better than traditional SDLC. You demo your progress every 2-weeks to the customer, so you are not so off course that you couldn't easily adjust your design to incorporate feedback. On top of that you have a cross-functional team (developers, quality assurance, analysts, etc) so that the team is approaching the problem from all points of view - not just the developer's.

In the end happy customers will buy your product, and customers are more likely to be happy when the product does what they want, and it will do what they want if they helped you build it. Or if you're M$ just muscle your way into people's homes on new PCs, but the rest of us need to keep customers happy.

Intel has the same problem

I worked in the software driver division, and the managers there had eliminated the testers, and everything was being done with automated testing suites. Every bug that I found was from running the software myself (using my tester hat). When will they learn that human driven test is the way to go?

Automatic testing is great for regression, and has its place. It still won't catch the majority of bugs, especially aesthetic problems.

I don't think this was the problem with Vista

As a former SDET at Microsoft (although admittedly, not on the Vista team, so I don't have first-hand knowledge), I would say that Joel is dead wrong here (like I said, I was an SDET, so I'm sure I'm a little biased in my views about the value of automated testing).

Well, he's absolutely right about some things. You cannot use automated scripts to find all the usability issues he enumerated. That's 100% true. But it's definitely not true that Microsoft's push for automated testing meant that testers stopped (or were encouraged to stop) testing for those issues manually.

Think about it this way - every piece of software has two buckets of bugs. In one bucket is the stuff that can be caught by automated tests: the crashes on bad input, the wrong UI showing up, etc, etc. In the other bucket are the things you will only catch with human eyes - usability stuff, as well as all the really weird corner cases that are prohibitively expensive to automate.

And of course, each tester only has a limited amount of time for each product cycle. Now can spend their time (and mental energy) manually performing the same tests over and over to catch bugs in the first bucket. But, it turns out, this is a huge waste of time and energy. So a sound testing policy is to automate as many tests as possible to catch those types of bugs (this includes both unit and functional tests).

Joel seems to assume that this effort to automate testing reduces the amount of time to do manual testing, when in fact the goal is to automate the boring stuff so testers will have MORE time to do the interested impossible-to-automate testing. For instance, on my team (and I know other teams do this), automating our tests (and running them nightly) let us schedule multi-day bug bashes (the entire team competes to manually find new and interesting bugs), app competitions (I worked in Visual Studio, so we would organize into small teams and use the product to build non-trivial applications), and full test passes (we ran all our automated tests and went through our checklist of manual tests as well).

So, from a hiring perspective, I'd disagree that "you absolutely positively need testers who are non-programmers". What you need is testers that can program but also can test manually. And it's not true that being a good SDET means you won't find good bugs manually (the most talented tester on my team excelled at writing automated tests but also found the most amazing nasty bugs manually). What is more important than having non-programmer testers on your team is to have people USE the product who are not programmers - and most importantly, who are not on the team. As for non-programmers, we had PMs (program managers) who would use the product constantly and aggressively look for usability issues, including holding multi-day usability studies with real users (PMs do a lot more than just worry about usability, and many of them were excellent programmers, but on a daily basis most of them did not write code). As users who were not on the team, we regularly released out software internally to other employees at Microsoft and to external beta users who gave us great feedback.

I'm not claiming that Vista is a great product or MS's testing strategy is flawless. But from my experience at the company, their approach to testing is very carefully considered and includes a focus on both automated and manual testing. I'm not sure what went wrong with Vista, but it's incorrect to say that MS in any way abandoned manual usability testing in it's quest to automate more tests.

Re: I don't think this was the problem with Vista

So, from a hiring perspective, I'd disagree that "you absolutely positively need testers who are non-programmers". What you need is testers that can program but also can test manually.

My point is that you need to expand the definition of "tester." You want end-users testing your products as much as possible. Ideally, people who know nothing, other than that they have a business problem.

Your approach of using Product Managers is a good example... as are public betas, or usability focus groups. Usability specialists who can code are a rare lot...

wtf is a database

wtf is a database professor?

sounds like you're a half-assed wanna-be developer.

plus, real engineers don't care about databases, that's why we have DBAs.

and small time developers that DO care about databases would never refer to a data structures professor as a "database professor"...

Re: wtf is a database professor?

hehehe... well, you could have followed the link to find out what a "database professor" is... he teaches data modeling in RDBMSes, data management, and database management systems. "Database management systems professor" might be more correct, but there's nothing wrong with shortening it to "database professor."

To further clarify, Carlis wasn't my data structures professor... that was Nikolaos Papanikolopoulos. Any "real engineer" would know that database systems and data structures are completely different things.

RCs?

While I agree with the basic premise of the article, I wonder what the purpose is of making betas and release candidates available then if that user feedback (i.e. your human test bed) isn't going to be properly implemented. One would think that many of the issues that Vista suffers from was addressed through the public beta testing period especially concerning usability issues.

I really have no problem with MSFT not having any human testers on their payroll, but I would want them to supplement that action by using beta feedback more appropriately.

Right On

A great piece of advice on designing software:

"You are not the user"

PS:Your captcha is innovative, but also slightly annoying. A user should only have to get the right word once between "Preview" and "Post"

Non-programming testers

Thanks for pointing out that technical testers, user experience personnel (designers and appointed UI experts), and general functional testers are different. All are important parts of the development stack, but Spolsky's extrapolations about their effect on Vista are disingenious at best, at worst simply stereotyping.

Maybe Joel should slow down on the blog-marketing and start fixing his own problems.

You forgot to mention the

You forgot to mention the impact of outsourcing to so called "INTELLIGENT HINDU" IT sweatshops in India.

well that's negative...

One of the benefits of outsourcing automated and semi-automated tests to India, is that they test the code the developers built at the end of the day. Then the error are there in the morning when the developers get into work...

That leaves more time for testing that cannot possibly be automated... which is incredibly hard to outsource because it requires vast knowledge of the product, and access to the engineers.

anyway, my two cents...

"One of the benefits of

"One of the benefits of outsourcing automated and semi-automated tests to India, is that they test the code the developers built at the end of the day. Then the error are there in the morning when the developers get into work..."

I must be missing your point...

Can't automated tests be run onshore overnight? Overnights and weekends are regularly used for automated test executions where I work. Why pay somebody to click a button and then sit on their a* for 8 hours making jokes about stupid Americans?

beta process

There's yet another good recipe for Microsoft of how not to screw testing.

Don't outsource reviewing beta bug reports to offshore $1/day drones who were trained to provide "support" of "customers' by launching prewritten templates, however irrelevant, and who are paid on per-incident basis, therefore interested to close as many "incidents" as possible by launching whatever template comes handy; however, with ritual polite "thank you for being Microsoft customer" in the end of each such humilitation.

Don't thus make beta testing a travesty.

beta process is to late to make a difference

In my opinion the bugs found in beta process are not really relevant because they are reported after the product has been designed and therefore to late in the development process to make any substantial changes to the usability of the product.

The archaic testing methods

The archaic testing methods you endorse cannot be done by cheap-labor immigrants from India with eccentric English. Nor can previously routine tasks demanded of graduates of more rigorous education systems.

Blame the H1B.

Read www.vdare.com

Get real.

Vista is a step back...

With Vista users face driver problems again? Give me a break! That was WIN95 or Win 3.1. Moreover the idea that an extended GUI or UI needs up to 6 GB of hard drive space is just ridiculous. We are talking about OS only. Companies aren't going to deploy VISTA because of its sheer size and their limited network capability.
Nowadays virtualization is the way to go anyway, which takes the burden of the software distribution from the network.
I remember when we run OS/2 on Novell on diskless workstations in 1997! It worked great and software distribution was not a task, because of the centralized servers.

VISTA one the contest of being the worst out of 10 worst IT products in 2007.

Usability testing

Automated test are used to test the functionality of the app, what you are taking about is design/usability issues. Usability can only be tested out manually since its all about HCI. You need usability testing as well as functional testing - come on everybody knows that....

Re: Usability testing

if everyone knows it... then why do so few people do it? Including, at least in this case, the Microsoft Vista team.

You'd be surprised how few people value usability testing in the enterprise software world. Its commonplace for consumer software, and public facing web sites, but enterprise software is waaaaaaaaaay behind the curve on usability. Its not that they know they should do it, but politics gets in the way... its that they just don't get it.

Software products start out

Software products start out with the requirement of being feature rich.

As the market matures, the emphasis is on being easy to use.

Jobs is great on designing products that are easy to use.

Microsoft products are only now starting to feel extreme market pressure for "ease of use"

Joel's products are competing in a mature market. The technical feature list and the algorithms have been well understood for at least the last decade or so. So Joel concentrates on "ease of use" and gets to start with a clean code base (ie from scratch).

Microsoft's OS advantage is the large amount of existing working programs. Compatibility is way more important than usability.

A lot of "ease of use" is "putting lipstick on a pig".

I once had to program to their winXP security APIs. There are bits that have different symbolic names. The security permissions aren't orthogonal. I originally did my testing by launching the program from the command line. It was very confusing to me why it wouldn't run from the start menu. I mean, how does it make sense to require LESS security permission to run from the command line than from the start menu?

Re: "One of the benefits of

Can't automated tests be run onshore overnight?

Absolutely, but that's why I said automated and semi-automated. Some tests can be automated, like unit tests... which can be run on-site. Others can't be automated, or they change so often that its cheaper to write a loose test plan, than a test script.

Also, there is value in hiring a specialist firm who knows how to write test drivers... I'm not just talking about India, this also means local contractors. Unless your testing process is a major part of your competitive advantage -- and sometimes it is -- you should think about hiring a specialist part-time.

Re: Software products start out

Microsoft's OS advantage is the large amount of existing working programs. Compatibility is way more important than usability. A lot of "ease of use" is "putting lipstick on a pig".

Sometimes... but in your examples of orthogonality, sometimes its as simple as a new facade or a new interface. Java could use a bit more orthogonality as well... why is it sometimes length() and sometimes size()? Crazy talk, I say...

I bet bugs were found and reported but not fixed

I think it probably more of an issue of the project priorities and what bugs were fixed.

I'd be interested in what the percentage of reported Vista bugs were "consistency, flow, and ease of navigation" bugs and then what percentage of these bugs made it through Bug Triage meetings and were actually fixed.

My guess would only "crashing" and "functionality not working" bugs were fixed in the last 18 months of the project and that the high-priority usability bugs didn't make the cut.

BTW, I think of Apple as having products that have good consistency, flow, and ease of navigation, has anybody ever taken part in an Apple "Bug Triage Meeting" and have any insight on how they treat these types of bugs?

Re: I bet bugs were reported and not fixed

Your point is perfectly in line with the idea of early usability testing. Agile folks talk all the time about writing the test driver before the code, but not many people do that with usability... probably because the UI is always a work in progress.

Maybe start usability tests as soon as you have a "wireframe" UI? Maybe as soon as the developers are using the UI intead of unit tests? Its hard to say...

What's certain is that Vista did it too late in the process...

Vista sucks bad

I hate Vista and I hate Microsoft. Microsoft is a monopoly and should be abolished.
Follow me on this one:

You know when they are going to build a new freeway and your house happens be in the way and the city comes to you and says,"sorry, you have to move and sell, we'll pay you ... but for the good of the state this has to be done! So get out of here"?????

Likewise, a united nations of IT, and telecom needs to go to Microsoft and tell them they simply cannot monopolize everything and simply take the OS away from them. Let smart programmers develop the Windows OS into an OPen source OS like Linux.

Releasing Vista proves that they simply want to make money because they think consumers have no choice and have to move to VISTA. This is actually, Fraud and Highway Robbery all in one. I'm being forced to move to a New OS for no reason or threatened to be left behind. THIS IS ABSOLUTE BULLSHIT! In other words, PAY UP or get left behind is what MS is saying.

Intel telling Microsoft the other day that there is "NO COMPELLING REASON TO GO TO VISTA", is the first blow or final blow!
My experience is that VISTA is horribly slow (and no it is not all 3rd party software ad ons that cause this)..

JUst moving things around and burying all commands under tons of menus doesn't constitute for a new operating system. If I didn't know better, MS tried to copy Apple by including a CAlender, and sort of tid bits that MAC has.

I create a file and it tells me I don't have the permission to move it or delete it! I am the administrator and I have already turned on the annoying security features.

New surprises pop up everyday in VISTA.

All of a sudden one day, none of USB devices were recognized or my 8 gig flash drive took 4 hours to open a folder!!!

VISTA blows donkey cock!

Why Dose vista Suck?

Vista is a step backwards, i had alot of extras on from my last pc, i was running ( windows xp) none of the shit will work with vista? why the heck would u up grade to a os thats not compatable with lets say for example your web cam ?? you mean to tell me if u have to purchaces vista you have to purchase new printers scanners web cams ect???? that such BULLSHIT fuck billy gates an a BIG FUCK YOU to MicroSoft you guys need take vista an shove it up your asses!!! i bet none of the ms employess use vista why would they it'll jus fuck up there nice expensive laptops!! cause mine is fucked an its new just 6mons old never had a problem with xp an as soon as i can get my hands on a xp os ima fly vista like the fucking frizzby it is!!!!

Vista sucks bad Part 2

As a follow up to my last comment. I've had it with Vista, so I installed a double boot of Vista/OPENSUSE Linux with KDE 4.0. It works great, I don't see why people bother with windows when they have linux.

Compelling reasons to go to Linux... (it is not Apple!!... I wish Mac and apple would go away).. Sorry but Mac OS is going to die a horrible death soon-- just like their computers. People want affordable, cheap solutions.. not $3000 computers that are outdated in 6 months.. (And don't give me that "a powermac is good for 10 years BS") Any computer is outdated because of technology--- rather quickly.

So you have Microsoft on one end and Apple on the other extreme. To me, Apple is following the same sort of bull that Microsoft is. Let's see, they want hella money for Macs, they keep releasing new Operating Systems for money and Ipod just sucks. Why pay $300, when you can buy a great Mp3 or Mp4 player that supports simple plug and play and folder song drops for about $100. (Try Sansa Players) My Ipod totally crashed out and took all my songs with it! I've had Itunes erase my songs many times and finally it started freezing.

So my final point-- Do you want a middle of the road operating system that can be installed on your computer?....GO FOR LINUX!
If all you do is check mail, browse the web, keep a journel and maintain a schedule in a calendar then LINUX OS is for you! Trust me, my OPENSUSE boots up in 20 seconds and shuts down in 15 seconds. When was Vista or XP ready as soon as your desktop came on? (usually there are a thousand processes running in windows)
Linux is ready the second the desktop appears. There is no lingering overhead or none that interfere with applications. Linux shuts down quick. Usually my Vista takes up to 2 minutes and sometimes (believe it or not) 5 minutes to shut down. Sometimes I just turn off the power strip on my vista machine.
You can do a Windows/Linux double boot.. rather seemlesly. OPENSUSE for example, takes care of the partition and everything else.. just download and install the CD. Then you will get a boot screen at start up asking which OS you want. It is so simple! Currently, I use Linux OS for 'you tube', multimedia, mail checking, personal stuff etc. The only time I go to windows is when I have to run specific engineering software that has no linux version. Let's face it, Vista just can't handle Multimedia well, it is doing something to overwork the CPU. which heats up.. which causes the fan to spin crazy and make unpleasant loud noises. The same multimedia works great in OPENSUSE! This proves that it is the OS and not the hardware!!!
The only tough decision you have to make is which linux distro you want in your computer. OPENSUSE and UBUNTU are the big desktop players, and RED Hat is robust as a desktop or a server. I like OPENSUSE.

RE: Vista sucks bad Part 2

Thank you, Vistor (not verified), for that specific information.

I have probably spent a collective 50 hours getting my games, music composition, and other numerous sundry programs to run on Vista Home Premium 64-bit - no exaggeration (in fact, 50 hours is very likely a conservative estimate). Driver issues, general usability issues, I wont' go into the details, because I'm moving on. I have these things called "hobbies", many of which involve "computers", many of which, I don't know, what's the word I'd looking for, make me "happy", which is I why I pursue them. And while the 'music scroll' in Cakewalk Music Creator looks great just scrolling through my song, for some reason, I'm just not happy imagining there is any actual music being produced in the background of this loveliness.

Anyway, I thank you because, although I've spent a lot of time searching for 'vista sucks' and other Vista-bashing material on the Web, just to ease some of my frustration, your post seems like a much more positive approach. Thank you for including the details about exactly how to install a dual-booted OS, and for mentioning you can see all the same media, but quicker, and that you can install the OS with a CD, rather than downloading a bunch of crap, and that you can easily select which OS to boot. I think your post is actually what is going to push me to go ahead and dual-boot Linux.

Too bad Linux is not advertised. Of course, it wouldn't be, because it's open-source. If there were the slightest marketing dept behind it, directed toward the average consumer, there would be no Vista or Microsoft.

God, I'm beginning to think I should write my congresswoman, and ask her to introduce a bill to sponser Linux advertising --- FOR THE PUBLIC WELL-BEING!!!!

-Don

yeah i agree

yeah i agree

Why does VISTA Suck

I agree I have xp on some of my pc's I'm beta testing Windows 7 64bit it uses half the memory of Vista ( it came with my HP laptop and if you change the OS you void the warrenty)....Windows 7 made my single core amd come to life and I only have a gig of ram in it. Wait for windows 7......till then keep XP.

Recent comments