Software and Data Preference Polls: Not Gauge of Quality

Bob Pelletier of CSI - It's that time of year again. Stocks & Commodities Magazine ran its annual "Reader's Choice" contest and the winners are trumpeting their accolades all over the investment rags. This brings up the annual controversy over user preference and just what "Reader's Choice Winner" really means.

For those of you who don't read S&C, let me explain the competition. The subscribers of S&C were sent ballots listing advertisers and potential advertisers of the magazine. They were asked to select their one choice in a number of categories.

Let me be clear on this: They did not compare the competing products or services as good, bad, worst, etc. . . . Nor were they asked to rate individual aspects of the contestants, such as quality, service, etc. The winner in each category was, quite simply, the one with the most votes.

The fact is, the so-called "Reader's Choice Award" in each category of software or data has absolutely nothing to do with the quality and performance of the product. Most readers may have known of only one product in each category. Regardless of the superiority of the other products about which nothing was known, the product familiar to the reader got the vote.

While this type of balloting shows the product most frequently used by S&C readers, it does not say on what basis (or under whose influence) the choice was made. None of us can know the private list of candidates each reader was familiar with when casting a vote. In this flawed study, a non-vote due to unfamiliarity with a product carried the same weight as a non-vote due to a bad experience.

The editors at S&C are the first to point out that quality has no bearing on the outcome, but this message is lost in the media circus created by the winners. MetaStock used their win to promote their software when compared to a competitor. Likewise, another winner showed their multiple awards in a recent magazine promotion.

Using S&C's "Reader's Choice Award" as evidence of progress in one's own accomplishments brings to mind George Santayama's quote, "Those who speak most of progress measure it by quantity and not by quality." There are more product longevity and satisfaction in quality than any other attribute of performance. Unfortunately, S&C left quality out of their questionnaire.

It is interesting to note how very laudable the study seems to the winners and how very biased it appears to the dozen or more losers in each category. As a "Finalist" in the data service category, we find ourselves in the crowd shouting "foul." When CSI lost out to competing data services, we felt compelled to advise S&C of the flaws in their study. Neither of the two letters we submitted to the editor, making several valid claims of bias, were accepted for printing.