Relay-Version: version B 2.10 5/3/83; site utzoo.UUCP
Path: utzoo!utgpu!water!watmath!clyde!rutgers!ames!oliveb!sun!gravity!klein
From: klein@gravity.UUCP
Newsgroups: comp.arch
Subject: Benchmarking and Realities of the Marketplace
Message-ID: <20189@sun.uucp>
Date: Tue, 2-Jun-87 14:23:41 EDT
Article-I.D.: sun.20189
Posted: Tue Jun  2 14:23:41 1987
Date-Received: Thu, 4-Jun-87 06:32:05 EDT
References: <4667@fritz.UUCP>
Sender: news@sun.uucp
Reply-To: klein@sun.UUCP (Mike Klein)
Distribution: world
Organization: Sun Microsystems, Mountain View, CA
Lines: 49

In article <4667@fritz.UUCP> martin@felix.UUCP (Martin McKendry) writes:
>Gentlemen:
>
>I have read with interest the discussion on benchmarking.  I am
>sure that a benchmarking bake-off would be a good thing
>for prospective customers.  But for the engineers in the
>audience, I would like to point out some of the social problems 
>associated with touting your company's wares.
>
>Any bake-off will have one winner and many losers.  As an engineer
>working for a particular vendor, you may have run tests that
>convince you that you will be a winner.  But be assured that in 
>some dimension you will be a loser.  And if you proposed that 
>your company enter, you will be the one "responsible" for the 
>exposure of "weakness".  

I am happy that someone finally pointed out these issues.  As engineers,
we all want to know the "truth" about performance and limitations.  While
it would be fantastically valuable for the progress of our industry to
understand the true values of architectural and implementation decisions,
which a good set of benchmarks may help with, it is pretty much inconceivable
to me that this can occur under the auspices of a trade show or conference.

Marketing computers and computer systems is a very complex job with
great risks and rewards.  A poor sales/marketing job, irrespective of the
technical merits, can sink a product faster than almost anything else
(except maybe poor quality).  Submitting a product to an open benchmarking
when it is known in advance that the product will not be the "best" is
poor marketing.  Submitting it to an open benchmarking when it is not known
in advance how the product will perform is also poor marketing.  No marketer,
salesperson, or any other high-level manager worth his/her salt will consent
to a public demonstration of their product's inferiority.  As a
representative of your company, if you benchmark your system and it performs
poorly, and you therefore cause the company to lose sales, you have just
done yourself and your coworkers a major disservice.

Maybe this isn't how it should be, but it is the reality of the marketplace.
I am speaking in very general terms here, and this should not reflect on,
or imply any actions of, my employer or anyone else's.  Most companies in
such a technical field try to present as much detailed technical information
as possible for their products.  For the meantime, we will need to rely on
open, informal communication such as we've had from AMD and National, for
example.  I commend them very highly for this.  But I don't expect to
see the AMD booth or the National booth running the same benchmarks, and
I understand the reasons why.
--
Mike Klein		klein@Sun.COM
Sun Microsystems, Inc.	{ucbvax,hplabs,ihnp4,seismo}!sun!klein
Mountain View, CA