SPEC

   subject: SPEC Run Rules for CINT92     date: January 23, 1992

                                          from: SPEC Steering Committee



                             ABSTRACT



   This document provides guidelines for how to run the
   benchmarks, how to report the results (which is explained
   more fully in the document SPEC Reporting Rules) and how to
   report results for modified versions of benchmarks.  It
   contains several examples of necessary types of changes.




   Memorandum to
   Performance Analysts









































                                          SPEC



   subject: SPEC Run Rules for CINT92     date: January 23, 1992

                                          from: SPEC Steering Committee



   1.  General

   The general philosophy behind this set of rules for running
   the SPEC benchmarks is to ensure repeatability and
   documentation of all factors pertinent to duplicating
   reported performance results. Suggestions for improving this
   run methodology should be made to SPEC for consideration in
   future releases.

   In general:

      - All times reported must be for runs that produce
        correct results.

      - Single user mode is allowed.  This must be documented
        in a report of the results.

      - These benchmarks have been run across a number of
        platforms with several compilers, but are not
        guaranteed to be free of language extensions that are
        commonly found, but not part of industry standard
        language specifications.

      - Use of software features (in pre-processors, compilers,
        etc.) which invoke, generate or use software designed
        specifically for any of the SPEC Benchmark Releases is
        not allowed.

      - Source code may not be modified unless it is required
        for portability. See Section 4 (Source Code Changes)
        for specifics.



   2.  Makefiles,_Compilers_and_Libraries

   Generic Makefiles are provided for convenience and insight
   into running the benchmarks. Typically a vendor supplied
   wrapper (M.ident) is used in conjunction with the generic
   Makefile. These are provided for convenience and can be
   replaced to facilitate running benchmarks on proprietary
   operating systems or to take advantage of optimizations not
   easily activated with the current method.  It is anticipated
   that the build process which is capable of building the
   highest performance executable will be employed, even if a











                              - 2 -



   different set of optimizations are activated for each
   benchmark.

   For example, one SPEC benchmark can be built with one set of
   options and another can be built with an entirely different
   set of options.

   Special libraries may be used as long as they

     1.  are products announced for availability to customers
         within 6 months of the date of test,

     2.  do not require changing benchmark source code,

     3.  implicitly replace routines in the benchmark source
         code,

     4.  are not "benchmark specific" (i.e.targeted
         specifically for a SPEC benchmark).

   Libraries can be specified in the Makefile. Source code
   changes to eliminate default high level language subroutines
   so that optimized library versions can be used is not
   allowed.

   Accuracy verification specifications (i.e. to diff or spiff)
   must not be overridden in Makefiles (or in any other
   manner).  Only SPEC approved alternative input files are
   allowed. Use of these alternate input files must be
   documented in the footnotes section.

   It should be recognized that each company is continuously
   improving its compilation system and that new optimizations
   may be required to achieve published performance ratings.
   If you are unable to reproduce a vendor's published rating,
   please contact the vendor and ensure that you have the
   latest Makefile wrappers and equivalent system
   configuration.



   3.  Pre-Processors

      - Use of a pre-processor on unmodified source code that
        is part of the identified released software is allowed.

      - Use of benchmark specific pre-processor features is not
        allowed.

      - Documented pre-processor options may be used, and
        should be specified either in the testing report or in











                              - 3 -



        the vendor Makefile (M.ident) provided on the SPEC
        tape.


   4.  Source_Code_Changes

   SPEC policy allows only portability changes to the source
   code.  When source code changes are made, a diff listing of
   the changes must be included in the testing report.

   SPEC encourages performance comparisons to be made across
   systems running the unmodified SPEC code, or if necessary,
   with the minimum required portability changes.


      - Portability

           o Source code changes required for standards
             compliance should be described in the testing
             report.  Appropriate standards documents should be
             cited.  All such changes should be reported to
             SPEC.  SPEC may consider incorporating such
             changes in future releases.  Whenever possible,
             SPEC will strive to develop and enhance benchmarks
             to be POSIX and ANSI compliant.

             SPEC recognizes that some source code changes may
             be necessary in order to meet specific language
             implementations.  Such changes must be documented,
             appropriate references must be cited and results
             marked as portability changes (denoted by * on any
             publicly reported results).  Once these changes
             are incorporated into the tape including alternate
             source files, then the asterisk may be dropped.

           o The double precision (real*8) floating point in
             SPEC benchmarks is intended to be 64 bits.  If
             this is not the case on the machine being tested,
             modify the appropriate declarations and forward to
             SPEC the source code changes required to
             accommodate the system.  Over time, SPEC desires
             to use compiler options to eliminate the need for
             such changes.

           o If any portability change is backed off, one of
             three things must happen:

               i.  benchmark code does not compile, or

              ii.  benchmark code does not execute, or












                              - 4 -



             iii.  benchmark code produces the wrong result.


   5.  Alternate_Source_Files

   Some portability changes made to the source files may be
   generally useful to other SPEC users.  These are classes of
   codes that meet the portability requirement (i.e., code
   won't compile, or won't execute, or produces wrong result),
   but are required by one or more target systems.  In such
   instances, the files are placed in a directory named
   "src.alt" in the benchmark directory.  All files placed in
   this directory have been reviewed and approved by the SPEC
   Steering Committee; alternate source files are otherwise not
   permitted.  When alternate source files have been approved
   for inclusion, however, the M.vendor makefile must be
   modified to retrieve the appropriate version.  In general,
   original copies of codes which are to be replaced by
   alternate copies are preserved in the directory "src.std"
   for safe-keeping.  When an alternate source file is used, it
   must be annotated in the footnotes section of the report.
   The asterisk (*) is not used to denote portability changes
   when alternate source files are employed.


   6.  Alternate_Results_Files

   On occasion, one platform will cause a variance in the
   results of the benchmark which may not necessarily be
   incorrect.  In such cases, an alternate results directory,
   called "result.alt", contains additional valid results.
   Only results that have been reviewed and approved by the
   SPEC Steering Committee are placed in this directory.
   Otherwise, all results must compare with the files in the
   "result.ref" directory.  In such cases, the M.vendor
   makefile must be modified to accommodate alternate results.
   An example of such a case can be seen with the benchmark
   085.gcc.  In this program, the output is sorted by a system
   library routine. As more vendors ran 085.gcc, it became
   apparent that the sort routine produced different orderings
   among systems.  Though the results were still equivalent,
   they could not be compared directly with the results.ref
   files, hence, the addition of alternate results became
   necessary.  If alternate results are used, they must be
   specified in the footnotes section of the report.

















                                          SPEC



   subject: SPEC Reporting Rules For      date: January 24, 1992
            CINT92
                                          from: SPEC Steering Committee



                             ABSTRACT



   This memorandum describes the Rules for reporting the
   results of SPEC CINT92 suite.  This is a companion document
   to SPEC Run Rules, which describes the process of running
   the benchmarks contained in the SPEC Suite.




   Memorandum to
   Performance Analysts










































                                          SPEC



   subject: SPEC Reporting Rules For      date: January 24, 1992
            CINT92
                                          from: SPEC Steering Committee



   1.  SPEC_CINT92_Introduction

   The SPEC CINT92 benchmark suite consists of CPU intensive
   benchmarks that are intended to be meaningful samples of
   applications which perform non-floating point logic and
   computations in a technical computing environment. SPEC
   CINT92 does not assess the ability of a system under test to
   handle disk, graphics, communication, or multi-user
   activity.

   Many of the SPEC CINT92 benchmarks have been derived from
   publicly-available application programs, and they are
   intended to be portable to as many current and future
   hardware platforms as possible.  Consequently, the version
   of the programs in this distribution contains no hardware
   dependencies.  Where possible, the programs have been
   structured to ensure that they do not unfairly favor one
   hardware architecture over another.  For this reason, the
   application programs in this distribution should not be used
   to assess the probable performance of commercially
   available, tuned versions of the same application.

   Different degrees of tuning and availability of specialized
   hardware features may make large (and varying) differences
   between the performance of the programs in this distribution
   and that of commercial applications for production use.

   The individual benchmarks in this suite may be similar, but
   not necessarily identical to benchmarks with the same name
   which are available from sources other than SPEC.  Therefore
   it may not be valid to compare the individual benchmark
   results of SPEC CINT92 to any benchmarks not obtained from
   SPEC.

   Reporting of results must follow the guidelines established
   in the "Reporting Guidelines" section of the distribution
   documentation. It is not intended that the SPEC CINT92
   benchmark suite be used as a replacement for the
   benchmarking of actual customer applications to determine
   vendor or product selection.
















                              - 2 -



   2.  REPORTING_GUIDELINES

   This section describes the standard reporting format that
   must be used when reporting results using the SPEC CINT92
   benchmarks to obtain a SPECint92 rating for a system. The
   SPECint92 rating is used to describe the speed of a system.

   EXAMPLES OF REPORTING RULES FOR SPEC RESULTS CAN BE SEEN IN
   THE QUARTERLY SPEC NEWSLETTER. 1 The intent of the SPECint92
   Reporting Guidelines is to describe a consistent method of
   reporting results to aid customers in interpreting and
   comparing results of different systems' performance.

   All measured and calculated values are reported to the
   nearest tenth.

   Unless noted to the contrary, the report must contain the
   following items of information:

   2.1  Table_of_Run_Times

   A table of 7 rows and 4 columns structured as follows:

     a.  One row for each individual benchmark in SPEC CINT92

     b.  Four columns


           i.  Name of benchmark (preferred order (numerical)
               shown in example below)

          ii.  SPEC Reference Time for each benchmark in the
               suite.

                 A.  008.espresso: 2270 seconds

                 B.  022.li: 6210 seconds

                 C.  023.eqntott: 1100 seconds

                 D.  026.compress: 2770 seconds

                 E.  072.sc: 4530 seconds



   __________

    1. Articles describing the suite and metrics can be found
       in the December 1991 SPEC Newsletter.












                              - 3 -



                 F.  085.gcc: 5460 seconds



         iii.  For SPECint92, Elapsed time (rounded to the
               nearest 0.1 second) for the System-Under-Test
               (SUT) taken to run each benchmark of SPEC CINT92
               suite.


               The elapsed times reported are for each
               benchmark run one at a time.  For the SPECint92,
               the elapsed time value measures the execution of
               a single copy of each benchmark.  The reported
               times are the best times that can be
               consistently reproduced.

               If any changes to an individual benchmark are
               made for portability purposes (e.g. to obtain a
               clean compile), the elapsed time should be
               followed by an "*", and a description of the
               change should be described in PART 7 of the
               report.  The asterisk is dropped after the
               portability change has been approved by the SPEC
               steering committee as a portability change and
               not a performance change and when this change
               has been implemented in the tape.  2

          iv.  For SPECint92, Column 4 has Ratio of SPEC
               Reference Time to SUT time (divide number in
               column 2 by number in column 3).  The ratio is
               without dimension, and will be referred to as
               SPECratio.


     c.  A row with the Geometric mean for the numbers listed
         in column 4.  This row is labeled as "Geom Mean:
         SPECint92".  The Geometric Mean is calculated as the
         6th root of the product of the individual values in
         the SPECratio column.

         For SPECint92, the Geometric Mean of the SPECratios in
         column 4 is defined as the SPECint92 of the SUT.

         Please note:  if an "*" sign appears in any of the


   __________

    2. REFER to portability rules in SPEC run rules document.












                              - 4 -



         rows to indicate a portability change in a benchmark,
         the "*" must be displayed after the Geometric Mean
         calculation for elapsed time and SPECratio.

   2.2  Identification_Information

   This information includes the date (month/year) that the
   benchmarks were run, the name and location of the company
   that ran them and the license number of the SPEC tape used.
   It is typically listed under the table generated in Section
   2.1.


   2.3  Hardware_Configuration

   A description of  the system (cpu/clock, fpu/clock) number
   of processors, relevant peripherals, etc are included in
   this space.  It is typically listed next to the table
   generated in Section 2.1.  The amount of information
   supplied should be sufficient to allow duplication of
   results by another party.  The following checklist is
   provided to show examples of hardware features which may
   affect the performance of SPEC CINT92 benchmarks.  The
   checklist is not intended to be all-inclusive, nor is each
   feature in the list required to be described.  The rule of
   thumb is: "if it affects performance or the feature is
   required to duplicate the results, describe it".

   HARDWARE PLATFORM FEATURES

      o Manufacturer's model number

      o CPU component ID and clock speed

      o Floating Point Unit (FPU) ID and clock speed

      o Number of CPUs and FPUs

      o Cache Size (per CPU), description and organization

      o Memory (Amount and description)

      o Disk subsystem configuration

           - Number of active connected disks

           - Disk controllers: ID, number and type

           - Disk: ID, number and type













                              - 5 -



      o Network Interface


   2.4  Software_Configuration


   The description of the software configuration used should be
   detailed enough so that another party could duplicate the
   results reported. This is typically reported after the
   hardware decription.  The following is a list of possible
   items that might be documented:

      o Operating System Type and Release level (with
        revision/update level if appropriate)

      o Compiler release levels used.  If multiple compilers
        are available, specify which ones were used.

      o Optional software required to reproduce results

      o File system type used

      o Firmware level


   2.5  Systems_Environment


   This section should document the overall systems environment
   used to run the benchmark. It is typically listed after the
   software description. The following is a list of possible
   items that might be documented:  SYSTEMS ENVIRONMENT
   CHECKLIST

      o Single or multi-user state

      o System tuning parameters (e.g. MAKEFS and TUNEFS
        parameters)

      o Process tuning parameters (e.g. stack size, time slice)

      o Background load, if any


   2.6  General_Availability_of_the_SUT


   This item indicates the availability (or projected
   availability) date for the system or components being
   benchmarked.  If both hardware and software are not
   available at the same time, list them separately. SPEC











                              - 6 -



   requires that the availability of hardware and software
   should be within 6 months of the date of publication of the
   data. It should be listed in Month/Year format.


   2.7  Individual_Benchmark_Notes


   This section documents ANY changes made to the individual
   benchmark source code or execution parameters for
   portability reasons.  For example, if difficulty arises in
   compiling or executing an individual benchmark because of
   portability problems, the difficulty should be described,
   and the code changes made to resolve the portability problem
   documented.  This should include module name, line number
   and actual source code changes.  This section must also
   include any use of alternate source files, alternate input
   files, or alternate result files, as noted in "Runrules"
   section of the distribution documentation.


   2.8  Graphical_Representation

   If desired, a graphical summary of the table from Section
   2.1 may be included as an additional item of information.

   If included for SPECint92, it should be shown as follows:

     a.  The 6 benchmarks comprising of  SPEC CINT92  suite
         should appear as individual points on the x-axis
         ordered by benchmark numbers.

     b.  The SPECratio of each benchmark (obtained from column
         4 in the table described above), should be plotted as
         a vertical bar graph.

     c.  A horizontal line should be drawn across the graph
         intersecting the y-axis at the SPECint92 geometric
         mean value.  This permits differentiation between each
         benchmark's SPECratio and the overall SPECint92
         geometric mean.



   3.  SAMPLE_REPORTS

















                              - 7 -



   3.1  SPECint92_-_Summary_of_SPEC_CINT92_Results

    __________________________________________________________
   | SPEC Benchmark|      SPEC    |  System Under|  SPECratio|
   |     CINT92    |   Reference  |      Test    |           |
   |               |  Time Elapsed|    (Sample)  |           |
   |               |      Time    |  Elapsed Time|           |
   |               |   (seconds)  |   (seconds)  |           |
   |_______________|______________|______________|___________|
   |_______________|______________|______________|___________|
   | 008.espresso  |      2270    |     1245.0   |    1.8    |
   |_______________|______________|______________|___________|
   | 022.li        |      6210    |     2275.0   |    2.7    |
   |_______________|______________|______________|___________|
   | 023.eqntott   |      1100    |      684.0   |    1.6    |
   |_______________|______________|______________|___________|
   | 026.compress  |      2770    |     1390.0   |    2.0    |
   |_______________|______________|______________|___________|
   | 072.sc        |      4530    |      783.0   |    5.8    |
   |_______________|______________|______________|___________|
   | 085.gcc       |      5460    |     1521.0   |    3.6*   |
   |_______________|______________|______________|___________|
   | Geometric Mean|              |              |    2.6*   |
   |_______________|______________|______________|___________|


   *Changes made to benchmark for portability reasons, see PART
   7 for details

   The geometric mean is the nth root of the product of n
   elements. Therefore, in this case it is the 6th root of the
   product of 6 SPECratios in the 4th column.


   3.2  Identification_Information

   Tested on mm/yy, tested by xxxxx of (location), SPEC License
   No. yyyyy.

   3.3  Hardware_Configuration

   Hardware Configuration Detail for "Sample" system

      o Vendor machine xxxxxxx, model number nnnnnnn

      o RISC processor xxxxxx; nn MHz clock speed

      o FPU xxxxxxxx, nn MHz clock speed

      o One processor/FPU












                              - 8 -



      o n MB main memory

      o n disk drives, each nnn megabytes capacity


   3.4  Software_Configuration

   Software Configuration Detail for "Sample" system

      o Operating System name, Version nnn, update nnn applied

      o FORTRAN Version nnn

      o C Version nnn

      o Other software:  (if applicable)

      o Firmware level:  (if applicable)



   3.5  General_Availability

   The harware wil be available mm/yy and the software will be
   available in nn/yy.

   Note: When a beta version is used, the date of the final
   release must be included which must be within 6 months of
   result publication.

   3.6  Systems_Environment

   Systems Environment Detail for "Sample" system

      o Tuning Parameters:

      o Background load:

      o System state:  (Mult-user or single user)


   3.7  Notes_On_Individual_Benchmark_Runs

      - 085.gcc: was compiled using gnu compiler * 085.gcc:
        changed variable name xxxx to yyyy to avoid library
        conflict.