************************************************************************

       ATM Forum Document Number: ATM Forum/95-1347

       ************************************************************************

       Title: Performance Benchmarking BOF

       ************************************************************************

       Abstract:

       We would like to form a new "birds of a feather" group on Performance
       Benchmarking of ATM products.

       ************************************************************************

       Source:

       Raj Jain
       The Ohio State University
       Department of CIS
       Columbus, OH 43210-1277
       Phone: 614-292-3989, Fax: 614-292-2911, Email: Jain@ACM.Org

       The presentation of this contribution at the ATM Forum is sponsored by
       NASA.

       ************************************************************************

       Date: October 1995, Honolulu

       ************************************************************************

       Distribution: ATM Forum Technical Working Group Members
       (Plenary, Traffic Management, and Testing)

       ************************************************************************

       Notice: This contribution has been prepared to assist the ATM
       Forum. It is offered to the Forum as a basis for discussion and
       is not a binding proposal on the part of any of the contributing
       organizations. The statements are subject to change in form and
       content after further study. Specifically, the contributors
       reserve the right to add to, amend or modify the statements
       contained herein.

       ************************************************************************


       Performance benchmarking is the issue of developing (or agreeing
       on) representative workloads for comparing the performance of
       various ATM Switches, NICs, and other products. While this area
       is somewhat related to testing and QoS (and therefore to traffic
       management).  None of the two group are working on performance
       benchmarking.

       Testing is mostly concerned with "Conformance" which is different
       from "performance." Although, there is a section on "performance"
       in the testing document, it does not deal with the benchmarking
       issue.

       Traffic management document specifies several QoS metric but many
       of these metrics may or may not be what the user eventually may
       use to compare different products alternatives. For example, a
       user running an ATM application may simply see increased response
       time when a switch has a high "Cell misinsertion ratio." Thus,
       the user may simply want to use the response time for a
       particular benchmark as the goodness metric for different
       switches.

       The goal of the performance benchmarking BOF would be to identify
       workloads and metrics for various ATM products that a user can
       justifiably use to compare various products. Although, it is upto
       the group to decide how to proceed on this, we would probably
       start with ATM Switches as the first set of products to work on.
       An industry-wide agreement about what are meaningful workloads
       and metric would certainly help the user and the industry.

       A good example of performance benchmarking group from TCP/IP
       community is the "Benchmarking Working Group" of the Internet
       Engineering Task Force (IETF). They started back in 1991 and have
       developed a set of metrics for comparing routers and now several
       LAN switching products. The metrics and workloads are used by
       Scott Bradner at Harvard University to publish performance
       results on various router and LAN switching products.  The
       networking industry also uses these same benchmarks, metrics, and
       results to compare their products. We, at ATM Forum, need to
       develop a similar set of metrics and benchmark for ATM products.