************************************************************************ ATM Forum Document Number: ATM Forum/95-1662 ************************************************************************ Title: Performance Benchmarking of ATM Switches ************************************************************************ Abstract: This contribution proposes several performance metrics for comparing ATM switches. This is the first such contribution in response to the decision in the October 1995 meeting of the Test group, where it was decided that benchmarking will be addressed by the group. The goal of this contribution is to begin the discussion. Since the metrics to be addressed include several related to traffic management, it is being distributed to TM group as well. ************************************************************************ Source: Raj Jain, Bhavana Nagendra The Ohio State University Department of CIS Columbus, OH 43210-1277 Phone: 614-292-3989, Fax: 614-292-2911, Email: Jain@ACM.Org The presentation of this contribution at ATM Forum is sponsored by NASA. ************************************************************************ Date: December 9-15, 1995 ************************************************************************ Distribution: ATM Forum Technical Working Group Members (Testing Group and Traffic Management) ************************************************************************ Notice: This contribution has been prepared to assist the ATM Forum. It is offered to the Forum as a basis for discussion and is not a binding proposal on the part of any of the contributing organizations. The statements are subject to change in form and content after further study. Specifically, the contributors reserve the right to add to, amend or modify the statements contained herein. ************************************************************************ SUMMARY OF OCTOBER 1995 DISCUSSION: Performance benchmarking is related to user perceived performance of ATM technology. For the success of ATM technology, it is important that the performance of existing and new applications be better than that on other competing networking technology. In other words, goodness of ATM will not be measured by cell level performance but by performance perceived at higher layers. Most of the Quality of Service (QoS) metrics, such as cell transfer delay (CTD), cell delay variation (CDV), cell loss ratio (CLR), and so on, may or may not be reflected directly in the performance perceived by the user. For example, while comparing two switches if one gives a CLR of 0.1% and a frame loss ratio of 0.1% while the other gives a CLR 1% but a frame loss ratio of only 0.05%, the second switch will be considered superior by many users. ATM Forum and ITU have standardized the definitions of QoS metrics. We need to do the same for higher level performance metrics. Without a standard definition, each vendor will use their own definition of common metrics such as throughput and latency resulting in a confusion in the market place. Avoiding such a confusion will help buyers eventually leading to better sales resulting in the success of the ATM technology. Based on these thoughts, the testing working group, in its October 1995 meeting at Honolulu, decided to add performance benchmarking, as a work item to its agenda [Jain]. INTRODUCTION In order to start the discussion on performance benchmarking, we are offering this contribution on benchmarking of switches. We hope that this will encourage others to bring in differing thoughts on this and other related topics such as benchmarking of network interface cards (NICs) and other interconnecting devices. In this proposal, we have gathered ideas from several sources including those from the Internet Engineering Task Force (IETF) benchmarking group [Bradner,RFC1242] and a few published papers on measured performance of switches [See bibliography]. The key goals of this proposal are: 1. To extend the benchmarking to all classes of service. Many past performance measurements concentrated on CBR service. We need to extend those to real time VBR, non-realtime VBR, ABR, and UBR. 2. To emphasize the end-user viewpoint where-ever possible. For example, data traffic may use UBR or ABR service class. Some ATM networks (switches) may offer one or both classes. The user may care more for the application throughput rather than the underlying mechanism used. The performance is, therefore, measured on several alternative protocol stacks. 3. To emphasize frame level metrics rather than cell level metrics. Most of the past measurements have concentrated on cell level metrics. 4. To consider performance of network management, connection setup, along with normal data transfer. Most of the past measurements have emphasized cell transfer throughput traffic. RESTRICTIONS: This being the first proposal, is limited in several respects: 1. This particular proposal concentrates on the data traffic (ABR and UBR service classes) since that is expected to be the bulk of traffic on ATM networks initially. Other service classes will be added later. 2. Only performance metrics are discussed here. For benchmarking, we also need to identify test configurations, traffic patterns (application behaviors), and applications. These issues will be addressed later. For example, user applications may include remote disk server using protocols like NFS, remote disk backup systems like RDUMP, remote tape access systems, TELNET, FTP and WEB. 3. The performance can be measured at several layers (above ATM layer), for example, network (e.g., IP), transport (e.g., TCP), application (e.g., FTP). At each layer, several alternative stacks are possible. For example, IP can use "Classical IP over ATM" (RFC 1577) or "LAN Emulation (LANE)." At this stage we are not limiting to any particular layer or stack and are defining terms that are applicable to multiple layers and multiple stacks. +-------------------+<---+ | USER LEVEL | | | APPLICATION (FTP) | | |--------+----------|<---| | TCP | UDP | | |--------+----------|<---|---- User perceived performance | IP | | |-----------+-------|<---| | RFC 1577 | LANE | | |-----------+-------|<---+ | AAL5 | |-------------------| | ABR | UBR | | ATM | |-------------------| | PHY | +-------------------+ Figure 1 - Examples of measurement alternatives ------------------------------------------------ 4. Test Configurations: Although, we'll be using the following configuration as an aid to explain the performance metrics, more thought needs to be given to develop exact test configuration. +-----+ +-----+ | | | | | H11 |------+ +------| H21 | | | | | | | +-----+ | | +-----+ | +---------+ +---------+ | | | | | | | +-----+ | | ATM | ATM LINK | ATM | | +-----+ | | | | | | | | | | | H12 |------+--+ SWITCH +==========+ SWITCH +--+------| H22 | | | | | | | | | | | +-----+ | | (S1) | | (S2) | | +-----+ | | | | | | | +---------+ +---------+ | | | +-----+ | | +-----+ | | | | | | | H13 |------+ +------| H23 | | | | | +-----+ +-----+ ETHERNET LAN CONNECTION ETHERNET LAN CONNECTION Figure 2 - A Sample Test Configuration -------------------------------------- The configuration consists of two ATM switches connected by an ATM link (155 Mbps). Hosts are connected to each of the switches and could represent an Ethernet LAN. The traffic generated will be bidirectional data traffic and specifically ABR and UBR will be used to create a realistic situation. PERFORMANCE METRICS: We propose that the metrics be grouped as follows: - General metrics - Protocol-Stack specific metrics - Traffic Management metrics - Network Management metrics General Performance Metrics : These metrics apply to most ATM networks and are not protocol specific. The tests for these metrics effectively characterize the basic features of the switch. Protocol-Stack Specific Metrics : These metrics apply to particular protocol stacks and need only be measured and tested if particular protocols are being used. Examples, of such protocols are RFC1577 and LANE, as discussed earlier. Traffic Management Metrics: These measure ability of the switches to avoid overload and to efficiently and fairly resolve contention among various VCs when there is overload. Network Management Metrics : These metrics are defined to aid characterization of the switch in responding to network management requests. Some of the discussion below is from RFC 1242 and its current version (an internet draft) [Bradner]. We are of course, open to comments, suggestions, and discussion, about applicability (or nonapplicability) of these metrics to ATM technology. GENERAL PERFORMANCE METRICS 1. Throughput : The maximum rate at which none of the frames are dropped by the ATM switch is the throughput without loss. Essentially we are looking at the behavior of a perfect switch which works with an efficiency of 100%. Data traffic (ABR/UBR) is passed through the switch and then the frames that are transmitted by the switch are counted. The load can be varied and efficiency calculated at each load. If the input and the output count are the same then the load is increased and the test is conducted again. The throughput without loss is the highest load at which the count of the output frames equals the count of the input frames. A graph of load vs throughput can be shown. Instead, the load can be kept constant and the frame size can be varied and its effect on the throughput can be studied. % Throughput = (Output count/ Input count) * 100 A model graph of load vs throughput would be: ^ | # # | # # % THROUGHPUT |--------- # | # | # | # | | # | # | # |<---- 0% loss | # | | # | | # | | # | | # | +-----------------------------------> LOAD Figure 3 - Graph of % throughput vs load ---------------------------------------- 2. Frame loss rate : Percentage of frames that should have been forwarded by the switch under steady state traffic that were not forwarded due to lack of resourses. This measurement reports the performance of the switch at an overloaded state. The device might lose frames that contain routing information and this may further reduce the performance as more frames need to be retrasmitted. The frame errors could be CRC errors and/ or cell termination errors. Frame loss rate = ((input_count - output_count) * 100) / input_count The first trial should be run at the load that corresponds to 100% of the maximum rate for the frame size. The load is progressively decreased by 10% until there are two successive trials with no frame loss. The results of the frame loss test should be reported as a graph of % loss vs load. 3. Back-to-Back Burst Size: Fixed length frames presented at a rate such that there is the minimum legal separation between frames over a short to medium period of time, starting from an idle state. This determines buffering capabilities of the ATM switch in hand. NFS, remote disk backup systems like rdump, and remote tape access systems, can be configured such that a single request can result in a block of data being returned, as much as 64K octets. A burst of frames with minimum inter-frame gaps is sent to the switch and the number of frames that have been forwarded by the switch is counted. If the throughput is 100% then the length of the burst is increased and the test is rerun. The back-to-back value is the longest burst that the device will handle without the loss of any frames. Measures the extent of data buffering in the switch. The average frame size count and standard deviation (optional) would be reported for each frame size tested. 4. Latency : The time interval starting from when the last bit of the input frame reaches the input port and ending when the first bit of the output frame is seen on the output port. This is valid as most connecting equipment store the data till the whole frame is received. But for cut-through devices, although this metric may be negative in some cases, it could still be considered as a store and forward device and the latency measured from last bit in to the first bit out. This helps in treating the devices uniformly and not be bothered by the internal architecture. After the throughput of the switch has been determined for a number of frame sizes, a stream of frames at a particular frame size is passed through the device at the determined throughput rate to a specific destination. The time at which the frame is fully transmitted is recorded (timestamp A). The receiver logic in the test equipment should be able to the tag information in the frame stream and record the time at which the entire tagged frame was received (timestamp B). Latency = Timestamp B - Timestamp A The reporting format would be rate and resultant latency for each frame size. 5. Call establishment time : This is the time taken to setup a connetion with the destination by the calling party. For short duration VCs, call establishment time is an important part of the user perceived performance. The time between the submission of a "call request" and the reception of the corresponding "ready indication" is defined as the call establishment time. The call establishment time may depend upon host processors, NICs, and other traffic on the link. The issue of what background load in the switch should be assumed and how to separate the switches contribution from that of other components remains to be discussed. TRAFFIC MANAGEMENT METRICS: 1. Load Control Latency: A set of VCs are established. After the system reaches the steady state, the load on one VC is suddenly increased, the time for the system to reach the steady state again is measured. Similarly, when the load is decreased, the time to reach steady state is measured. 2. Burst Throughput: Frames are sent at differing burst (frame burst) sizes and the steady state throughput is measured. Depending upon the underlying service class (UBR, ABR), the bursty performance may be different than steady state performance. This is particularly important for request-response (client-server) applications. 3. Throughput in the Presence of Higher Priority Traffic: The throughput of ABR traffic is measured when a VBR VC shares the path with data traffic. The characteristics of the VBR traffic need to be clearly specified. 4. Fairness: A configuration similar to "GFC" configuration used in early versions of traffic management document can be used to measure the fairness of various switches. NETWORK MANAGEMENT METRICS: [To be discussed] APPLICATION SPECIFIC PERFORMANCE METRICS: [To be discussed] BIBLIOGRAPHY: [Bradner] Scott Bradner, "Benchmarking Methodology for Network Interconnect Devices", Internet Draft. [Mandeville] Robert Mandeville, European Network Laboratories, Data Comm Magazine, March 1995, p 69. [Wakid] Wakid ey al, "Architectures for BISDN Networks : A Performance Study", Advanced Systems Divison, National Institute of Standards and Technology, (301)-975-4855, http://www.hpcc.gov/blue94/section.4.7.html [LANQuest] ATM Cell Congestion Loss Across Switch (CCLAS) Throughput Analysis, LANQuest Labs, (408) 894-1000. [RFC1577] M. Laubach, Classical IP and ARP over ATM, RFC 1577, Jan 1994 [SNCI] Scott Bradner, "The 1995 Ethernet to ATM Evaluation", SNCI [Mier] Mier and Smithers, "ATM to the Desktop", Product Testing, Communications Week, September 25, 1995 [RFC1242] Scott Bradner, "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, July 1991. [Rowe] Martin Rowe, "Wealth of ATM Testers - Answers Most Needs", Test and Measurement World, Sept 1995, p 55 [Krivda] Cheryl D. Krivda, "Analyzing ATM Adapter Performance: The Real-World Meaning of Benchmarks," http://www.efficient.com/dox/EM.html [Jain] Raj Jain, "Performance Benchmarking BOF," AF-ALL/95-1347, October 1995. Note: All our past ATM forum contributions and presentations are available on-line: http://www.cse.wustl.edu/~jain/