******************************************************************* ATM Forum Document Number: ATM_Forum/96-1173 ******************************************************************* Title: Fairness, Call Establishment Latency and Other Performance Metrics ******************************************************************* Abstract: This contribution defines four new performance metrics for ATM switches and networks. These are throughput fairness, frame loss ratio, maximum frame burst size, and call establishment latency. ******************************************************************* Source: Raj Jain, Gojko Babic, Bhavana Nagendra, Chi-Chung Lam The Ohio State University Department of CIS Columbus, OH 43210-1277 Phone: 614-292-3989, Fax: 614-292-2911, Email: Jain@ACM.Org The presentation of this contribution at the ATM Forum is sponsored by NASA. ******************************************************************* Date: August 1996 ******************************************************************* Distribution: ATM Forum Technical Working Group Members (AF-TEST) ******************************************************************* Notice: This contribution has been prepared to assist the ATM Forum. It is offered to the Forum as a basis for discussion and is not a binding proposal on the part of any of the contributing organizations. The statements are subject to change in form and content after further study. Specifically, the contributors reserve the right to add to, amend or modify the statements contained herein. ******************************************************************* 1. THROUGHPUT FAIRNESS: ---------------------- 1.1 Definition: Given n contenders for the resources, throughput fairness indicates how far the actual individual allocations are from the ideal allocations. In the most general case of a network, ideal allocation is defined by max-min allocation to various contending virtual circuits. For the simplest case of n VCs sharing a link with a total throughput T, the throughput of each VC should be T/n. If the actual measured throughputs of n VCs sharing a system (a single switch or a network of switches) are found to be {T1, T2, ..., Tn}, where the optimal max-min throughputs should be {T^1, T^2, ..., T^n}, then the fairness of the system under test is quantified by the "fairness index" computed as follows [1,3]: (sum(Xi))**2 Fairness Index = ------------ n*sum(Xi**2) Where, Xi=Ti/T^i is the relative allocation to ith VC. This Fairness Index has the following desirable properties: 1. It is dimensionless. The units used to measure the throughput (bits/sec, cells/sec, frames/sec) do not affect its value. 2. It is a normalized measure that ranges between zero and one. The maximum fairness is 100% and the minimum 0%. This makes it intuitive to interpret and present. 3. If all Xi's are equal, the allocation is fair and the fairness index is one. 4. If n-k of n Xi's are zero, while the remaining k Xis are equal and non- zero, the fairness index is k/n. Thus, a system which allocates all its capacity to 80% of VCs has a fairness index of 0.8 and so on. 1.2 Load Level and Traffic Pattern: Throughput fairness is quantified via the fairness index for each of the throughput experiments in which there are either multiple VCs or multiple input or output ports. Thus, it applies to all three throughput measures (lossless, peak, and full-load) and all four traffic patterns (n-to-n straight, n-to-n cross, n-to-1, and 1-to-n) described in Section 3 of baseline draft [2]. Note that in the case of n-to-n cross, there are n**2 VCs and, therefore, n**2 should be substituted in place of n in the fairness index. In the case of 1-to-n pattern, there is only one VC and all input is expected to be multicast to n output ports. The fairness will measure the equality of throughputs to the output ports. No additional experiments are required for throughput fairness. The detailed results obtained for the throughput tests are analyzed to compute the fairness. 1.3 Statistical Variation: The throughput tests are run NRT times for TRT seconds each. Recall that NRT and TRT are parameters. The fairness is computed for each individual run. Let Fi be the fairness for the ith run, then the mean fairness is computed as follows: Mean Fairness = sum(Fi)/NRT 1.4 Background Traffic: The throughput tests are conducted with and without background traffic. Higher priority VBR traffic can act as background traffic. Further details for measurements with background traffic (multiple service classes simultaneously) are to be specified. Until then all performance testing will be done without any background traffic. 1.5 Reporting Results: The fairness index values are reported for each of the throughput experiments in the tabular format specified in Table 3.1 of the current baseline [2]. Additional columns are added for fairness next to "Std Err" columns for each of the three throughput measures. Note that fairness index is not limited to throughput. It can be applied to other metrics, such as latency. However, extreme unfairness in latency is expected show up as unfairness in throughput and vice versa. Therefore, it is not required to quantify fairness of latency. 2. FRAME LOSS RATIO: ------------------ 2.1 Definition: Frame loss ratio is defined as the percentage of frames that are not forwarded by a system under test (SUT) due to lack of resources. Frame loss ratio = 100*(Input frame count - output frame count)/(input frame count) There are two frame loss ratio metrics that are of interest to a user. i. Peak-throughput frame loss ratio - It is the frame loss ratio at a frame load for the peak throughput. ii. Full-load frame loss ratio - It is the frame loss ratio at a frame load for the full-load throughput. These metrics are related to the throughput: Frame Loss Ratio = (Input Rate - Throughput)/Input Rate Thus, no additional experiments are required for frame loss ratios. These can be derived from tests performed for throughput measurements provided the input rates are recorded. 2.2 Unit The frame loss ratio is expressed as a percentage of input frames. 2.3 Traffic Patterns FLRs are measured for each of the four traffic patterns (n-to-n straight, n-to-n cross, n-to-1, and 1-to-n) specified for throughput measurements in Section 3.1.4 of baseline draft [2]. 2.4 Statistical Variation: The throughput experiments are repeated NRT times for TRT seconds each. Here, NRT and TRT are parameters. If FLR_i is the frame loss ratio for the ith run: Input Rate_i - Throughput_i Frame Loss Ratio FLR_i = --------------------------- Input Rate_i Since frame loss ratio is a "ratio," its average cannot be computed via straight summation [1]. The average average frame loss ratio for NRT runs is computed as follows: Sum(Input Rate_i) - Sum(Throughput_i) Average Frame Loss Ratio FLR = ------------------------------------- Sum(Input Rate_i) The average is reported as the FLR for the experiment. 2.5 Reporting Results FLR values are reported for peak throughput and full-load throughput experiments in the tabular format specified in Table 3.1 of the current baseline [2]. Additional columns are added for FLR next to fairness columns for each of the above two throughput measures. 3. MAXIMUM FRAME BURST SIZE (MFBS): ---------------------------------- 3.1 Definition: Maximum Frame Burst Size (MFBS) is the maximum number of frames that source end systems can send at the peak rate through a system under test without incurring any loss. MFBS measures the data buffering capability of the SUT and its ability to handle back-to-back frames. Many applications and transport layer protocol drivers often present a burst of frames to AAL for transmission. For such applications, Maximum Frame Burst Size provides an useful indication. This metric is particularly relevant to UBR service category since the UBR sources are always allowed to send a burst at peak rate. ABR sources may be throttled down to a lower rate if a switch runs out of buffer. 3.2 Units MFBS should be expressed in octets of AAL payload field. This is preferred over number of frames or cells. The former requires specifying the frame size and the latter is not very meaningful for a frame-level metric. Also, number of cells has to be converted to octets for use by AAL users. It may be useful to indicate the frame size for which MFBS has been measured. If MFBS is found to be highly variable with frame size, a number of common AAL payload field sizes such as 64 octets, 536 octets, 1518 octets, and 9188 octets may be used (exact sizes to be specified). 3.3 Statistical Variations The number of frames sent in the burst is increased successively until a loss is observed on any VC. The maximum number of frames that can be sent without loss are reported as MFBS. The tests should be repeated NRT times. The average of NRT repetitions is reported as the MFBS for the system under test. 3.4 Traffic Patterns: The MFBS is measured for n-to-1 traffic pattern specified in Section 3.1.4 (of baseline draft). Optionally, it can be measured for other traffic patterns also. The value obtained for n-to-1 pattern is expected to be smaller than that for other patterns. 4. CALL ESTABLISHMENT LATENCY: ----------------------------- 4.1 Definition: For short duration VCs, call establishment latency is an important part of the user perceived performance. Informally, the time between submission of a call setup request to a network and the receipt of the connect message from the network is defined as the call establishement latency. The time lost at the destination while the destination was deciding whether to accept the call is not under network control and is, therefore, not included in call setup latency (See Figure 1). Src. Sw. Sw. Dest. | | | | |\ | | | | \ | | | | \|_______|_____|_____________ | |\ | | | | \Setup | Latency for Setup Message | | \ | | | | \|_____|_____________ | | |\ | | | | \ | | | | \| | | | | | | | | | | | /| | | | / | | | |/____|____________ | | /| | | | / | | Latency for Connect Message | | / | | | |/______|_____|____________ | /|Connect| | | / | | | |/ | | | | | | | Figure 1: Call establishment Thus, the sum of the latency experienced by the setup message and the resulting connect message is the call setup latency. The main problem in measuring these latencies is that both these messages span multiple cells with intervening idle cells. Unlike previous X.25, frame relay, and ISDN networks, the messages in ATM networks are not contiguous. Therefore, the MIMO latency metric defined in Section 3.2 (of baseline draft [2]) is used. Thus, Call Establishment Latency = MIMO Latency for SETUP message + MIMO latency for the corresponding Connect message Recall that the MIMO latency for a frame is defined as the minimum of last-bit-in-to-last-bit-out (LILO) and the difference of first-bit-in-to- last-bit-out (FILO) and normal frame output time (NFOT). MIMO Latency = Min{LILO, FILO-NFOT} 4.2 Units Call establishment latency is measured in units of time. 4.3 Configurations: The call establishment latency as defined above applies to any network of switches. In practice, it has been found that the latency depends upon the number of switches and the number of PNNI group hierarchies traversed by the call. It is expected that measurements will be conducted on multiple switches connected in a variety of ways. In all cases, the number of switches and number of PNNI group hierarchies traversed will be indicated. The simplest configuration is that of a single switch connecting both the source and the destination end systems. 4.4 Statistical Variations The latency measurement is repeated NRT times. Each time a different node pair is selected randomly as the source and destination end system. The average and standard error of NRT such measurements is reported. For a single n-port switch it is expected that all n ports are equally probable candidates to be source and destination end-system. MOTION: ------ Adopt the text of this contribution (96-1372) for inclusion in the baseline draft. References: ---------- [1] R. Jain, "The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling ," Wiley- Interscience, New York, NY, April 1991. [2] R. Jain and B. Nagendra, "ATM Forum Performance Testing Specification: Baseline Text," ATM_Forum/96-0810R1, August 1996. [3] R. Jain, D. Chiu, and W. Hawe, "A Quantative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer Systems," DEC Technical Report, DEC-TR-301, September 1984, 38 pp. Available as http://www.cse.wustl.edu/~jain/papers/fairness.ps All our papers and ATM Forum contributions are available on-line: http://www.cse.wustl.edu/~jain/