Performance And Power Benchmarking

Performance And Power Benchmarking

اسلاید 1: 10/24/05 ELEC65001Performance And Power BenchmarkingKhushboo Sheth Department of Electrical and Computer Engineering

اسلاید 2: 210/24/05 ELEC6500PerformanceReducing Response Time (Execution Time)- the time between the start and the completion of a task. Total time required for the computer to complete a task, including disk accesses, memory accesses, I/O activities, operating systems overhead, CPU execution time, etc. Increasing Throughput-the total amount of work done in a given time.

اسلاید 3: 310/24/05 ELEC6500PerformancePerformance and Execution time relation for a computer X can be given as PerformanceX = 1 -------------- Execution time XIf comuter X is n times faster than computer Y then the execution time on Y is n times longer than it is on X :PerformanceX = Execution Time Y = n---------------- --------------------PerformanceY Execution Time X

اسلاید 4: 410/24/05 ELEC6500Execution TimeElapsed Time - total time to complete a task, including disk accesses, memory accesses, I/O activities, operating systems overhead, etc.CPU Time – the time the CPU spends computing for the task and does not include time spent waiting for I/O or running other programs ( response time = elapsed time not the CPU time )User CPU Time – the CPU time spent in the programSystem CPU Time – the CPU time spent in the operating system performing tasks on behalf of the program

اسلاید 5: 510/24/05 ELEC6500Computing CPU Execution TimeComputers are constructed using a clock that runs at a constant rate and determines when event take place in the hardware. These discrete time intervals are called clock cycles. Clock rate is the inverse of clock period.CPU Execution time = CPU clock cycles * Clock cycle for a program for a program timeCPU clock cycles = Instructions * Average clock cycles for a program per instruction CPU Time = Instruction count * CPI * Clock cycle timeSeconds = Instruction * Clock cycles * Seconds ---------- ------------ ------------- --------- Program Program Instruction Clock cycles

اسلاید 6: 610/24/05 ELEC6500Evaluating PerformanceThe computer may be evaluated using a set of BENCHMARKS – programs specifically chosen to measure the performance.The benchmarks form a workload that the user hopes will predict the performance of the actual workload.“Synthetic” benchmarks – specially created programs that impose the workload on the component“Application” benchmarks – run actual real-world programs on the system.Application Benchmarks usually give a much better measure of real world performance on a given system, synthetic benchmarks still have their use for testing out individual components like a hard disk or networking device.

اسلاید 7: 710/24/05 ELEC6500Types Of BenchmarksReal ProgramWord processing softwareTool software of CDAUser`s application software (MIS)KernelContains key codesNormally abstracted from actual programPopular kernel-Livermore loopLinpack benchmark (contains basic linear algebra subroutine written in FORTRAN language)Results are represented in MFLOPSToy BenchmarkUser can program it and use it to test computer`s basic components.

اسلاید 8: 810/24/05 ELEC6500Types of BenchmarksSynthetic BenchmarkProcedure for programming synthetic benchmarkTake statistics of all type of operations from plenty of application programsGet proportion of each operationWrite a program based on the proportion aboveIts results are represented in KWIPS (Kilo Whetstone Instructions Per Second). Not suitable for measuring pipeline computersTypes of Synthetic BenchmarksWhetstone – is a benchmark for evaluating the power of computers. It was first written in Algol60 at the National Physical Laboratory in the United Kingdom. It originally measured computing power in units of kilo-WIPS. Results for a variety of languages, compilers and system architectures have been obtained and modern workstations typically achieve more than 1,000,000 kWIPS. It primarily measures the floating point arithmetic performance.

اسلاید 9: 910/24/05 ELEC6500Types of Benchmarks Types of Synthetic BenchmarksDhrystone – is a benchmark invented in 1984 by Reinhold P. Weicker. It contains no floating point operations, thus the name is a pun on the then popular Whetstone benchmark for floating point operations. The o/p from the benchmark is the number of Dhrystones per second (the number of iterations of the main code loop per second). One common representation of the Dhrystone benchmark is the DMIP-Dhrystone MIPS-obtained when the Dhrystone score is divided by 1,757 (the number of Dhrystones per second obtained on the VAX 11/780, a 1 MIPS machine). The Dhrystone benchmark contains mainly integer and string operations. But like most synthetic benchmarks, the Dhrystone benchmark is not particularly useful in measuring the performance of real-world computer systems and has fallen into disuse replaced by benchmarks that more closely resemble typical actual usage.

اسلاید 10: 1010/24/05 ELEC6500SPECThe Standard Performance Evaluation Corporation (SPEC) is a non-profit organization that aims to produce fair, impartial and meaningful benchmarks for computers. SPEC was founded in 1988 and is financed by its member organizations which include all leading computer & software manufacturers. SPEC benchmarks are widely used today in evaluating the performance of computer systems.The benchmarks aims to test real-life situations. SPEC_WEB, for example, tests web servers performance by performing various types of parallel HTTP requests, and SPEC_CPU tests CPU performance by measuring the run time of several programs such as the compiler gcc and the chess program crafty. The various tasks are assigned weights based on their perceived importance; these weights are used to compute a single benchmark result in the end.SPEC benchmarks are written in a platform neutral programming language (usually C or FORTRAN) and the interested parties may compile the code using whatever compiler they prefer for their platform, but may not change the code. Manufacturers have been known to optimize their compilers to improve performance of the various SPEC benchmarks.

اسلاید 11: 1110/24/05 ELEC6500Various Current SPEC BenchmarksSPEC CPU2000,combined performance of CPU, memory and compilerCIN2000 (“SPECint”) ,testing integer arithmetic, with programs such as compilers, interpreters, word processors, chess programs, etc.CFP2000(“SPECfp”) , testing floating point performance, with physical simulations, 3D graphics, image processing, computational chemistry, etc.SPECWEB99, web server performance, measured by setting up a network of client machines that stress the server with parallel requests.SPEC HPC2002, testing high end parallel computing systems with applications such as weather prediction and computational chemistry.SPEC JVM98, performance of a java client server running a java virtual machine.SPEC MAIL2001, performance of a mail server, testing SMTP and POP protocolSPEC SFS97_R1, NFS file server throughput and response time.

اسلاید 12: 1210/24/05 ELEC6500Power BenchmarkingThe power benchmarking of a computer is fundamentally the notion of determining how much energy the computer is consuming in order to accomplish some measure of work.The BDTI (Berkeley Design Technology Inc.), EEMBC (EDN Embedded Microprocessor Benchmark Consortium), and SPEC (Standard Performance Evaluation Corp) benchmark organizations support benchmark suites that highlight a processors performance when performing application specific tasks. Researchers at BDTI and EEMBC are both working on how to extend their benchmark suites to measure and compare a processor’s energy efficiency as opposed to power consumption when performing application specific tasks.

اسلاید 13: 1310/24/05 ELEC6500Power Benchmark StrategyThere are 3 primary areas of interest when benchmarking the characteristics of “low power” systems employing power management techniques to achieve low power goals.First – actual power consumption of the system under typical user conditions, presumably under power management spectrum.Second – system operability or usability under power management conditions. Its clear that one could achieve remarkable power characteristics at the cost of system performance and the response time.Third – impact of power management techniques on system reliability.An appropriate benchmarking strategy for power managed systems must address these three areas in order to postulate an overall system figure merit, low power without sacrificing system operability or reliability. It would be one that characterizes the system power consumption while the system was carrying out some useful task.

اسلاید 14: 1410/24/05 ELEC6500Power BenchmarkingThe primary interest in power benchmarking is power consumed over the course of exercising a given application or in the case of multi-tasking environments, multiple applications running simultaneously. Specifically, what is the system power consumption as an application is exercised and the system transitions through various power managed power states. This is fundamentally a question of system expectations from both the application and end user’s perspective and how a power management facility might be able to exploit these expectations to reduce the system power consumption.If a specific system component isn’t being used and is unlikely to be used in the immediate time frame, its level of readiness might be compromised in order to reduce its and ultimately the system power consumption. A Word Processing application being used in EDIT mode might not access the system Fixed Disk for an extended period of time. The power management facility might recognize this as a flag that suggests that the Fixed Disk is unlikely to be called upon in the near time frame. Based on this determination the power management facility might exploit this system expectation as an opportunity to transition the fixed disk to a lower power state. This scenario could progress to a point when the fixed disk is actually completely powered off, its lowest power state and lowest state of readiness.

اسلاید 15: 1510/24/05 ELEC6500Power BenchmarkingThe energy consumption of a system is therefore, the aggregate power dissipated by its components over time, at varying power states. In terms of time , it is the energy required to execute a given task to completion. This can be reflected at the system level as the summation of the energy requirements of each subtask and can be computed by the following expression: m Pt = II (Pn) * Tc 1 ------ 3600 where, Pt = “Task Energy” in watt hours, WHrs. m = no. of power transitions occurring during the task Pn = Segmented power dissipation during a given power state Tc = “Task Cycle” time, time required to complete the task Pn = Tsn * Ps ------ Tc where, Tsn = time duration of the power state n, Ps = power level during the power state

اسلاید 16: 1610/24/05 ELEC6500Power BenchmarkingThis expression accurately reflects the energy consumed by a system during the execution of a task but does not reflect any notion of system operability, specifically the time spent to complete the task.It is unclear how to define and apply a consistent approach to measure energy efficiency and correlating it with a performance point. Both BDTI and EEMBC now propose that the core and local memories to a workload is sufficient, provided that proper disclosure of the testing configuration exists.Standard power and energy efficiency benchmarks are coming to fruition and a lot of opportunity exists for people to refine them. The importance of power benchmarks will continue to grow, especially because a growing number of processors have similar or identical core architectures. However, just like performance benchmarks, power benchmarks require developers to practice due diligence when mapping the benchmark data and testing configuration to their project’s requirements. Reference : David A. Patterson and John L. Hennessy, James W. Davis, EDNAsia.com