The Discrete Event System Specification (DEVS) formal modeling and simulation (M&S) framework (which supports hierarchical and modular model composition) has been widely used to understand, analyze and develop a variety of systems. Numerous DEVS simulators have been developed; nevertheless, evaluating the performance of such simulators is a complex process and it has been usually done using ad hoc methods. DEVStone, instead, is a synthetic benchmark that can be used to automate the evaluation of the performance of DEVS-based simulators. DEVStone generates a suite of models with varied structure and behavior automatically. To do so, it uses a standardized mechanism that can be the basis for comparisons between simulation software tools. As a proof of the concept, we present various tests in which DEVStone was used to study the efficiency of five different simulation engines. In this case, we compared various versions of the CD++ simulator, and then compared its performance with the ‘A Discrete Event System Simulator’ (ADEVS) M&S tool. This is the first effort in which these simulation tools have been thoroughly compared with a very demanding set of experiments. The use of DEVStone allowed a standardized and exhaustive method to compare different features of the simulation software. We show how the basic ideas used for DEVStone facilitates performance analysis for upgrades and updates of a given simulation engine, while also providing a common metric to compare different M&S environments.
Zeigler B., Kim T. and Praehofer H.Theory of Modeling and Simulation, 2nd edn. New York: Academic Press, 2000.
2.
El-Osery A., Burge J., Jamshidi M., Fathi M. and Akbarzadeh M-R.V-LAB: A virtual laboratory for autonomous agents - SLA based controllers . IEEE Syst Man Cybernet2002; 32: 791-803.
3.
Barros F. and Ball GLFire modeling using dynamic structure cellular automata. In: III International Conference on Forest Fire Research, III International Conference On Forest Fire Research, 14th Conference on Fire and Forest Meteorology , 1998, Vol. I, 879-888.
4.
Liu Q. and Wainer G.Parallel environment for DEVS and Cell-DEVS models. SIMULATION2007; 83: 449-471.
5.
Filippi J-B. , Morandini F., Balbi J. and Hill DRC.SIMULATION2010; 86: 629-646.
6.
Wainer G.Applying Cell-DEVS methodology for modeling the environment. SIMULATION2006; 82: 635-660.
7.
Cunning SJ, Schulz S. and Rozenblit JWAn embedded system’s design verification using object-oriented simulation . SIMULATION1999; 72: 238-249.
8.
Kim J-K., Kim YG and Kim TGDHMIF: DEVS-based hardware model interchange format. In: Proceedings of the European Simulation Symposium, Marseille , France, 2001.
9.
Lee J. and Chi S.Using symbolic DEVS simulation to generate optimal traffic signal timings . SIMULATION2005; 81: 153-170.
10.
Wainer G.Developing a software tool for urban traffic modeling. Software Pract Exper2007; 37: 1377-1404.
11.
Gambardella LM , Rizzoli AE and Zaffalon M.Simulation and planning of an intermodal container terminal. SIMULATION1998; 71: 107-116.
12.
Pérez E., Ntaimo L., Bailey C. and McCormack P.Modeling and simulation of nuclear medicine patient service management in DEVS. SIMULATION2010; 86: 481-501.
13.
Byon E., Pérez E., Ding Y. and Ntaimo L.Simulation of wind farm operations and maintenance using discrete event system specification. SIMULATION2010 ; in press.
14.
Daicz S., Troccoli A. and Wainer G.Experiences in modeling and simulation of computer architectures using DEVS . SIMULATION2001; 18: 179-202.
15.
Chen Y. and Sarjoughian HSA component-based simulator for MIPS32 processors. SIMULATION2010; 86: 271-290.
16.
Zeigler BP, Song H., Kim T. and Praehofer H.DEVS framework for modeling, simulation, analysis, and design of hybrid systems . In: Hybrid Systems II (Lecture Notes in Computer Science . Vol. 999), Berlin : Springer, 1995, 529-551.
17.
Glinsky E. and Wainer G.Performance analysis of real-time DEVS models. In: Proceedings of the SCS Winter Simulation Conference, San Diego, CA, 2002.
18.
Wainer G.CD++: A toolkit to develop DEVS models. Software Pract Exper2002; 32: 1261-1306.
19.
Nutaro J.ADEVS website, http://www.ornl.gov/∼1qn/adevs/index.html (accessed 1 October 2010).
20.
Wainer G., Jafer S. and Liu Q.Advanced parallel simulation of DEVS models in CD++. In: Wainer G and Mosterman P (eds) Discrete-Event Modeling and Simulation: Theory and Applications. London: Taylor and Francis, 2010.
21.
Moallemi M. and Wainer G.Designing an interface for real-time and embedded DEVS. In: Proceedings of 2010 Symposium on Theory of Modeling and Simulation, DEVS’10, Orlando, FL, 2010.
22.
Díaz A. , Vázquez V. and Wainer G.Application of the ATLAS language in models of urban traffic. In: Proceedings of the Annual Simulation Symposium, Seattle, WA, 2001 .
23.
Ameghino J. , Troccoli A. and Wainer G.Models of complex physical systems using Cell-DEVS. In: Proceedings of the 34rd Annual Simulation Symposium, Seattle, WA. 2001 .
24.
Ameghino J., Wainer G. and Glinsky E.Applying Cell-DEVS in models of complex systems. In: Proceedings of the SCS Summer Computer Simulation Conference, Montreal, Canada, 2003.
25.
Wainer G.Discrete-Event Modeling and Simulation: A Practitioner’s Approach . Boca raton, FL: CRC Press, 2009.
26.
Mittal S., Risco-Martín JL and Zeigler BPDEVS/SOA: A cross-platform framework for net-centric modeling and simulation in DEVSUnified process. SIMULATION2009; 85: 419-450.
27.
Sarjoughian HS and Zeigler BPDEVSJAVA: Basis for a DEVS-based collaborative M&S environment . In: Proceedings of the SCS International Conference on Web-based Modeling and Simulation, San Diego, CA, 1998.
28.
Kim TGDEVSim++: C++ based simulation with hierarchical modular DEVS models. User’s Manual CORE Lab. Taejon, Korea: EE Dept, KAIST, 1994.
29.
Himmelspach J. , Ewald R. and Uhrmacher AMA flexible and scalable experimentation layer for JAMES II. In: Proceedings of the Winter Simulation Conference, Miami, FL. 2008.
30.
Filippi J-B. , Bernardi F. and Delhom M.The JDEVS environmental modeling and simulation environment. In: Proceedings of the the IEMSS’02 Conference on Integrated Assessment and Decision Support, Lugano, Switzerland, 2002.
31.
Wainer G., Al-Zoubi K., Mittal S., Risco-Martín JL, Sarjoughian H. and Zeigler BPAn Introduction to DEVS standardization. In: Wainer G and Mosterman P (eds) Discrete-Event Modeling and Simulation: Theory and Applications. London: Taylor and Francis, 2010.
32.
Barros FJModeling and Simulation of dynamic structure heterogeneous flow systems . SIMULATION2002; 78: 18-27.
33.
Sun Y. and Hu X.Performance measurement of dynamic structure DEVS for large-scale cellular space models. SIMULATION2009; 85: 335-351.
34.
Copp JBThe COST Simulation Benchmark. Description and Simulator Manual. Luxembourg: Office for Official Publications of the European Communities.
35.
Knight P., Corder A., Liedel J., Gidens J., Drake R., Jenkins C., et al. Evaluation of run time infrastructure (RTI) Implementations . In: The Huntsville Simulation Conference, Huntsville, AL, 2002.
36.
Troccoli A. and Wainer G.Performance analysis of cellular models with parallel Cell-DEVS. In: Proceedings of the SCS Summer Computer Simulation Conference, Florida, 2001.
37.
Breitenecker F., Wassertheurer S., Popper N. and Zauner G.Benchmarking of simulation systems-the ARGESIM comparisons. In: First Asia International Conference on Modeling and Simulation, Pukhet, Thailand, 2007.
38.
Weicker RPDhrystone: A synthetic systems programming benchmark. Communications of the ACM1984; 27: 1013-1030.
39.
Chow A. and Ziegler B.Parallel DEVS: A parallel, hierarchical, modular modeling formalism . In: Proceedings of the SCS Winter Simulation Conference , Orlando, FL, 1994.
40.
Martin D., McBrayer T., Radhakrishan R. and Wilsey P.Time Warp Parallel Discrete Event Simulator. Technical report, Computer Architecture Design Laboratory, University of Cincinnati , 1997.
41.
Kim K., Kang W., Sagong B. and Seo H.Efficient distributed simulation of hierarchical DEVS models: transforming model structure into a non-hierarchical one. In: Proceedings of the 33rd Annual Simulation Symposium, Washington, DC, 2000.
42.
Liu J.Real-time Systems.Englewood Cliffs, NJPrentice-Hall, 2000.
