World Scientific
  • Search
  •   
Skip main navigation

Cookies Notification

We use cookies on this site to enhance your user experience. By continuing to browse the site, you consent to the use of our cookies. Learn More
×
Our website is made possible by displaying certain online content using javascript.
In order to view the full content, please disable your ad blocker or whitelist our website www.worldscientific.com.

System Upgrade on Tue, Oct 25th, 2022 at 2am (EDT)

Existing users will be able to log into the site and access content. However, E-commerce and registration of new users may not be available for up to 12 hours.
For online purchase, please visit us again. Contact us at [email protected] for any enquiries.

DEVELOPMENT OF SCIENTIFIC SOFTWARE: A SYSTEMATIC MAPPING, A BIBLIOMETRICS STUDY, AND A PAPER REPOSITORY

    https://doi.org/10.1142/S0218194013500137Cited by:26 (Source: Crossref)

    Scientific and engineering research is heavily dependent on effective development and use of software artifacts. Many of these artifacts are produced by the scientists themselves, rather than by trained software engineers. To address the challenges in this area, a research community often referred to as "Development of Scientific Software" has emerged in the last few decades. As this research area has matured, there has been a sharp increase in the number of papers and results made available, and it has thus become important to summarize and provide an overview about those studies. Through a systematic mapping and bibliometrics study, we have reviewed 130 papers in this area. We present the results of our study in this paper. Also we have made the mapping data available on an online repository which is planned to be updated on a regular basis. The results of our study seem to suggest that many software engineering techniques and activities are being used in the development of scientific software. However, there is still a need for further exploration of the usefulness of specific software engineering techniques (e.g., regarding software maintenance, evolution, refactoring, re(v)-engineering, process and project management) in the scientific context. It is hoped that this article will help (new) researchers get an overview of the research space and help them to understand the trends in the area.

    References

    • S. Smith and W. Yu, Advances in Engineering Software 40(11), 1155 (2009). Crossref, ISIGoogle Scholar
    • S.   Smith and L.   Lai , A new requirements template for scientific computing , International Workshop on Situational Requirements Engineering Processes — Methods, Techniques and Tools to Support Situation-Specific Requirements Engineering Processes ( 2005 ) . Google Scholar
    • D. W. Kaneet al., Bioinformatics 7(1), 273 (2006). Google Scholar
    • H.-t. Li, An Ontology-based process model for scientific research software development, International Conference on Software Engineering and Data Mining (2010) pp. 299–304. Google Scholar
    • W. S.   Smith , J.   Carette and J.   McCutchan , Commonality analysis of families of physical models for use in scientific computing , International Workshop on Software Engineering for Computational Science and Engineering ( 2008 ) . Google Scholar
    • R. Sanders and D. Kelly, IEEE Software 25(4), 21 (2008). Crossref, ISIGoogle Scholar
    • M. A.   Heroux , J. M.   Willenbring and M. N.   Phenow , Improving the development process for CSE software , International Conference on Parallel, Distributed and Network-Based Processing ( 2007 ) . Google Scholar
    • S.   Larson and B.   Morrison , Managing software requirements in the context of the scientific enterprise , IEEE Aerospace Conference ( 2000 ) . Google Scholar
    • C. J.   Roy , Practical software engineering strategies for scientific computing , AIAA Computational Fluid Dynamics Conference ( 2009 ) . Google Scholar
    • J.   Segal , Some challenges facing software engineers developing software for scientists , Workshop on Software Engineering for Computational Science and Engineering ( 2009 ) . Google Scholar
    • S. Smith, Systematic development of requirements documentation for general purpose scientific computing software, Proc. of IEEE International Conf. on Requirement Engineering (2006) pp. 205–215. Google Scholar
    • J. Segal, Empirical Software Engineering 10(1), 517 (2005). Crossref, ISIGoogle Scholar
    • J. P. Kennyet al., Scientific Programming 16(4), 287 (2008). Crossref, ISIGoogle Scholar
    • R. H. Foghet al., Bioinformatics 21(8), 1678 (2005). Crossref, ISIGoogle Scholar
    • I.   Gorton et al. , A high-performance event service for HPC applications , International Workshop on Software Engineering for High Performance Computing Applications ( 2007 ) . Google Scholar
    • L.   Gross et al. , Pure and Applied Geophysics   165 , 653 ( 2008 ) . Crossref, ISIGoogle Scholar
    • M. Sala, ACM Transactions on Mathematical Software 32(3), (2006). Crossref, ISIGoogle Scholar
    • F.   Hernández , P.   Bangalore and K.   Reilly , Automating the development of scientific applications using domain-specific modeling , Proceedings of the Second International Workshop on Software Engineering for High Performance Computing System Applications ( 2005 ) . Google Scholar
    • R. C. Gentlemanet al., Genome Biology 5(10), (2004). Crossref, ISIGoogle Scholar
    • R. P. Kendall, D. E. Post and A. Mark, Case study of the NENE code project, IEEE Magazine on Computing in Science and Engineering, 2010 . Google Scholar
    • M. C. Smithet al., Creating a flexible environment for testing scientific software, Proceedings of SPIE — The International Society for Optical Engineering (2004) pp. 288–296. Google Scholar
    • T.   Cickovski , T.   Matthey and J. u. A.   Izaguirre , Design patterns for generic object-oriented scientific software , International Conference on Software Engineering ( 2005 ) . Google Scholar
    • D. W. I. Rouson, H. Adalsteinsson and J. Xia, ACM Transactions on Mathematical Software 37(1), (2010). Crossref, ISIGoogle Scholar
    • D. C. Arnold and J. J. Dongarra, Developing an architecture to support the implementation and development of scientific computing applications, Working Conference on the Architecture of Scientific Software (2000) pp. 39–55. Google Scholar
    • D. Woollard, C. Mattmann and N. Medvidovic, Injecting software architectural constraints into legacy scientific applications, Workshop on Software Engineering for Computational Science and Engineering (2009) pp. 65–71. Google Scholar
    • A.   Sharma and S. S.   Sarangdevot , Investigating the application of AOP methodology in development of bioinformatics software using Eclipse-AJDT environment , International Conference & Workshop on Emerging Trends in Technology ( 2011 ) . Google Scholar
    • D. E.   Bernholdt , R. C.   Armstrong and B. A.   Allan , Managing complexity in modern high end scientific computing through component-based software engineering , Workshop on Productivity and Performance in High-End Computing ( 2004 ) . Google Scholar
    • B. A. Allanet al., Scientific Programming 16(4), 315 (2008). Crossref, ISIGoogle Scholar
    • Donna Kaminskyj   Long et al. , Mind the Gap! Bridging the dichotomy of design and implementation , International Workshop on Software Engineering for Computational Science and Engineering ( 2011 ) . Google Scholar
    • H. P. L.   Are Magnus Bruaset , ACM Transactions on Mathematical Software   ( 1997 ) . Google Scholar
    • M.   Urbano Cuadrado , M. D.   Luque de Castro and M. A.   Gómez-Nieto , Trends in Analytical Chemistry   ( 2006 ) . Google Scholar
    • M. Sala, K. S. Stanley and M. A. Heroux, ACM Transanctions on Mathematical Software 34(2), (2008). Google Scholar
    • K.   Morris , D. W. I.   Rouson and J.   Xia , On the object-oriented design of reference-counted shadow objects , International Workshop on Software Engineering for Computational Science and Engineering ( 2011 ) . Google Scholar
    • A.-S. S.   Hai-Tao Li and Y.-X.   Meng , Parallel development and independent integration model for scientific research software engineering , International Conference on Computer Science and Software Engineering ( 2008 ) . Google Scholar
    • C. Blilie, Computing in Science and Engineering 4(3), 48 (2002). Crossref, ISIGoogle Scholar
    • R.   Neely , Practical software quality engineering on a large multiDisciplinary HPC development team , International Workshop on Software Engineering for High Performance Computing Applications ( 2004 ) . Google Scholar
    • Rune   Møllegaard Friborg and B.   Vinter , Journal of Computational Science   ( 2011 ) . Google Scholar
    • X. Jiao, M. T. Campbell and M. T. Heath, Roccom: An object-oriented, data-centric software integration framework for multiphysics simulations, International Conference on Supercomputing (2003) pp. 358–368. Google Scholar
    • G.   Rodríguez-Gómez , J.   Muñoz-Arteaga and R.   Fernández-Del-Busto , Scientific software design through scientific computing patterns , International Conference on Modelling, Simulation, and Optimization ( 2004 ) . Google Scholar
    • B.   Norris , Software architecture issues in scientific component development , International Conference on Applied Parallel Computing ( 2004 ) . Google Scholar
    • D.   Woollard , T.   Berger and W.   Yamada , Software engineering for neural dynamics simulations: A case study , International Workshop on Software Engineering for High Performance Computing Applications ( 2004 ) . Google Scholar
    • T. Q.   Andrea Lani , Dries   Kimpe and Herman   Deconinck , Stefan Vandewalle and Stefaan Poedts, The COOLFluiD framework: Design solutions for high performance object oriented scientific computing software , International Conference on Computational Science ( 2005 ) . Google Scholar
    • P. V. D.   Velde and G. D.   Mallinson , Advances in Engineering Software   ( 2007 ) . Google Scholar
    • S. M.   Gallo , R.   Miller and C. M.   Weeks , The design of a portable scientific tool: A case study using SnB , Proceedings of the ACM/IEEE Conference on Supercomputing ( 1996 ) . Google Scholar
    • R. Armstronget al., Toward a common component architecture for high-performance scientific computing, IEEE International Symposium on High Performance Distributed Computing (1999) pp. 13–24. Google Scholar
    • G.   Scherp and W.   Hasselbring , Towards a model-driven transformation framework for scientific workflows , Procedia Computer Science (Conference Proceedings) ( 2010 ) . Google Scholar
    • D. W. I.   Rouson , Scientific Programming   ( 2008 ) . Google Scholar
    • C. Macaulayet al., IEEE Software 26(1), 96 (2009). Crossref, ISIGoogle Scholar
    • I.   Gorton et al. , Velo: Riding the knowledge management wave for simulation and modeling , International Workshop on Software Engineering for Computational Science and Engineering ( 2011 ) . Google Scholar
    • L.   Hochstein and V.   Basili , IEEE Computer   ( 2008 ) . Google Scholar
    • O.   Marques and T.   Drummond , Building a software infrastructure for computational science applications: Lessons and solutions , Proceedings of the Second International Workshop on Software Engineering for High Performance Computing System Applications ( 2005 ) . Google Scholar
    • P. K.   Chilana , C. L.   P and A. J.   Ko , Comparing bioinformatics software development by computer scientists and biologists: An exploratory study , Workshop on Software Engineering for Computational Science and Engineering ( 2009 ) . Google Scholar
    • J. R. Caryet al., Computer Physics Communications 105(1), 20 (1997). Crossref, ISIGoogle Scholar
    • B. J.   Henz and D. R.   Shires , Coupling scientific applications within an object-oriented programming framework: Improving development time and software quality , International Conference on Parallel and Distributed Processing Techniques and Applications ( 2004 ) . Google Scholar
    • D.   Hudak et al. , Developing a computational science IDE for HPC systems , International Workshop on Software Engineering for High Performance Computing Applications ( 2007 ) . Google Scholar
    • R. Arora, P. Bangalore and M. Mernik, Developing scientific applications using generative programming, Workshop on Software Engineering for Computational Science and Engineering (2009) pp. 51–58. Google Scholar
    • R. Kendallet al., IEEE Software 25(4), 59 (2008). Crossref, ISIGoogle Scholar
    • M.   Hatlo et al. , IEEE Transactions on Nuclear Science   ( 2005 ) . Google Scholar
    • S. M. Easterbrook and T. C. Johns, Computing in Science and Engineering 11(6), 65 (2009). Crossref, ISIGoogle Scholar
    • D. M. Beazley and P. S. Lomdahl, Feeding a large-scale physics application to Python, International Python Conference (1997) pp. 21–28. Google Scholar
    • D. Kelly, D. Hook and R. Sanders, Computing in Science and Engineering 11(5), 48 (2009). Crossref, ISIGoogle Scholar
    • E. V.   Wyk , M.   Heimdahl and Y.   Saad , From models to efficient code: It's all in the middle , International Workshop on Software Engineering for High Performance Computing Applications ( 2004 ) . Google Scholar
    • J. E. Hannayet al., How do scientists develop and use scientific software?, Workshop on Software Engineering for Computational Science and Engineering (2009) pp. 1–8. Google Scholar
    • M.   Umarji and C.   Seaman , Informing design of a search tool for bioinformatics , International Workshop on Software Engineering for Computational Science and Engineering ( 2008 ) . Google Scholar
    • V.   Basili et al. , Large efficient table-top teraflop computing , International Workshop on Software Engineering for Computational Science and Engineering ( 2008 ) . Google Scholar
    • P. F. Duboi, Maintaining correctness in scientific programs, IEEE Magazine on Computing in Science and Engineering, 2005 . Google Scholar
    • T. E. Oliphant, Python for scientific computing, IEEE Magazine on Computing in Science and Engineering, 2007 . Google Scholar
    • J. J.   Marshall , R. R.   Downs and S.   Samadi , Earth Science Informatics   ( 2010 ) . Google Scholar
    • W. Yu and S. Smith, Reusability of FEA software: A program family approach, Workshop on Software Engineering for Computational Science and Engineering (2009) pp. 43–50. Google Scholar
    • Y. K. G.   Iwai , T.   Saski and Y.   Watse , SAGA-based user environment for distributed computing resources: A universal Grid solution over multi-middleware infrastructures , Procedia Computer Science (Conference Proceedings) ( 2010 ) . Google Scholar
    • S. Faulk, E. Loh, Michael L. Van De Vanter, S. Squires and L. G. Votta, Scientific computing's productivity gridlock: How software engineering can help, IEEE Magazine on Computing in Science and Engineering, 2009 . Google Scholar
    • D. Woollardet al., IEEE Software 25(4), 37 (2008). Crossref, ISIGoogle Scholar
    • K. S. Ackroydet al., IEEE Software 25(4), 44 (2008). Crossref, ISIGoogle Scholar
    • P.   Luksch et al. , IEEE Concurreny   ( 1997 ) . Google Scholar
    • D. S.   Mark Vigder , Janice Singer, Software automation in scientific research organizations , International Workshop on Software Engineering for Computational Science and Engineering ( 2008 ) . Google Scholar
    • M. Erwig and Z. Fu, ACM Transactions on Software Engineering and Methodology 14(2), 168 (2005). Crossref, ISIGoogle Scholar
    • A. Dall'Osso, Future Generation Computer Systems 19(2), 143 (2003). Crossref, ISIGoogle Scholar
    • T. L. Dahlgren and P. T. Devanbu, Adaptable assertion checking for scientific software components, International Workshop on Software Engineering for High Performance Computing System Applications (2004) pp. 64–69. Google Scholar
    • H.   Naguib and Y.   Li , Applying software engineering methods and tools to CSE research projects , International Conference on Computational Science ( 2010 ) . Google Scholar
    • D.   Kelly and R.   Sanders , Assessing the quality of scientific software , Workshop on Software Engineering for Computational Science & Engineering ( 2008 ) . Google Scholar
    • D.   Post , R.   Kendall and W.   Earl , Case study of the falcon code project , Proceedings of the Second International Workshop on Software Engineering for High Performance Computing System Applications ( 2005 ) . Google Scholar
    • M. G. Cox and P. M. Harris, Analytica Chimica Acta 380(3), 339 (1999). Crossref, ISIGoogle Scholar
    • R. G.   Diane Kelly and Yizhen   Shao , Journal of Computational Science   ( 2011 ) . Google Scholar
    • G.   Avrunin , S.   Siegel and A.   Siegel , Finite-state verification for high performance computing , International Workshop on Software Engineering for High Performance Computing Applications ( 2005 ) . Google Scholar
    • R.   Palmer et al. , Gauss: A framework for verifying scientific computing software , Proceedings of the Workshop on Software Model Checking ( 2005 ) . Google Scholar
    • S.   Asgari et al. , Generating testable hypotheses from tacit knowledge for high productivity computing , International Workshop on Software Engineering for High Performance Computing Applications ( 2005 ) . Google Scholar
    • M.   Heroux , Improving CSE software through reproducibility requirements , International Workshop on Software Engineering for Computational Science and Engineering ( 2011 ) . Google Scholar
    • T. L.   Dahlgren and P. T.   Devanbu , Improving scientific software component quality through assertions , International Conference on Software Engineering ( 2005 ) . Google Scholar
    • R.   Graya and D.   Kelly , Investigating test selection techniques for scientific software using Hook's mutation sensitivity testing , International Conference on Computational Science ( 2010 ) . Google Scholar
    • S. A. Vilkomiret al., Modeling input space for testing scientific computational software: A case study, International Conference on Computational Science (2008) pp. 291–300. Google Scholar
    • D. Hook and D. Kelly, IEEE Design & Test 11(6), 40 (2009). Google Scholar
    • D.   Kelly , S.   Thorsteinson and D.   Hook , IEEE Software   ( 2011 ) . Google Scholar
    • J. C. Carveret al., Software development environments for scientific and engineering software: A series of case studies, International conference on Software Engineering (2007) pp. 550–559. Google Scholar
    • D. E.   Post and R. P.   Kendall , International Journal of High Performance Computing Applications   ( 2004 ) . Google Scholar
    • C.   Morris , Some lessons learned reviewing scientific code , International Workshop on Software Engineering for Computational Science and Engineering ( 2008 ) . Google Scholar
    • J. Segal, Some problems of professional end user developers, IEEE Symposium on Visual Languages and Human-Centric Computing (2007) pp. 111–118. Google Scholar
    • H.   Remmel et al. , Supporting the testing of scientific frameworks with software product line engineering — A proposed approach , International Workshop on Software Engineering for Computational Science and Engineering ( 2011 ) . Google Scholar
    • D. Hook and D. Kelly, Testing for trustworthiness in scientific software, Workshop on Software Engineering for Computational Science and Engineering (2009) pp. 59–64. Google Scholar
    • R.   Sanders and D.   Kelly , The challenge of testing scientific software , Conference for the Association for Software Testing ( 2008 ) . Google Scholar
    • L. Hatton, IEEE Computational Science & Engineering 4(2), 27 (1997). Crossref, ISIGoogle Scholar
    • S.   Booth and D.   Henty , Verification strategies for high performance computing software , International Workshop on Software Engineering for High Performance Computing Applications ( 2004 ) . Google Scholar
    • E.   Loh , M. L. V. D.   Vanter and L. G.   Votta , Can software engineering solve the HPCS problem? , Proceedings of the Second International Workshop on Software Engineering for High Performance Computing System Applications ( 2005 ) . Google Scholar
    • D. Kelly, Journal of Systems and Software 82(5), 851 (2009). Crossref, ISIGoogle Scholar
    • M.   Wagner , Evolution from a scientific application to an applicable product , European Conference on Software Maintenance and Reengineering ( 2007 ) . Google Scholar
    • Y.   Li , Reengineering a scientific software and lessons learned , International Workshop on Software Engineering for Computational Science and Engineering ( 2011 ) . Google Scholar
    • Jeffrey   Overbey et al. , Refactorings for Fortran and high-performance computing , International Workshop on Software Engineering for High Performance Computing Applications ( 2005 ) . Google Scholar
    • J. Pitt-Franciset al., Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 366(1878), 3111 (2008). Crossref, ISIGoogle Scholar
    • W. A. Wood and W. L. Kleb, IEEE Software 20(3), 30 (2003). Crossref, ISIGoogle Scholar
    • D. Kane, Introducing agile development into bioinformatics: An experience report, Proceedings of the Conference on Agile Development (2003) pp. 132–140. Google Scholar
    • J.   Segal , Models of scientific software development , International Workshop on Software Engineering for Computational Science and Engineering ( 2008 ) . Google Scholar
    • J. Segal, Computer Supported Cooperative Work 18(6), 581 (2009). Crossref, ISIGoogle Scholar
    • D. E.   Post , The challenge for computational science , International Workshop on Software Engineering for High Performance Computing Applications ( 2004 ) . Google Scholar
    • J.   Willenbring , M.   Heroux and R.   Heaphy , The Trilinos software lifecycle model , International Workshop on Software Engineering for High Performance Computing Applications ( 2007 ) . Google Scholar
    • M.-S. Wuet al., Integrating performance tools with large-scale scientific software, IEEE International Parallel and Distributed Processing Symposium (2007) pp. 1–8. Google Scholar
    • M.   Vigder , End-user software development in a scientific organization , ICSE Workshop on Software Engineering Foundations for End User Programming ( 2009 ) . Google Scholar
    • T. Epperly, S. R. Kohn and G. Kumfert, Component technology for high-performance sceintific simulation software, Working Conference on the Architecture of Scientific Software (2000) pp. 69–86. Google Scholar
    • J. P. Kennyet al., Journal of Computational Chemistry 25(14), 1717 (2004). Crossref, ISIGoogle Scholar
    • V.   Maxville , Preparing scientists for scalable software development , Workshop on Software Engineering for Computational Science and Engineering ( 2009 ) . Google Scholar
    • J.   Howison and J. D.   Herbsleb , Scientific software production: Incentives and collaboration , Proceedings of the ACM Conference on Computer Supported Cooperative Work ( 2011 ) . Google Scholar
    • S.   Samadi et al. , Strategies for enabling software reuse within the earth science community , International Geoscience and Remote Sensing Symposium ( 2004 ) . Google Scholar
    • C.   Halverson et al. , Towards an ecologically valid study of programmer behavior for scientific computing , International Workshop on Software Engineering for Computational Science and Engineering ( 2008 ) . Google Scholar
    • V. R. Basiliet al., IEEE Software 25(4), 29 (2008). Crossref, ISIGoogle Scholar
    • J. F. Cremer, R. S. Palmer and R. E. Zippel, Transactions of the Society for Computer Simulation International 14(1), 37 (1997). Google Scholar
    • S. M.   Baxter et al. , PLoS Computational Biology   ( 2006 ) . Google Scholar
    • J.   Segal , Scientists and software engineers: A tale of two cultures , Proceedings of Annual Meeting of the Psychology of Programming Interest Group ( 2008 ) . Google Scholar
    • R.   Baxter , Software engineering is software engineering , International Workshop on Software Engineering for High Performance Computing Applications ( 2004 ) . Google Scholar
    • Y.   Alexeev et al. , Journal of Physics   16 , ( 2005 ) . Google Scholar
    • R. A. Bartlett, Integration strategies for computational science & engineering software, Workshop on Software Engineering for Computational Science and Engineering (2009) pp. 35–42. Google Scholar
    • C.   Morris and J.   Segal , Some challenges facing scientific software developers: The case of molecular biology , IEEE International Conference on e-Science ( 2009 ) . Google Scholar
    • L.   Nguyen-Hoan , S.   Flint and R.   Sankaranarayana , A survey of scientific software development , International Symposium on Empirical Software Engineering and Measurement ( 2010 ) . Google Scholar
    • D. F. Kelly, IEEE Software 24(6), 118 (2007). Crossref, ISIGoogle Scholar
    • V. K. Decyk, C. D. Norton and H. J. Gardner, Computing in Science and Engineering 9(4), 68 (2007). Crossref, ISIGoogle Scholar
    • R. L. Glass and T. Y. Chen, Journal of Systems and Software 68(1), 77 (2003). Crossref, ISIGoogle Scholar
    • R. L. Glass and T. Y. Chen, Journal of Systems and Software 76(1), 91 (2005). Crossref, ISIGoogle Scholar
    • V. Garousi and T. Varma, Canadian Journal on Computer and Information Science 3(2), 19 (2010). Google Scholar
    • V. Garousi and T. Varma, Canadian Journal on Computer and Information Science 5(4), 1 (2012). Google Scholar
    • D. Budgenet al., Using mapping studies in software engineering, Proceedings of Annual Workshop of the Psychology of Programming Interest Group (2008) pp. 195–204. Google Scholar
    • K.   Petersen et al. , Systematic mapping studies in software engineering , International Conference on Evaluation and Assessment in Software Engineering ( 2008 ) . Google Scholar
    • M. T.   Sletholt et al. , A literature review of agile practices and their effects in scientific software development , International Workshop on Software Engineering for Computational Science and Engineering ( 2011 ) . Google Scholar
    • B.   Kitchenham and S.   Charters , Evidence-Based Software Engineering   ( 2007 ) . Google Scholar
    • V. Garousi, Classification and trend analysis of UML books (1997–2009), Journal on Software & System Modeling (SoSyM), to Appear . Google Scholar
    • F.   Elberzhager , J.   Münch and V. T. N.   Nha , Information and Software Technology   54 , 1 ( 2012 ) . Crossref, ISIGoogle Scholar
    • S. Aliet al., IEEE Transactions on Software Engineering 36(6), 742 (2010). Crossref, ISIGoogle Scholar
    • V. R. Basili, Software modeling and measurement: The Goal/Question/Metric paradigm, Technical report, University of Maryland at College Park 1992 . Google Scholar
    • R. Farhoodi, V. Garousi, D. Pfahl and J. Sillito, Systematic mapping of developing scientific software, online repository on Google Docs, http://www.softqual.ucalgary.ca/projects/2012/SM_CSE/, Last accessed: April 2012 . Google Scholar
    • P.   Runeson et al. , Case Study Research in Software Engineering: Guidelines and Examples ( John Wiley & Sons , 2012 ) . CrossrefGoogle Scholar
    • B. D.   Nicola , Bibliometrics and Citation Analysis: From the Science Citation Index to Cybermetrics ( Scarecrow Press , 2009 ) . Google Scholar
    • R.   Gentleman , R Programming for Bioinformatics ( CRC Press , 2009 ) . Google Scholar
    • Y.   Liu and A.   Milanova , Static analysis for inference of explicit information flow , Proceedings of ACM Workshop on Program Analysis for Software Tools and Engineering ( 2008 ) . Google Scholar
    • R. Kendallet al., IEEE Software 25(4), 59 (2008). Crossref, ISIGoogle Scholar
    • T. D.   Cook and D. T.   Campbell , Quasi-Experimentation — Design and Analysis Issues for Field Settings ( Houghton Miffin , 1979 ) . Google Scholar
    Remember to check out the Most Cited Articles!

    Check out our titles in C++ Programming!