World Scientific
  • Search
Skip main navigation

Cookies Notification

We use cookies on this site to enhance your user experience. By continuing to browse the site, you consent to the use of our cookies. Learn More
×
Our website is made possible by displaying certain online content using javascript.
In order to view the full content, please disable your ad blocker or whitelist our website www.worldscientific.com.

System Upgrade on Tue, Oct 25th, 2022 at 2am (EDT)

Existing users will be able to log into the site and access content. However, E-commerce and registration of new users may not be available for up to 12 hours.
For online purchase, please visit us again. Contact us at [email protected] for any enquiries.

THE IMPACT OF MESSAGE-BUFFER ALIGNMENT ON COMMUNICATION PERFORMANCE

    Of the many factors that contribute to communication performance, perhaps one of the least investigated is that of message-buffer alignment. Although the generally accepted practice is to page-align buffer memory for best performance, our studies show that the actual relationship of buffer alignment to communication performance cannot be expressed with such a simple formula. This paper presents a case study in which porting a simple network performance test from one language to another resulted in a large performance discrepancy even though both versions of the code consist primarily of calls to messaging-layer functions. Careful analysis of the two code versions revealed that the discrepancy relates to the alignment in memory of the message buffers. Further investigation revealed some surprising results about the impact of message-buffer alignment on communication performance: (1) different networks and node architectures prefer different buffer alignments; (2) page-aligned memory does not always give the best possible performance, and, in some cases, actually yields the worst possible performance; and, (3) on some systems, the most significant factor affecting network performance is the relative alignment of send and receive buffers with respect to each other.

    References

    • Pallas, GmbH. Pallas MPI Benchmarks—PMB, Part MPI-1, 2000. Available from ftp://ftp.pallas.com/pub/PALLAS/PMB/PMB-MPI1.pdf . Google Scholar
    • R. Reussneret al., Recent Advances in Parallel Virtual Machine and Message Passing Interface: Proc. 5th European PVM/MPI Users' Group Meeting (EuroPVM/MPI'98), Lecture Notes in Computer Science 1497, eds. V. Alexandrov and J. Dongarra (Springer-Verlag, Liverpool, United Kingdom, 1998) pp. 52–59, http://www.mpi-sb.mpg.de/~sanders/papers/europvm-mpi98.ps.gz. CrossrefGoogle Scholar
    • Q. O.   Snell , A. R.   Mikler and J. L.   Gustafson , Proc. 1996 ISMM Int. Conf. on Intelligent Information Management Systems , ed. J. S.   Wong ( ACTA Press , Washington, DC , 1996 ) ,   http://www.scl.ameslab.gov/netpipe/paper/netpipe.ps . Google Scholar
    • W. Gropp and E. Lusk, Recent Advances in Parallel Virtual Machine and Message Passing Interface: Proc. 6th European PVM/MPI Users' Group Meeting (EuroPVM/MPI'99), Lecture Notes in Computer Science 1697, eds. J. Dongarra, E. Luque and T. Margalef (Spriger-Verlag, Barcelona, Spain, 1999) pp. 11–18, http://www.mcs.anl.gov/~gropp/bib/papers/1999/pvmmpi99/mpptest.pdf. CrossrefGoogle Scholar
    • S. Pakin. CONCEPTUAL: A network correctness and performance testing language, in Proc. Int. Parallel and Distributed Processing Symposium (IPDPS 2004), Santa Fe, New Mexico, 2004, Available from http://www.c3.lanl.gov/~pakin/papers/ipdps2004.pdf . Google Scholar
    • S. Pakin. Reproducible network benchmarks with CONCEPTUAL, in Proc. Euro-Par 2004, Pisa, Italy, 2004, Available from http://www.c3.lanl.gov/~pakin/papers/europar2004.pdf . Google Scholar
    • S. Pakin. CONCEPTUAL user's guide. Technical Report LA-UR 03-7356, Los Alamos National Laboratory, Los Alamos, New Mexico, 2003. Available from http://www.c3.lanl.gov/~pakin/software/conceptual/conceptual.pdf . Google Scholar
    • F. Petriniet al., IEEE Micro 22, 46 (2002), http://www.c3.lanl.gov/~fabrizio/papers/ieeemicro.pdf. Crossref, ISIGoogle Scholar
    • W. Groppet al., Parallel Computing 22, 789 (1996), ftp://ftp.mcs.anl.gov/pub/mpi/mpicharticle.ps. Crossref, ISIGoogle Scholar
    • J. Beecroft, D. Addison, F. Petrini and M. McLaren. Quadrics QsNet II: A network for supercomputing applications, in Proc. Hot Chips 15, Palo Alto, California, 2003, Available from http://www.c3.lanl.gov/~fabrizio/papers/hot03.pdf . Google Scholar
    • Mellanox Technologies, Inc. Introduction to InfiniBand, 2003. Document Number 2003WP. Available from http://www.mellanox.com/technology/shared/IB_Intro_WP_190.pdf . Google Scholar
    • N. J. Bodenet al., IEEE Micro 15, 29 (1995), http://www.myri.com/research/publications/Hot.ps. Crossref, ISIGoogle Scholar
    • LAN/MAN Standards Committee. Carrier sense multiple access with collision detection (CSMA/CD) access method and physical layer specifications. IEEE Standard 802.3, IEEE Computer Society, Technical Committee on Computer Communications, New York, New York, 2002. ISBN 0-7381-3089-3. Available from http://standards.ieee.org/getieee802/download/802.3-2002.pdf . Google Scholar