THE IMPACT OF MESSAGE-BUFFER ALIGNMENT ON COMMUNICATION PERFORMANCE
Abstract
Of the many factors that contribute to communication performance, perhaps one of the least investigated is that of message-buffer alignment. Although the generally accepted practice is to page-align buffer memory for best performance, our studies show that the actual relationship of buffer alignment to communication performance cannot be expressed with such a simple formula. This paper presents a case study in which porting a simple network performance test from one language to another resulted in a large performance discrepancy even though both versions of the code consist primarily of calls to messaging-layer functions. Careful analysis of the two code versions revealed that the discrepancy relates to the alignment in memory of the message buffers. Further investigation revealed some surprising results about the impact of message-buffer alignment on communication performance: (1) different networks and node architectures prefer different buffer alignments; (2) page-aligned memory does not always give the best possible performance, and, in some cases, actually yields the worst possible performance; and, (3) on some systems, the most significant factor affecting network performance is the relative alignment of send and receive buffers with respect to each other.
References
- Pallas, GmbH. Pallas MPI Benchmarks—PMB, Part MPI-1, 2000. Available from ftp://ftp.pallas.com/pub/PALLAS/PMB/PMB-MPI1.pdf . Google Scholar
- , Recent Advances in Parallel Virtual Machine and Message Passing Interface: Proc. 5th European PVM/MPI Users' Group Meeting (EuroPVM/MPI'98),
Lecture Notes in Computer Science 1497, eds.V. Alexandrov and J. Dongarra (Springer-Verlag, Liverpool, United Kingdom, 1998) pp. 52–59, http://www.mpi-sb.mpg.de/~sanders/papers/europvm-mpi98.ps.gz. Crossref, Google Scholar - , Proc. 1996 ISMM Int. Conf. on Intelligent Information Management Systems , ed.
J. S. Wong ( ACTA Press , Washington, DC , 1996 ) , http://www.scl.ameslab.gov/netpipe/paper/netpipe.ps . Google Scholar - , Recent Advances in Parallel Virtual Machine and Message Passing Interface: Proc. 6th European PVM/MPI Users' Group Meeting (EuroPVM/MPI'99),
Lecture Notes in Computer Science 1697, eds.J. Dongarra , E. Luque and T. Margalef (Spriger-Verlag, Barcelona, Spain, 1999) pp. 11–18, http://www.mcs.anl.gov/~gropp/bib/papers/1999/pvmmpi99/mpptest.pdf. Crossref, Google Scholar - S. Pakin. CONCEPTUAL: A network correctness and performance testing language, in Proc. Int. Parallel and Distributed Processing Symposium (IPDPS 2004), Santa Fe, New Mexico, 2004, Available from http://www.c3.lanl.gov/~pakin/papers/ipdps2004.pdf . Google Scholar
- S. Pakin. Reproducible network benchmarks with CONCEPTUAL, in Proc. Euro-Par 2004, Pisa, Italy, 2004, Available from http://www.c3.lanl.gov/~pakin/papers/europar2004.pdf . Google Scholar
- S. Pakin. CONCEPTUAL user's guide. Technical Report LA-UR 03-7356, Los Alamos National Laboratory, Los Alamos, New Mexico, 2003. Available from http://www.c3.lanl.gov/~pakin/software/conceptual/conceptual.pdf . Google Scholar
- IEEE Micro 22, 46 (2002), http://www.c3.lanl.gov/~fabrizio/papers/ieeemicro.pdf. Crossref, ISI, Google Scholar
- Parallel Computing 22, 789 (1996), ftp://ftp.mcs.anl.gov/pub/mpi/mpicharticle.ps. Crossref, ISI, Google Scholar
- J. Beecroft, D. Addison, F. Petrini and M. McLaren. Quadrics QsNet II: A network for supercomputing applications, in Proc. Hot Chips 15, Palo Alto, California, 2003, Available from http://www.c3.lanl.gov/~fabrizio/papers/hot03.pdf . Google Scholar
- Mellanox Technologies, Inc. Introduction to InfiniBand, 2003. Document Number 2003WP. Available from http://www.mellanox.com/technology/shared/IB_Intro_WP_190.pdf . Google Scholar
- IEEE Micro 15, 29 (1995), http://www.myri.com/research/publications/Hot.ps. Crossref, ISI, Google Scholar
- LAN/MAN Standards Committee. Carrier sense multiple access with collision detection (CSMA/CD) access method and physical layer specifications. IEEE Standard 802.3, IEEE Computer Society, Technical Committee on Computer Communications, New York, New York, 2002. ISBN 0-7381-3089-3. Available from http://standards.ieee.org/getieee802/download/802.3-2002.pdf . Google Scholar


