高性能计算

Heterogenenity and LAM/MPI(zz)

2010年3月26日 阅读(37)

http://www.lam-mpi.org/faq/category11.php3
Table of contents:

  1. What exactly is considered a "heterogeneous" cluster?
  2. Does LAM/MPI work on heterogeneous clusters?
  3. Do different versions of LAM/MPI constitute heterogeneous clusters?
  4. How do I install LAM on a heterogeneous cluster?
  5. How do I de<lambootde< across a heterogeneous cluster?
  6. How do I execute the right binary on each node for each architecture in a heterogeneous system?
  7. Can I mix 32 and 64 bit executables in a single parallel MPI job?

[ Return to FAQ ]

1. What exactly is considered a "heterogeneous" cluster?

It is probably easier to define what a "homogeneous" cluster is — a "heterogeneous" cluster is anything that is not a "homogeneous" cluster.

A homogeneous cluster is one where all the nodes have the same:

The first requirement — same architecture — has a bit of leeway. For example, two Pentium III machines with different amounts of RAM or a different CPU speed would still be considered homogeneous. In general, homogeneity is determined by whether the software compiled on one machine can run natively on another. In the case of the same CPU but different amounts of RAM or a different CPU speed, this is most likely true. This is not necessarily true between a Pentium II and a Pentium III, for example.

For example, the following cluster is considered homogeneous:

Note that it wasn’t necessary to list the Linux kernel version and/or de<glibcde< version because they’re all the same by virtue of being the same Linux distribution and version.

The following are some example clusters that are not homogeneous — they are heterogeneous:

[ Top of page | Return to FAQ ]

2. Does LAM/MPI work on heterogeneous clusters?

Yes — that’s one of the reasons that LAM/MPI exists.

LAM/MPI will work between just about any flavor of POSIX (with a few restrictions). That is, you can have two completely different machines (e.g., a Sun machine and an Intel-based machine), and LAM will run on both of them. More importantly, you can run a single parallel job that spans both of them.

LAM will transparently do any data conversion necessary.

An important restriction is that LAM does not currently support systems that have datatypes that are different sizes. For example, if an integer is 64 bits on one machine and is 32 bits on another, LAM’s behavior is undefined. Also, LAM requires that floating point formats be the same. That is, endianness can be different, but the same general format must be obeyed by all participating machines (e.g., older Alpha machines do not adhere to IEEE floating point standard by default — such machines can be used in parallel jobs with other similar machines, but to use them in a heterogeneous situation would require adherence to the IEEE floating point standards so that all nodes in the parallel job understand the same floating point formats).

Indeed, what is the Right Thing for an MPI to do in these kinds of situations, anyway? There really is no good answer — having MPI truncate when 64 bit integers are sent to 32 bit integers is not desirable, nor is having the MPI translate from one floating point format to another (for similar loss of precision reasons).

[ Top of page | Return to FAQ ]

3. Do different versions of LAM/MPI constitute heterogeneous clusters?

Strictly speaking, yes.

BUT different versions of LAM will not work together. In order to successfully de<lambootde< and de<mpirunde<, you must use the same version of LAM/MPI on all nodes, regardless of their operating system, architecture, etc.

So a better answer is really: Yes, but don’t ever, ever do this.

[ Top of page | Return to FAQ ]

4. How do I install LAM on a heterogeneous cluster?

In general, LAM must be compiled and installed separatedly for each different kind of node in a heterogeneous cluster.

Other questions in this FAQ discuss how to install LAM across a [homogeneous] cluster — there are two general schemes:

Both of these methods are possible for heterogeneous clusters as well. Physically installing LAM on each node in the cluster is the safest, least complicated way to do this. However, it is potentially the most labor intensive, and most difficult to maintain over time.

In most cases, there will be multiple nodes of each kind in a heterogeneous cluster. As such, it may be useful to consider a heterogeneous cluster to be a group of homogeneous clusters. So although local policies and requirements may vary, the LAM Team recommends that LAM is installed on a networked filesystem in each homogeneous cluster.

NOTE: There are some scalability issues with using networked filesystems on large clusters. As such, it may not be sufficient or desirable to use the common filesystem model at your site, depending on the size of your cluster and your choice of networked filesystem. YMMV.

For example, consider a cluster of 16 Pentium II nodes running Red Hat 7.0 and second group of 16 Pentium III nodes running Red Hat 7.1. Both the architecture difference and operating system difference make these sub-clusters heterogeneous.

In the common filesystem model, LAM will need to be installed twice for the heterogeneous cluster described above — once for the PII/RH7.0 machines, and once for the PIII/RH7.1 machines. Each machine in the cluster will need to either mount the appropriate LAM installation, and/or user paths will need to be set appropriately on each node in the cluster to point to the appropriate LAM installation.

The same holds true for more obviously-heterogeneous clusters, such as a group of UltraSparc machines running Solaris and a group of Pentium III machines running some flavor of Linux.

[ Top of page | Return to FAQ ]

5. How do I de<lambootde< across a heterogeneous cluster?

In additions to the normal requirements for de<lambootde<, the additional requirements must be satisfied:

  1. All nodes being de<lambootde<ing must be using the same version of LAM (this is actually always a requirement — this is just a clarification that "heterogeneous" does not mean "different versions of LAM/MPI").
  2. Each user’s de<$PATHde< must be setup properly to find the Right version of LAM/MPI on each node. That is, if multiple installations of LAM are available on each node, the user’s de<$PATHde< must be set to find the appropriate installation for that node. For example, if LAM is installed on a networked filesystem for two different architectures in:

    de<de<
    /home/lam/sparc-sun-solaris2.8
    /home/lam/linux-redhat7.1
    /home/lam/linux-suse7.2

    If de</home/lamde< is NFS mounted on all nodes in the cluster, the user’s de<$PATHde< must be set to use one of those three trees as appropriate for the kind of node that they are logged in to. This is typically set in the user’s dot files (e.g., de<$HOME/.profilede<, de<$HOME/.cshrcde<, etc.), or in a system-wide default dot file (these vary between different operating systems).

[ Top of page | Return to FAQ ]

6. How do I execute the right binary on each node for each architecture in a heterogeneous system?

There are three cases:

Remember, it may be necessary to have different versions of the MPI binary for each OS version as well as each machine architecture. For example, you may need to have a separate versions for Solaris 2.5 and 2.6. This is also true when running between different linux distributions — as shown in the example above where Red Hat and SuSE are considered different operating systems and therefore have their own copy of de<foode<.

[ Top of page | Return to FAQ ]

7. Can I mix 32 and 64 bit executables in a single parallel MPI job?

By definition, a mixture of 32 and 64 bit machines is a heterogenous cluster.

LAM/MPI allows two possibilities for mixing 32 and 64 bit machines in a single parallel job:

  1. Most 64 bit operating systems have the capacity to generate 32 bit executables. By doing so, one can make the cluster "homogenous" (at least in terms of bit size). Once all the executables (including relevant libraries) are 32 bits, one can run MPI jobs as if it were a homogenous cluster. Note that LAM/MPI libraries and executables should also be built as 32 bit libraries/executables.

    This solution works well and avoids many complicated situations which arise out of mixing 32 and 64 bit executables and are outside the scope of MPI (see discussion below).

  2. The differences in datatype sizes between 32 and 64 bit machcines are likely to create problems. Consider the scenario where a 64 process sends a message containing de<MPI_LONGde< data to a 32 bit process. What is the size of the datatype? On the 64 bit machine, each de<MPI_LONGde< is likely to be 64 bits, but on the 32 bit machine, they are likely to be 32 bits. So what should the 32 bit process do when it receives the data?

    There is, unfortunately, no good answer to this. Obvious choices include invoking an error or truncating the data, neither of which are attractive. Debugging such applications is non-trivial and therefore this is not the preferred solution.

    NOTE: LAM/MPI has not been tested in this kind of configuration! It may work (if the user application stays away from messages with mismatched data sizes), but it may not… Consider yourself warned.

You Might Also Like