openmpi-* subports, and publishing of binaries

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

openmpi-* subports, and publishing of binaries

Christopher Nielsen
Given how long the various openmpi-* subports take to build, it would be beneficial to publish binaries for our users.

This is currently disabled in the Portfile, with a brief mention that it’s due to ‘hwloc’... and the tight coupling between the compiled binaries, and the hardware it’s compiled on.

Digging through all information resources available - including git history, old trac tickets, mailing list archives, etc - I couldn’t find much discussion surrounding this.

So, assuming this isn’t a simple licensing issue (?), does anyone have an answer?

Of note, my goal is to eliminate any technical barriers (if possible), by engaging the OpenMPI developers. But before doing that, any relevant background would be helpful.

Cheers and Thanks,
-Chris
Reply | Threaded
Open this post in threaded view
|

Re: openmpi-* subports, and publishing of binaries

ryandesign2
Administrator
On Feb 1, 2021, at 09:49, Christopher Nielsen wrote:

> Given how long the various openmpi-* subports take to build, it would be beneficial to publish binaries for our users.
>
> This is currently disabled in the Portfile, with a brief mention that it’s due to ‘hwloc’... and the tight coupling between the compiled binaries, and the hardware it’s compiled on.
>
> Digging through all information resources available - including git history, old trac tickets, mailing list archives, etc - I couldn’t find much discussion surrounding this.
>
> So, assuming this isn’t a simple licensing issue (?), does anyone have an answer?
>
> Of note, my goal is to eliminate any technical barriers (if possible), by engaging the OpenMPI developers. But before doing that, any relevant background would be helpful.

According to the comment in the portfile, "openmpi depends on hwloc and that would be different for a user's machine".

The hwloc project is here: https://www.open-mpi.org/projects/hwloc/

According to "port info hwloc", "The Portable Hardware Locality (hwloc) software package provides a portable abstraction (across OS, versions, architectures, ...) of the hierarchical topology of modern architectures, including NUMA memory nodes, sockets, shared caches, cores and simultaneous multithreading."

I'm not familiar with all of that but I assume the situation is that openmpi, via hwloc, will determine the capabilities available on the build machine's processor and try to use those capabilities to the fullest at runtime. If something built on our build machines with their Intel Xeon processors were distributed to a user with let's say an Intel Core 2 Duo processor, it is possible that the processor features that openmpi/hwloc determined were available on the Xeon are not available on the Core 2 Duo, and that the program would crash or work incorrectly in some way.

If what I've described is accurate, then to accomplish what you want, you would need to modify the openmpi port so that it does not do that: does not use hwloc, does not attempt to discover specific processor features. Make it build the same on any Intel processor. Then we could distribute the binary.

A variant could be added that reintroduces the processor-specific optimizations. We have standardized on the variant name "native" for this purpose. Users who wish to have the very fastest software, at the expense of portability, can enable this variant either on a port by port basis or in their variants.conf file.


Reply | Threaded
Open this post in threaded view
|

Re: openmpi-* subports, and publishing of binaries

Eric Borisch
This seems off.

Hwloc’s own page talks about the availability of pre-compiled packages, and its demos focus on determining what threads to bind to (on OSes where explicit binding is possible; last I checked it wasn’t on OSX) if you want to maximize or minimize shared resources. (You may want shared L2 cache between threads for cooperative tasks, or to put threads on cores with different memory busses - and bind memory allocs, too - if your code is burning through large, and separable, chunks of memory at max bandwidth.)

It is designed to be used at runtime to provide a uniform (across OSes) way to interrogate the running system’s topology.

This is separate from flags like -march which lock in instruction sets which will be used  — and can create decidedly non-portable binaries.

Just my 2c,
  - Eric


On Wed, Feb 3, 2021 at 12:48 AM Ryan Schmidt <[hidden email]> wrote:
On Feb 1, 2021, at 09:49, Christopher Nielsen wrote:

> Given how long the various openmpi-* subports take to build, it would be beneficial to publish binaries for our users.
>
> This is currently disabled in the Portfile, with a brief mention that it’s due to ‘hwloc’... and the tight coupling between the compiled binaries, and the hardware it’s compiled on.
>
> Digging through all information resources available - including git history, old trac tickets, mailing list archives, etc - I couldn’t find much discussion surrounding this.
>
> So, assuming this isn’t a simple licensing issue (?), does anyone have an answer?
>
> Of note, my goal is to eliminate any technical barriers (if possible), by engaging the OpenMPI developers. But before doing that, any relevant background would be helpful.

According to the comment in the portfile, "openmpi depends on hwloc and that would be different for a user's machine".

The hwloc project is here: https://www.open-mpi.org/projects/hwloc/

According to "port info hwloc", "The Portable Hardware Locality (hwloc) software package provides a portable abstraction (across OS, versions, architectures, ...) of the hierarchical topology of modern architectures, including NUMA memory nodes, sockets, shared caches, cores and simultaneous multithreading."

I'm not familiar with all of that but I assume the situation is that openmpi, via hwloc, will determine the capabilities available on the build machine's processor and try to use those capabilities to the fullest at runtime. If something built on our build machines with their Intel Xeon processors were distributed to a user with let's say an Intel Core 2 Duo processor, it is possible that the processor features that openmpi/hwloc determined were available on the Xeon are not available on the Core 2 Duo, and that the program would crash or work incorrectly in some way.

If what I've described is accurate, then to accomplish what you want, you would need to modify the openmpi port so that it does not do that: does not use hwloc, does not attempt to discover specific processor features. Make it build the same on any Intel processor. Then we could distribute the binary.

A variant could be added that reintroduces the processor-specific optimizations. We have standardized on the variant name "native" for this purpose. Users who wish to have the very fastest software, at the expense of portability, can enable this variant either on a port by port basis or in their variants.conf file.


Reply | Threaded
Open this post in threaded view
|

Re: openmpi-* subports, and publishing of binaries

ryandesign2
Administrator


On Feb 3, 2021, at 03:29, Eric Borisch wrote:

> This seems off.
>
> Hwloc’s own page talks about the availability of pre-compiled packages, and its demos focus on determining what threads to bind to (on OSes where explicit binding is possible; last I checked it wasn’t on OSX) if you want to maximize or minimize shared resources. (You may want shared L2 cache between threads for cooperative tasks, or to put threads on cores with different memory busses - and bind memory allocs, too - if your code is burning through large, and separable, chunks of memory at max bandwidth.)
>
> It is designed to be used at runtime to provide a uniform (across OSes) way to interrogate the running system’s topology.
>
> This is separate from flags like -march which lock in instruction sets which will be used  — and can create decidedly non-portable binaries.


You can always ask the committer if he has any further explanation.

https://github.com/macports/macports-ports/commit/919fd5b27f1ca9c481250203e3dedbac433ecce4

https://github.com/macports/macports-ports/commit/f803f507183a2f0e92a0c976afd1ec7a1e0ad923

https://github.com/macports/macports-ports/commit/de413f22fba8d3e348ba6fdd21470505101f3dce