More Infiniband tips

  • Article

So, you've installed your infiniband cards, but got limited performance improvement for your MSMPI application. Before you look at the code again, you may want to have a look at whether the winsock direct providers were installed as well. In some cases, installing the adapter just gives you IP over Infiniband. Your vendor may have supplied specific WSD providers and utilities, in which case please refer back to their installation instructions.

If you are using the OpenFabrics (https://www.openfabrics.org) stack, keep in mind that:
- installsp –i installs wsd providers
- installsp –r removes them
- installsp –l lists what is available

There are also a number of system environment variables that may be useful in tuning the performance of that stack:

- IBWSD_POLL - Sets the number of times to poll the completion queue. Reduces latency at the cost of CPU utilization. Default is 500. If you use polling, make sure to set MPICH_DISABLE_SHM = 1 as well. This disables shared memory on the local computer and prevents two threads from polling the same completion queue.
- IBWSD_SA_RETRY - Sets the number of times to retry query requests. Default is 4. Increase if connection setup fails.
- IBWSD_SA_TIMEOUT - Sets the number of milliseconds to wait before retrying query requests. Default is 4. Increase if connection setup fails.
- IBWSD_NO_IPOIB - Setting this environment variable to <> 0  prevents fall back to IPoIB.
- Cluster variable MPICH_SOCKET_SBUFFER_SIZE: Send buffer size for  socket and shared memory channel used by MSMPI. Default = 32KB. Set to 0 (e.g. with cluscfg  setenvs) to avoid copy on send and increase b/w use. Use ONLY with WSD and IB. It can cause problems with TCP/IP stack.

Also, remember that you can set the MPI network at a job level by using the MPICH_NETMASK variable, e.g.:

mpiexec -env MPICH_NETMASK <network address>/<mask> myApp.exe

mpiexec is also a good way of setting the IB stack variables, e.g.:

mpiexec -env MPICH_SOCKET_SBUFFER_SIZE 0 -env IBWSD_POLL 500 -env MPICH_DISABLE_SHM 1 myapp.exe