# MPI IV: MPI pitfalls and non-blocking communication

Today's [assignment](https://classroom.github.com/a/BMEL_lUe).

The assignment repo is based on last class's repo, and contains the additional
series of commits that I talked about last class, about organizing code (in
this case related to MPI 1-d domain decomposition) in order to avoid duplication
and separate computational/parallelization aspects from the numerical
implementation.

## MPI pitfalls

You may well already have noticed that when using MPI communications, it's not too
difficult to get our programs to misbehave (of course, it's not fair to only
blame MPI, it's typically a problem with how we use it). This may take the form
of your program crashing with some error message, or it may lead to the program
hanging indefinitely, doing nothing (nothing useful, anyway). In which case,
Ctrl-C will interrupt and kill it, by the way.

Sometimes the problems can be quite subtle, and a code which works just fine on
one machine, or for one case, stops working when moving to another machine, or
when changing parameters, like the size of your domain.

```cpp

#include <mpi.h>

#include <vector>

int main(int argc, char** argv)
{
  MPI_Init(&argc, &argv);

  int rank, size;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);

  // Make sure we're being run on (exactly) 2 procs
  if (size != 2) {
    if (rank == 0) {
      std::cerr << "This program needs to be run on exactly 2 MPI processes.\n";
    }
    MPI_Abort(MPI_COMM_WORLD, 1);
  }

  if (argc != 2) {
    // This is a nice way of checking for an error
    if (rank == 0) {
      std::cerr << "Usage: " << argv[0] << " <N>\n"
                << "where <N> is the size of the message to exchange.\n";
    }
    MPI_Abort(MPI_COMM_WORLD, 1);
  }
  int N = atoi(argv[1]);

  std::vector<double> buf_send(N), buf_recv(N);

  if (rank == 0) {
    MPI_Send(buf_send.data(), N, MPI_DOUBLE, 1, 1234, MPI_COMM_WORLD);
    MPI_Recv(buf_recv.data(), N, MPI_DOUBLE, 1, 1234, MPI_COMM_WORLD,
             MPI_STATUS_IGNORE);
  } else if (rank == 1) {
    MPI_Send(buf_send.data(), N, MPI_DOUBLE, 0, 1234, MPI_COMM_WORLD);
    MPI_Recv(buf_recv.data(), N, MPI_DOUBLE, 0, 1234, MPI_COMM_WORLD,
             MPI_STATUS_IGNORE);
  }

  MPI_Finalize();
  return 0;
}
```

#### Your turn

This program demonstrates one of the problems you may encounter. It requires to
be run on exactly two processors, otherwise it'll complain right away. It just
exchanges messages between the two processors, and it makes it easy to change
the size of the message on the command line. I think you may find it's quite
similar in principle to the code you (and I) wrote to fill ghost points, though
it sends more than just a single ghost point value.

* Add the code above to the repo, by copy & pasting it to a new file `mpi_dl_test.cxx` in the assignment repo and adding it to the build. Run it like

```sh
[kai@mbpro mpi]$ mpirun -n 2 ./mpi_dl_test 10
```

In this case 10 double precision numbers will be sent between the two procs.

- Try different message sizes by changing the '10' on the command line to other
  numbers. What do you observe? Why? (If you don't find anything weird, you may
  have to change your message size more substantially.) You may want to put in
  some print statements to see what's up.

- How can you fix the problem?

#### Your turn

- Fix the `mpi_dl_test` code so that it's always correct.

## UNH's "Premise" supercomputer

UNH operates a cluster called "Premise", which is more or less a traditional cluster of
commodity hardware. It is available for use by UNH students, faculty and staff, and is a good place to get a feel for using a real supercomputer, learn how to use a batch system, have a system that one can actually perform some parallel scaling analysis without interference, or use it for your final project.

It's optional, but if you want to get more information and possibly get an account, you can check out more information here: <https://premise.sr.unh.edu/>, including how to request an account. If you do want to request an account, please let me know as well, so I can help coordinate with the Research Computing Center on how to make that work efficiently.

## Homework

- As usual, follow the steps above, make commits and add comments in the
  feedback pull request. After the solutions are posted, add a reflection
  statement.