| Test Name | ISP Results |
| diff2.c This is the Diffusion 2D example. |
# Processes: 4 No Deadlocks 1 Interleaving output |
| monte.c This is the actual, unmodified, monte carlo code. |
# Processes: 4 # Processes: 5 # Processes: 6 |
| Game of Life (mlife) In this example, ISP looks for memory leaks. This example was presented at EuroPVM/MPI 2007 by Gropp and Lusk as a tutorial on advanced MPI-2, and has 522 lines of C code. We provide the results of running ISP on this example. |
# Processes: 4 # Processes: 9 |
| Game of Life (mlife2d) In this example, ISP looks for memory leaks. This example was presented at EuroPVM/MPI 2007 by Gropp and Lusk as a tutorial on advanced MPI-2, and has 552 lines of C code. We provide the results of running ISP on this example. |
# Processes: 4 # Processes: 9 |
Acknowledgements
We are grateful to Dr. Bronis R. de Supinski of the Lawrence Livermore National Laboratory for providing us the tests labeled "Umpire Tests" below.
Disclaimer
These experiments are being reported in the best interest of comparing tools on a scientific basis. No implication of the superiority of ISP over Marmot is expressed or intended. In fact, ISP is a new tool needing lot more improvements. These experiments only show that with a model-checking based search using representative interleavings (as provided by partial order reductions - our POE algorithm in particular) we seem to be able to find certain bugs always - as captured by these tests. Similar advantages of model checkers over testing tools are well known in model checking - our data is only to highlight that the same holds for MPI runtime verification also.
The table below represents the running of the Umpire test cases on three different platforms. All of the tests provide a copy-and-paste of the console session in which the tests where run. There weren't any special compile flags for Marmot and MPICH. I used the "marmotcc" and "mpicc" scripts in order to compile the tests.
The tests that end in "_mod" are slightly modified versions of the associated Umpire test. The modification is just a swap of ranks 1 and 2.
| # | Test Name | ISP using MPICH2 1.0.7 | Marmot 2.0.0 using MPICH2 1.0.7 | MPICH 1.2.6 |
| 1 | any_src-can-deadlock.c | No Deadlock Detected 2 Interleavings output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 2 | any_src-can-deadlock10.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 7 / 10 Runs output |
Deadlocked in 8 / 10 Runs output |
| 2.1 | any_src-can-deadlock10_mod.c | Deadlock Detected 2 Interleavings output |
Caught Deadlock in 5 / 10 Runs output |
Deadlocked in 3 / 10 Runs output |
| 3 | any_src-can-deadlock11.c | Deadlock Detected 2 Interleavings output |
Caught Deadlock in 5 / 10 Runs output |
Deadlocked in 3 / 10 Runs output |
| 4 | any_src-can-deadlock2.c | No Deadlock Detected 2 Interleavings output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 5 | any_src-can-deadlock3.c | No Deadlock Detected 2 Interleavings output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 6 | any_src-can-deadlock4.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 4 / 10 Runs output |
Deadlocked in 6 / 10 Runs output |
| 6.1 | any_src-can-deadlock4_mod.c | Deadlock Detected 2 Interleavings output |
Caught Deadlock in 3 / 10 Runs output |
Deadlocked in 5 / 10 Runs output |
| 7 | any_src-can-deadlock5.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 6 / 10 Runs output |
Deadlocked in 6 / 10 Runs output |
| 7.1 | any_src-can-deadlock5_mod.c | Deadlock Detected 2 Interleavings output |
Caught Deadlock in 6 / 10 Runs output |
Deadlocked in 4 / 10 Runs output |
| 8 | any_src-can-deadlock6.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 6 / 10 Runs output |
Deadlocked in 9 / 10 Runs output |
| 8.1 | any_src-can-deadlock6_mod.c | Deadlock Detected 2 Interleavings output |
Caught Deadlock in 4 / 10 Runs output |
Deadlocked in 3 / 10 Runs output |
| 9 | any_src-can-deadlock7.c | Deadlock Detected 2 Interleavings output |
Caught Deadlock in 5 / 10 Runs output |
Deadlocked in 2 / 10 Runs output |
| 10 | any_src-can-deadlock8.c | Deadlock Detected 2 Interleavings output |
Caught Deadlock in 6 / 10 Runs output |
Deadlocked in 3 / 10 Runs output |
| 11 | any_src-can-deadlock9.c | Deadlock Detected 2 Interleavings output |
Caught Deadlock in 5 / 10 Runs output |
Deadlocked in 1 / 10 Runs output |
| 12 | any_src-deadlock.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 13 | any_src-waitall-deadlock.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 14 | any_src-waitall-deadlock2.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 15 | any_src-waitall-deadlock3.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 16 | any_src-waitany-deadlock.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 17 | any_src-waitany-deadlock2.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 18 | any_src-wait-deadlock.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 19 | any_src-wait-deadlock2.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 20 | basic-deadlock.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 21 | basic-deadlock-comm_create.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 22 | basic-deadlock-comm_dup.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 23 | basic-deadlock-comm_split.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 24 | bcast-deadlock.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 25 | change-send-buffer.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 10 Runs output |
No Deadlock in 30 Runs output |
| 26 | collective-misorder.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 27 | collective-misorder2.c | Deadlock Detected 1 Interleaving output |
Didn't Catch Deadlock in 20 Runs output |
Didn't Deadlock in 30 Runs output |
| 28 | collective-misorder-allreduce.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 29 | comm-bcast-deadlock.c | Deadlock Detected 1 Interleaving output |
Didn't Catch Deadlock in 20 Runs output |
Didn't Deadlock in 30 Runs output |
| 30 | comm-deadlock.c | Deadlock Detected 1 Interleaving output |
Didn't Catch Deadlock in 20 Runs output |
Didn't Deadlock in 30 Runs output |
| 31 | comm-dup-no-error.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 32 | comm-dup-no-free.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 33 | comm-dup-no-free2.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 34 | comm-simple.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 35 | comm-split-no-free.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 36 | comm-translate-ranks.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 37 | complex-deadlock.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 38 | deadlock-config.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
Deadlocked in 10 / 10 Runs output |
| 38.1 | deadlock-config.c | In ISP's default mode, it does not catch the deadlock. A big enough buffer is assumed, so MPI_Send does not block. If the blocking MPI_Send option is enabled, it will catch the deadlock, as seen here. It is recommended to run the program using both options. |
||
| 39 | dropped-req.c | Deadlock Detected 1 Interleaving output |
Didn't Catch Deadlock in 15 Runs output |
Didn't Deadlock in 30 Runs output |
| 40 | finalize-deadlock.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 41 | hello.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 42 | irecv-deadlock.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 43 | irecv-isend-ok.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 44 | irecv-isend-ok2.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 45 | lost-request-waitall.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 10 Runs output |
No Deadlock in 30 Runs output |
| 46 | no-error.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 47 | no-error2.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 48 | no-error3.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 49 | no-error3-any_src.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 50 | no-error-any_src.c | No Deadlock Detected 2 Interleavings output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 51 | no-error-interleaved-isend.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 52 | no-error-test.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 53 | no-error-testall.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 54 | no-error-testany.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 55 | no-error-vector-isend.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 56 | no-error-wait.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 57 | no-error-waitall-any_src2.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 58 | no-error-waitall-any_src3.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 59 | no-error-waitany.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 60 | no-error-wait-any_src.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 61 | no-error-wait-any_src2.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 62 | no-error-wait-any_src3.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 63 | no-error-wait-any_src4.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 64 | no-error-waitany-any_src.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 65 | partial-recv.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 66 | sendrecv-deadlock.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 67 | send-recv-ok.c | No Deadlock Detected 1 Interleaving output |
No Deadlock in 20 Runs output |
No Deadlock in 30 Runs output |
| 68 | waitall-deadlock.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 69 | waitany-deadlock.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |
| 70 | wait-deadlock.c | Deadlock Detected 1 Interleaving output |
Caught Deadlock in 10 / 10 Runs output |
Deadlocked in 10 / 10 Runs output |