C# Threading Verification with CHESS
Note: The information contained in this document pertains to the older version of CHESS (without data race detection). If I get some time in the future, I'll update the content using data race detection.
Implementing and debugging parallel algorithms and data structures is a hard problem. This is mainly due to Heisenbugs--a
class of bugs that are difficult to find and reproduce. This class of bugs includes race conditions, livelock, and deadlock in multithreaded programming. Verification tools for finding and reproducing Heisenbugs are becoming more mainstream. This project will investigate CHESS, a Microsoft Research tool being used for this purpose.
For this project, I used the CHESS integrated environment with Microsoft Visual Studio .NET (as opposed
to the command line tool). The interface was really nice and usable. The general method is to create the
class library to be tested, and a test project. Each method in the test project has attributes added to
it which tell the CHESS library to attach to them during testing. CHESS also allows for additional testing
parameters to be specified as attributes above the method definitions, such as the schedule string to reproduce
a problem that was found.
CHESS mainly detects deadlocks and livelocks by itself. Because it produces all possible thread schedules,
assertion statements added by the programmer makes it possible to detect race conditions. With the current
release, CHESS is not actually capable of finding race conditions by itself. This capability is to be added
in the future, as noted by the CHESS team on their website. This is a huge drawback for the current release.
This means that the extent to which CHESS is capable of finding race conditions lies on how well or how
exhaustively the programmer writes assert statements throughout the code. Throughout the testing for this project,
it has been noticed that CHESS also needs a bit of suggestion as to where to preempt threads. Currently,
CHESS only preempts a thread two times throughout an execution, and CHESS seems blind to which variables
are being shared among threads. A suggestion can be given to CHESS by declaring all shared variables between
threads as volatile, and by setting the "MonitorVolatiles" attribute to "true." This tells CHESS
to also insert preemptions around volatiles.
When CHESS does find a deadlock, livelock, or when the runtime finds an assertion error, CHESS provides a nice
output window to let the programmer know what happened, and better yet, the schedule string to reproduce the error
without having to wait through all previous schedules. This schedule string is added as a method attribute,
which allows use of the debugger to set breakpoints and gain visibility into the bug.
In my experience with CHESS so far, most of the deadlocks I've found are in my test harness code. This would be similar to using some data structure in applications code. There were situations where I thought I had coded something correctly but there turned out to be deadlock or livelock errors. This turned out to be very useful. I didn't find very many race errors (I inserted some to see how it would work). I feel that this is usually the case for many developers, critical sections seem to be well thought out and shared data seems to get protected. Most of the errors in my experience in multithreaded programming come from corner cases in the locking API that the developer isn't aware of, usually causing deadlock.
Overall Strengths and Weaknesses of CHESS
Note that I am not a CHESS expert, some of these issues may be able to be mitigated.
- Deadlock/Livelock Detection. CHESS handles this very well. In every instance that my code had one of these conditions, CHESS detected it and I could usually debug it. There were instances of deadlocks where I had trouble getting visibility into the states of the locks using the Monitor class, and couldn't tell whether the thread context switched as a result of the lock already being locked, or if it was a CHESS schedule switch.
- Integrated with VS.NET. The debugging capability that Visual Studio has is very good, adding CHESS to this is a big advantage.
- Replay. This is obviously a great feature for debugging.
- All Thread Schedules Explored. This is a big advantage for finding data races and deadlocks.
- Some tests inconclusive. In some instances I would get a message saying that the test was inconclusive. In many cases this happens if there is no multithreading behavior in the programs that were tested. I also got messages that the test had timed out, even for relatively simple programs (after about 20 minutes of running with ~150 lines and 4 threads competing to use a data structure). There is little documentation about these situations on the CHESS website.
- Some races undetected. In some instances, I would deliberately insert a race by removing a lock to a critical section, retest with my usual assert statements, and CHESS wouldn't catch the race. This was handled by putting "volatile" keywords around all of the shared variables between threads and turning on the "MonitorVolatiles" option.
- Difficult visibility into deadlocks. In some of my deadlocks, it was difficult to see whether a lock was actually locked, or CHESS was context switching the thread. I replaced Monitor.Enter statements with Monitor.TryEnter to get more visibility into lock states. It would be useful to have a tool that shows the state of locks and maybe with the line in source on which they were locked. This was mainly only a challenge for fine-grained locking on data structures with many nodes (many locks).
- Automatic race detection. This is coming in a future CHESS release I believe. I would love to see how it works, probably some sort of digest on memory or objects for each schedule and see if the values differ?
- Long running-time. This isn't exactly characteristic of only CHESS, but all tools that experience state space explosion. Some of my programs (quite small) took tens of minutes to run. I would be interested in whether CHESS is feasible for enterprise level software testing. My guess is that there is still work to do.
From the book, "The Art of Multiprocessor Programming" by Maurice Herlihy and Nir Shavit, the algorithms for the FIFO Read Write Lock, Unbounded Lock Free Queue, and Priority Queue Heap have been chosen. A description of each of these will be found in later sections.
Differences with Java and C#
As noted, the algorithms that are used in this project were taken from the course textbook, "The Art of Multiprocessor Programming" by Herlihy and Shavit. These algorithms are written in Java, meaning they needed to be translated
into C#. Java has a much richer API for threading (as well as just about everything) than C# does, and there were no
direct translations between Java code and C# code. To handle this, classes were written to mimick what Java provided in their API to get the algorithms to look similarly to what was provided in the book. In each of the project descriptions, if this was needed, it will be noted.
The FIFOReadWriteLock is the "fair" version of the read/write lock given in the textbook given in Ch. 8 on pages 186-187 in figures 8.10, 8.11, and 8.12 (with applicable modifications from the errata). This lock allows for multiple simultaneous readers or one writer. When the writer attempts to lock the RWLock, any subsequent readers attempting the lock the RWLock will be put into a wait loop. This is called a FIFO RWLock because when a writer requests the lock, the request blocks new readers allowing the writer to acquire the lock in FIFO order.
Sources and Solution:
Verification with CHESS:
The ReadWriteRunner class was written to test this RWLock. This class creates some number of readers and writers which will compete for the lock. The UnitTest class has a number of test methods which creates a different number of readers and writers. The translation of the FifoReadWriteLock was fairly straight forward. When verifying with CHESS I did find deadlocks in the ReadWriteRunner class when multiple writers had finished, and there was one reader waiting around on the condition variable that was not being signalled.
Using CHESS (example snapshots)
Unbounded Lock Free Queue
The unbounded lock free queue is the queue implementation in the book that advertises the use of no locks. This source in the book is in Ch. 10 on pages 230 - 231 in figures 10.9, 10.10, and 10.11 (with applicable modifications from the errata). The transation from Java was not direct. In C#, there is no such class for an AtomicReference, so I had to implement my own which simply uses locks. In C#, this isn't technically "lock free," but if there were such a class in C#, it would be the same as in Java.
Sources and Solution:
Verification with CHESS:
The ParallelTasks class is used to add generic producer and consumer threads to a pool of threads to be run concurrently (similar to the CHESS samples). The AtomicList class is an atomic version of the List type in C#, needed for testing assert statements between threads when detecting race errors. Finally, the UnitTest1 class contains the main unit tests.
With this example, I did try to check for data races (by deliberately inserting them). In the AtomicReference class, I removed the lock to try to get things to break. Unfortunately, no assertions were violated. After reading up on some of the CHESS documentation, CHESS can be configured to preempt around volatile variables. After setting this and declaring the shared variables to be volatile, I was able to see the race condition.
Fine-Grained Priority Queue Heap
The fine-grained priority queue heap is a ordered heap implemented as an array that allows for concurrent add and remove operations. The fine-grained property comes from being able to percolate items up and down the heap as a sequence of atomic steps, which can be interleaved with other such steps. For example, a removeMin operation can proceed and not be delayed if a previous add call is in the middle of percolating up the tree.
Sources and Solution:
Verification with CHESS:
The implementation of this data structure was tricky. I found a few deadlock situations that I haven't yet resolved. I feel that maybe this has something to do with my inexperience with the C# threading API. The solution will hopefully be found soon...
Overall, CHESS is a useful tool. Because multithreaded programming is becoming more mainstream, tools such as this are essential in the development process. These dynamic verification tools are not perfect however. In the case of CHESS, there is still the burdon on the programmer to write good race tests. Hopefully the new release of CHESS will bring better methods for finding these bugs, because the absence of races is still limited to how well the programmer can test for them. What CHESS does is increase the degree of confidence that tested software will perform as expected, though it is not exhaustive and could not advertise perfection. There is obviously much more work to go into tools such as CHESS.
Last modified: Mon July 5 15:246:52 MDT 2009