C# Threading Verification with CHESS

Carson Jones
CS6966 Project

Note: The information contained in this document pertains to the older version of CHESS (without data race detection). If I get some time in the future, I'll update the content using data race detection.

Introduction

Implementing and debugging parallel algorithms and data structures is a hard problem. This is mainly due to Heisenbugs--a class of bugs that are difficult to find and reproduce. This class of bugs includes race conditions, livelock, and deadlock in multithreaded programming. Verification tools for finding and reproducing Heisenbugs are becoming more mainstream. This project will investigate CHESS, a Microsoft Research tool being used for this purpose.

CHESS Overview

For this project, I used the CHESS integrated environment with Microsoft Visual Studio .NET (as opposed to the command line tool). The interface was really nice and usable. The general method is to create the class library to be tested, and a test project. Each method in the test project has attributes added to it which tell the CHESS library to attach to them during testing. CHESS also allows for additional testing parameters to be specified as attributes above the method definitions, such as the schedule string to reproduce a problem that was found.

CHESS mainly detects deadlocks and livelocks by itself. Because it produces all possible thread schedules, assertion statements added by the programmer makes it possible to detect race conditions. With the current release, CHESS is not actually capable of finding race conditions by itself. This capability is to be added in the future, as noted by the CHESS team on their website. This is a huge drawback for the current release. This means that the extent to which CHESS is capable of finding race conditions lies on how well or how exhaustively the programmer writes assert statements throughout the code. Throughout the testing for this project, it has been noticed that CHESS also needs a bit of suggestion as to where to preempt threads. Currently, CHESS only preempts a thread two times throughout an execution, and CHESS seems blind to which variables are being shared among threads. A suggestion can be given to CHESS by declaring all shared variables between threads as volatile, and by setting the "MonitorVolatiles" attribute to "true." This tells CHESS to also insert preemptions around volatiles.

When CHESS does find a deadlock, livelock, or when the runtime finds an assertion error, CHESS provides a nice output window to let the programmer know what happened, and better yet, the schedule string to reproduce the error without having to wait through all previous schedules. This schedule string is added as a method attribute, which allows use of the debugger to set breakpoints and gain visibility into the bug.

In my experience with CHESS so far, most of the deadlocks I've found are in my test harness code. This would be similar to using some data structure in applications code. There were situations where I thought I had coded something correctly but there turned out to be deadlock or livelock errors. This turned out to be very useful. I didn't find very many race errors (I inserted some to see how it would work). I feel that this is usually the case for many developers, critical sections seem to be well thought out and shared data seems to get protected. Most of the errors in my experience in multithreaded programming come from corner cases in the locking API that the developer isn't aware of, usually causing deadlock.

Overall Strengths and Weaknesses of CHESS

Note that I am not a CHESS expert, some of these issues may be able to be mitigated.

Algorithms Chosen

From the book, "The Art of Multiprocessor Programming" by Maurice Herlihy and Nir Shavit, the algorithms for the FIFO Read Write Lock, Unbounded Lock Free Queue, and Priority Queue Heap have been chosen. A description of each of these will be found in later sections.

Differences with Java and C#

As noted, the algorithms that are used in this project were taken from the course textbook, "The Art of Multiprocessor Programming" by Herlihy and Shavit. These algorithms are written in Java, meaning they needed to be translated into C#. Java has a much richer API for threading (as well as just about everything) than C# does, and there were no direct translations between Java code and C# code. To handle this, classes were written to mimick what Java provided in their API to get the algorithms to look similarly to what was provided in the book. In each of the project descriptions, if this was needed, it will be noted.

FIFOReadWriteLock

The FIFOReadWriteLock is the "fair" version of the read/write lock given in the textbook given in Ch. 8 on pages 186-187 in figures 8.10, 8.11, and 8.12 (with applicable modifications from the errata). This lock allows for multiple simultaneous readers or one writer. When the writer attempts to lock the RWLock, any subsequent readers attempting the lock the RWLock will be put into a wait loop. This is called a FIFO RWLock because when a writer requests the lock, the request blocks new readers allowing the writer to acquire the lock in FIFO order.

Sources and Solution:

Verification with CHESS:

The ReadWriteRunner class was written to test this RWLock. This class creates some number of readers and writers which will compete for the lock. The UnitTest class has a number of test methods which creates a different number of readers and writers. The translation of the FifoReadWriteLock was fairly straight forward. When verifying with CHESS I did find deadlocks in the ReadWriteRunner class when multiple writers had finished, and there was one reader waiting around on the condition variable that was not being signalled.

Using CHESS (example snapshots)


Unbounded Lock Free Queue

The unbounded lock free queue is the queue implementation in the book that advertises the use of no locks. This source in the book is in Ch. 10 on pages 230 - 231 in figures 10.9, 10.10, and 10.11 (with applicable modifications from the errata). The transation from Java was not direct. In C#, there is no such class for an AtomicReference, so I had to implement my own which simply uses locks. In C#, this isn't technically "lock free," but if there were such a class in C#, it would be the same as in Java.

Sources and Solution:

Verification with CHESS:

The ParallelTasks class is used to add generic producer and consumer threads to a pool of threads to be run concurrently (similar to the CHESS samples). The AtomicList class is an atomic version of the List type in C#, needed for testing assert statements between threads when detecting race errors. Finally, the UnitTest1 class contains the main unit tests.

With this example, I did try to check for data races (by deliberately inserting them). In the AtomicReference class, I removed the lock to try to get things to break. Unfortunately, no assertions were violated. After reading up on some of the CHESS documentation, CHESS can be configured to preempt around volatile variables. After setting this and declaring the shared variables to be volatile, I was able to see the race condition.


Fine-Grained Priority Queue Heap

The fine-grained priority queue heap is a ordered heap implemented as an array that allows for concurrent add and remove operations. The fine-grained property comes from being able to percolate items up and down the heap as a sequence of atomic steps, which can be interleaved with other such steps. For example, a removeMin operation can proceed and not be delayed if a previous add call is in the middle of percolating up the tree.

Sources and Solution:

Verification with CHESS:

The implementation of this data structure was tricky. I found a few deadlock situations that I haven't yet resolved. I feel that maybe this has something to do with my inexperience with the C# threading API. The solution will hopefully be found soon...


Conclusion

Overall, CHESS is a useful tool. Because multithreaded programming is becoming more mainstream, tools such as this are essential in the development process. These dynamic verification tools are not perfect however. In the case of CHESS, there is still the burdon on the programmer to write good race tests. Hopefully the new release of CHESS will bring better methods for finding these bugs, because the absence of races is still limited to how well the programmer can test for them. What CHESS does is increase the degree of confidence that tested software will perform as expected, though it is not exhaustive and could not advertise perfection. There is obviously much more work to go into tools such as CHESS.


Last modified: Mon July 5 15:246:52 MDT 2009