Consistency Management in Distributed Systems (4/17/2000) ============================================================================ BACKGROUND: Memory consistency model: a formal specification of how the memory system will appear to the program. --> often referred to as a "contract" between the programmer and the memory architecture Memory consistency protocol: protocol used by the memory system to implement the "contract" defined by the consistency model. --> for example, you can implement sequential consistency using either a write-update or a write-invalidate protocol (and many variants of any of the above) How do programmers and memory architects USE the consistency model? Programmers: tells them how much synchronization is needed to guarantee CORRECTNESS has large impact on PROGRAMMABILITY Architects: dictates the degree to which things can be done lazily has tremendous impact on the potential PERFORMANCE Is a memory consistency model only important for a multiprocessor (or a distributed system)? NO --> even inside uniprocessors there are opportunities for things to be done lazily or out of order (e.g., the interaction between DMA and write buffers). Memory coherence: a read to some location must always return the value most recently written to that location (only considers one location at a time). ========================================================================== SEQUENTIAL CONSISTENCY: Lamport (1979) proposed the SEQUENTIAL CONSISTENCY multiprocessor memory consistency model so that multiprocessors could be programmed similar to a multithreaded uniprocessor... DEFINITION: A multiprocessor system is sequentially consistent if the result of any execution is the same as if the operations were executed in SOME sequential order, and the operations of each individual processor appear in this sequence in the order specifies by its program. Note: SC does not say WHAT the agreed upon sequential order must be, just that one must exist. Two issues: - maintaining program order among operations from individual processors - maintaining a single sequential order among operations from all processors --> want to make it appear that each memory operation executes ATOMICALLY with respect to other operations Example: Dekker's Algorithm P1: P2: Flag1 = 1; Flag2 = 1; if (Flag2 == 0) if (Flag1 == 0) ... ... Issue: This algorithm counts on the fact that if P2 sees Flag1 equal to 0, then the write of (Flag2 = 1) will be seen on P1 when it checks Flag2. It should NOT be possible for both processors to see "0" when it reads the other flag. Example: P1: a = 1; P2: b = 1; P3: c = 1; print(b,c); print(a,c); print(a,b); What are the legal outputs? ========================================================================== IMPLEMENTING SEQUENTIAL CONSISTENCY Illustrate: - bus-based write-invalidate - bus-based write-update - directory-based write-invalidate Key observation: need to have a single instant in time when the data "universally" changes from the old value to the new one --> this "sequentializes" the accesses Questions for discussion: - Is such a strict model necessary? - What kind of performance impact can this have? * making everything SYNCHRONOUS makes it SLOW ========================================================================== WEAK CONSISTENCY: Idea: What if we separate out those variables that inherently involve race conditions (synchronization variables) from the shared data that the synchronization is used to protect? --> the synchronization points tell us WHEN consistency matters! --> we can propagate changes to non-synchronization ASYNCHRONOUSLY as long as we guarantee that the changes have propagated by the time that the synchronization operations would allow them to be seen Operational definition of weak consistency: 1. Accesses to synchronization variables are sequentially consistent. 2. No access to a synchronization variable is allowed to be performed until all previous writes have completed. 3. No data access (read or write) is allowed to be performed until all preceding synchronization accesses have been performed. Note: "Performed" has a very specific meaning in this model. >> A read is performed when no subsequent write can affect its value. >> A write is performed when all subsequent reads will return it. Key observations: -> we can now "pipeline" or "coalesce" the propagation of changes -> this model only works if there is enough "synchronization" to avoid race conditions Illustrate: -> directory-based weakly consistent write invalidate -> problem with race conditions: - polling on a shared variable - doing writes in a CS, but not the reads ========================================================================== RELEASE CONSISTENCY: Observation: logically, there is a difference between locking and unlocking (or, similarly, between entering and leaving a critical section). The inventors of RC extended WC to exploit this fact. ACQUIRE accesses tell the memory system that you need up to date values. RELEASE accesses tell the memory system that you have created new values. >> Want to propagate changes marked by "release" to readers who "acquire" the lock. >> Barriers act as both a release and an acquire. Operational definition of release consistency: 1. Before an ordinary access to a shared variable is performed, all previous acquires done by the process must have completed. 2. Before a release is allowed to be performed, all previous reads and writes done by the process must have completed. 3. Acquire and release operations must be processor consistent (sequential consistency is not strictly required). Illustrate: - pipelined write-invalidate - batched write-update (Munin) ========================================================================== LAZY RELEASE CONSISTENCY: Idea: rather than eagerly "synch up" data at a release point, why not "lazily" wait until the subsequent acquire? >> avoids invalidating/updating replicas that are not being used >> handles migratory data more efficiently (only get the new data when you need it) Tricky issues: - How do you locate the most up to date values? - When do you know you can drop your "diffs"? Original inventors of RC claim this is just a sub-case of RC, based on a somewhat stilted reading of the formal definition of RC. ========================================================================== OTHER MODELS Causal consistency: all processes see all causally-related shared accesses in the same order. PRAM: All processes see writes from each other processor in the order they were issued. Writes from different processors may not always be in the same order on different processors. (!) Processor consistency: PRAM + memory coherence. Entry consistency: shared data related to a specific critical region are made consistent when a critical region is entered --> breaks down when you want to share pieces of an array!