Correctness as spectrum not as binary.

In distributed system correctness is not about a program being complied well with the computer and it’s network, it’s only correctness can be only specified by it’s definition, assumptions and failure model.

One of the area in which we can see this happen is when system breaks the idea of single machine model and it’s assumptions. Networks can drop or reorder messages, clock drift, nodes can fail in mid operations and network partitions can occur. These are normal conditions, not rare edge cases.

Correctness in Distributed system can be understood within it’s spectrum of consistency models:

Linearizability: strongest; operations appear atomic and respect real-time order. It makes a distributed system behave as if there were one single machine. Every operation appears to happen instantly at some point between when it starts and when it finishes, and everyone sees operations in the same real-world order.

Sequential consistency: preserves each client’s operation order but relaxes real-time ordering. It is slightly weaker. Each individual client’s operations happen in the order that client performed them, but different clients do not necessarily see operations in real-time order.

Causal consistency: preserves the order of causally related operations. Focuses on relationships between operations. If one operation directly depends on another, everyone must see them in that order.

Eventual consistency: replicas converge, but reads may be stale or conflicting. Much weaker guarantee. Different replicas are allowed to temporarily contain different values. Reads can therefore return stale data.

No formal guarantees: systems may provide little or no consistency promise. The system does not promise a particular consistency behavior. A read might be stale, replicas could be temporarily disagree, and there may not even be a guarantee that replicas eventually converge.

Linearizability works in consensus mechanisms like Raft or Paxos which introduce latency and can reduce availability during partitions.

And in this spectrum of correctness, there is no universally “best” consistency model, each model has it’s own correctness, latency and coordination.

In this impossibility of results makes these tradeoffs fundamental, like in FLP, that deterministic consensus cannot be guaranteed in a fully asynchronous system with even one possible process failure, because consensus algorithms reply on assumptions like timeouts and bounded message delays.

In CAP theorem it take one approach about what happens when a distributed system experiences such things as network partition.

Suppose you have database servers:

Server A  ←── network ──→  Server B

Both normally communicate and keep their data synchronized.

Now the network breaks:

Server A      X      Server B

Server A cannot communicate with Server B. This is called a partition. At this point, the system has 3 desirable properties:

Consistency (C) means every client sees the same, most recent value.

Availability (A) means every request gets a response, even if some part of the system is failing.

Partition tolerance (P) means the system continues operating without caring communication failures between nodes.

The important part is that when a partition actually happens, you cannot simultaneously guarantee both strong Consistency and Availability.

Suppose the value is:

balance = $100

A client connected to Server A changes it:

balance = $50

But the partition prevents Server A from telling Server B.

Now another client asks Server B:

"What is the balance?"

Server B has two choices.

It can return:

$100

Then Server B remains available, but the answer is stale, violating strong consistency.

Or it can refuse to answer:

"Cannot safely determine the current value."

Then you preserve consistency, but sacrifice availability.

In this approach it’s about communication between replicas breaks, you must choose if you want to keep serving potentially stale data or stop serving until consistency can be restored.

The point is that partition tolerance is not really an optional feature in a distributed system. Networks can fail, so distributed systems have to deal with partitions. CAP is mostly about the C vs A decision during a partition.

A database may technically behave accordingly with it’s consistency model but the application sometimes expects stronger guarantees.

PACELC adds another trade-off method

CAP talks about what happens when things go wrong.

PACELC: What do you trade off when there is a partition, and what do you trade off when there isn’t?

The idea is:

P → A or C

If there is a Partition, choose between Availability and Consistency.

But:

E → L or C

Else, when there is no partition, choose between Latency and Consistency.

Why?

Suppose your database has replicas:

Client → Server A → Server B → Server C

You could make every write wait until all replicas confirm it.

That gives you stronger consistency:

Write
  ↓
A writes
  ↓
B confirms
  ↓
C confirms
  ↓
Response

But now the client has to wait for several machines and network round trips.

So:

More consistency
       ↓
More coordination
       ↓
Higher latency

Alternatively, Server A could immediately respond:

Client → A
        ↓
     "Success"

and synchronize with B and C afterward.

Now:

Lower latency
       ↓
Less coordination
       ↓
Potentially weaker consistency

This is important because distributed systems do not only make you choose during failures. Even under normal operation, stronger guarantees cost time.

So PACELC gives you a more complete way to think about distributed-system design:

Partition?
   yes → Availability vs Consistency

Partition?
   no  → Latency vs Consistency

That is why consistency is not simply “good” and eventual consistency is not simply “bad.” You are choosing a point in a multidimensional trade-off space.

CRDTs demonstrate explicit correctness by design

CRDTs are interesting ones because they approach the problem from a completely different direction. Normally, when multiple machines modify the same data, you have to coordinate them.

For example:

Server A: +1
Server B: +1

If both modify a shared counter, you have to make sure the updates don’t overwrite each other incorrectly.

One approach is:

A → coordinate with B
B → coordinate with A
A/B → agree on result

That coordination costs time and can become difficult during network partitions.

A CRDT instead tries to make the data structure itself mathematically safe to merge.

For example, suppose:

Replica A: +1
Replica B: +1

Instead of requiring A and B to coordinate immediately, each can perform its update independently.

Later:

A + B

The replicas merge their state.

The important property is that the merge operation is constructed so that it behaves correctly regardless of the order in which updates arrive.

The source describes three important mathematical properties:

Commutative: order does not matter.

A + B = B + A

Associative: grouping does not matter.

(A + B) + C = A + (B + C)

Idempotent: applying the same information again does not change the result.

merge(X, X) = X

Because the data structure has these properties, replicas can receive updates in different orders and still eventually converge to the same state. So the important idea isn’t about: “CRDTs are eventually consistent.”

It’s more about that you can design the data structure so that correctness is a mathematical property of the structure itself.

Rather than making sure that the distributed system handles conflicts correctly, you directly construct the system so that certain bad states are mathematically impossible.

This is the total different way of than the others.

A common mistake is:

Build system
   ↓
Add database
   ↓
Add replication
   ↓
Add retries
   ↓
Hope it is consistent

The distributed-systems approach should be closer to:

1. What does "correct" mean?
        ↓
2. What guarantees are required?
        ↓
3. What failures can happen?
        ↓
4. Which consistency model fits?
        ↓
5. What invariants must always hold?
        ↓
6. Design the implementation around those guarantees

For example, a banking system.

You might define:

Invariant:
An account must never spend money that does not exist.

That immediately tells you that weak, stale reads only be unacceptable for certain operations.

But for something like a social media like counter, exact instantaneous consistency may not matter:

User clicks Like
      ↓
counter temporarily differs between replicas
      ↓
replicas eventually converge

That may be perfectly acceptable.

A system is not simply “correct” or “incorrect” by itself. You first have to define what behavior you consider correct in the requirements in areas of consistency model, algorithms and failure handling. You cannot actually ask about if the distributed system is correct until you specify what that’s “correct” means in which parameters of each sections of the system.