ceremonyclient/pebble/docs/range_deletions.md

# Range Deletions

TODO: The following explanation of range deletions does not take into account
the recent change to prohibit splitting of a user key between sstables. This
change simplifies the logic, removing 'improperly truncated range tombstones.'

TODO: The following explanation of range deletions ignores the
kind/trailer that appears at the end of keys after the sequence
number. This should be harmless but need to add a justification on why
it is harmless.

## Background and Notation

Range deletions are represented as `[start, end)#seqnum`. Points
(set/merge/...) are represented as `key#seqnum`. A range delete `[s, e)#n1`
deletes every point `k#n2` where `k \in [s, e)` and `n2 < n1`.
The inequality `n2 < n1` is to handle the case where a range delete and
a point have the same sequence number -- this happens during sstable
ingestion where the whole sstable is assigned a single sequence number
that applies to all the data in it.

There is additionally an infinity sequence number, represented as
`inf`, which is not used for any point, that we can use for reasoning
about range deletes.

It has been asked why range deletes use an exclusive end key instead
of an inclusive end key. For string keys, one can convert a desired
range delete on `[s, e]` into a range delete on `[s, ImmediateSuccessor(e))`.
For strings, the immediate successor of a key
is that key with a \0 appended to it. However one cannot go in the
other direction: if one could represent only inclusive end keys in a
range delete and one desires to delete a range with an exclusive end
key `[s, e)#n`, one needs to compute `ImmediatePredecessor(e)` which
is an infinite length string. For example,
`ImmediatePredecessor("ab")` is `"aa\xff\xff...."`. Additionally,
regardless of user needs, the exclusive end key helps with splitting a
range delete as we will see later. 

We will sometimes use ImmediatePredecessor and ImmediateSuccessor in
the following for illustrating an idea, but we do not rely on them as
something that is viable to produce for a particular kind of key. And
even if viable, these functions are not currently provided to
RockDB/Pebble.

### Visualization

If we consider a 2 dimensional space with increasing keys on the X
axis (with every possible user key represented) and increasing
sequence numbers on the Y axis, range deletes apply to a rectangle
whose bottom edge sits on the X axis.

The actual space represented by the ordering in our sstables is a one
dimensional space where `k1#n1` is less than `k2#n2` if either of the
following holds:

- k1 < k2

- k1 = k2 and n1 > n2 (under the assumption that no two points with
the same key have the same sequence number).

```
  ^
  |   .       > .        > .        > yy
  |   .      >  .       >  .       >  .
  |   .     >   .      >   .      >   .
n |   V    >    xx    >    .     >    V
  |   .   >     x.   >     x.   >     . 
  |   .  >      x.  >      x.  >      .
  |   . >       x. >       x. >       .
  |   .>        x.>        x.>        .
  ------------------------------------------>
                k        IS(k)    IS(IS(k))
```

The above figure uses `.` to represent points and the X axis is dense in
that it represents all possible keys. `xx` represents the start of a
range delete and `x.` are the points which it deletes. The arrows `V` and
`>` represent the ordering of the points in the one dimensional space.
`IS` is shorthand for `ImmediateSuccessor` and the range delete represented
there is `[k, IS(IS(k)))#n`. Ignore `yy` for now.

The one dimensional space works fine in a world with only points. But
issues arise when storing range deletes, that represent an action in 2
dimensional space, into this one dimensional space.

## Range Delete Boundaries and the Simplest World

RocksDB/Pebble store the inclusive bounds of each sstable in one dimensional
space. The range deletes two dimensional behavior and exclusive end key needs
to be adapted to this requirement. For a range delete `[s, e)#n`,
the smallest key it acts on is `s#(n-1)` and the largest key it
acts on is `ImmediatePredecessor(e)#0`. So if we position the range delete
immediately before the smallest key it acts on and immediately after
the largest key it acts on we can give it a tight inclusive bound of
`[s#n, e#inf]`.  

Note again that this range delete does not delete everything in its
inclusive bound. For example, range delete `["c", "h")#10` has a tight
inclusive bound of `["c"#10, "h"#inf]` but does not delete `"d"#11`
which lies in that bound. Going back to our earlier diagram, the one
dimensional inclusive bounds go from the `xx` to `yy` but there are
`.`s in between, in the one dimensional order, that are not deleted.

This is the reason why one cannot in general
use a range delete to seek over all points within its bounds. The one
exception to this seeking behaviour is that when we can order sstables
from new to old, one can "blindly" use this range delete in a newer
sstable to seek to `"h"` in all older sstables since we know those
older sstables must only have point keys with sequence numbers `< 10`
for the keys in interval `["c", "h")`. This partial order across
sstables exists in RocksDB/Pebble between memtable, L0 sstables (where
it is a total order) and across sstables in different levels.

Coming back to the inclusive bounds of the range delete, `[s#n, e#inf]`:
these bounds participate in deciding the bounds of the
sstable. In this world, one can read all the entries in an sstable and
compute its bounds. However being able to construct these bounds by
reading an sstable is not essential -- RocksDB/Pebble store these
bounds in the `MANIFEST`. This latter fact has been exploited to
construct a real world (later section) where the bounds of an sstable
are not computable by reading all its keys.

If we had a system with one sstable per level, for each level lower
than L0, we are effectively done. We have represented the tight bounds
of each range delete and it is within the bounds of the sstable. This
works even with L0 => L0 compactions assuming they output exactly one
sstable.

## The Mostly Simple World

Here we have multiple files for levels lower than L0 that are non
overlapping in the file bounds. These multiple files occur because
compactions produce multiple files. This introduces the need to split a
range delete across the files being produced by a compaction.

There is a clean way to split a range delete `[s, e)#n` into 2 parts
(which can be recursively applied to split into arbitrarily many
parts): split into `[s, m)#n` and `[m, e)#n`. These range deletes
apply to non-overlapping points and their tight bounds are `[s#m,
m#inf]`, `[m#n, e#inf]` which are also non-overlapping.

Consider the following example of an input range delete `["c", "h")#10` and
the following two output files from a compaction:

```
          sst1            sst2
last point is "e"#7 | first point is "f"#20
```

The range delete can be split into `["c", "f")#10` and `["f",
"h")#10`, by using the first point key of sst2 as the split
point. Then the bounds of sst1 and sst2 will be `[..., "f"#inf]` and
`["f"#20, ...]` which are non-overlapping. It is still possible to compute
the sstable bounds by looking at all the entries in the sstable.

## The Real World

Continuing with the same range delete `["c", "h")#10`, we can have the
following sstables produced during a compaction:

```
         sst1       sst2         sst3        sst4     sst5
points: "e"#7 | "f"#12 "f"#7 | "f"#4 "f"#3 | "f"#1 | "g"#15
```

The range deletes written to these ssts are

```
      sst1           sst2            sst3           sst4          sst5
["c", "h")#10 | ["f", "h")#10 | ["f", "h")#10 | ["f", "h")#10 | ["g", "h")#10
```

The Pebble code that achieves this effect is in
`rangedel.Fragmenter`. It is a code structuring artifact that sst1
does not contain a range delete equal to `["c", "f")#10` and sst4 does
not contain `["f", "g")#10`. However for the range deletes in sst2 and
sst3 we cannot do any better because we don't know what the key
following "f" will be (the compaction cannot look ahead) and because
we don't have an `ImmediateSuccessor` function (otherwise we could
have written `["f", ImmediateSuccessor("f"))#10` to sst2, sst3). But
the code artifacts are not the ones introducing the real complexity.

The range delete bounds are

```
      sst1        sst2, sst3, sst4          sst5
["c"#10, "h"#inf] ["f"#10, "h"#inf]   ["g"#10, "h"#inf]

```

We note the following:

- The bounds of range deletes are overlapping since we have been
  unable to split the range deletes. If these decide the sstable
  bounds, the sstables will have overlapping bounds. This is not
  permissible.

- The range deletes included in each sstable result in that sstable
  being "self-sufficient" wrt having the range delete that deletes
  some of the points in the sstable (let us assume that the points in
  this example have not been dropped from that sstable because of a
  snapshot).

- The transitions from sst1 to sst2 and sst4 to sst5 are **clean** in
  that we can pretend that the range deletes in those files are actually:

```
      sst1           sst2            sst3           sst4          sst5
["c", "f")#10 | ["f", "g")#10 | ["f", "g")#10 | ["f", "g")#10 | ["g", "h")#10
```

We could achieve some of these **clean** transitions (but not all) with a
code change. Also note that these better range deletes maintain the
"self-sufficient" property.

### Making Non-overlapping SSTable bounds

We force the sstable bounds to be non-overlapping by setting them to:

```
      sst1              sst2           sst3            sst4              sst5
["c"#10, "f"#inf] ["f"#12, "f"#7] ["f"#4, "f"#3] ["f"#1, "g"#inf] ["g"#15, "h"#inf]
```

Note that for sst1...sst4 the sstable bounds are smaller than the
bounds of the range deletes contained in them. The code that
accomplishes this is Pebble is in `compaction.go` -- we will not discuss the
details of that logic here but note that it is placing an `inf`
sequence number for a clean transition and for an unclean transition
it is using the point keys as the bounds.

Associated with these narrower bounds, we add the following
requirement: a range delete in an sstable must **act-within** the bounds of
the sstable it is contained in. In the above example:

- sst1: range delete `["c", "h")#10` must act-within the bound `["c"#10, "f"#inf]`

- sst2: range delete `["f", "h")#10` must act-within the bound `["f"#12, "f"#7]`

- sst3: range delete `["f", "h")#10` must act-within the bound `["f"#4, "f"#3]`

- sst4: range delete `["f", "h")#10` must act-within the bound ["f"#1, "g"#inf]

- And so on.

The intuitive reason for the **act-within** requirement is that 
sst5 can be compacted and moved down to a lower level independent of
sst1-sst4, since it was at a **clean** boundary. We do not want the
range delete `["f", "h")#10` sitting in sst1...sst4 at the higher
level to act on `"g"#15` that has been moved to the lower level. Note
that this incorrect action can happen due to 2 reasons:
  
1. the invariant that lower levels have older data for keys
   that also exist in higher levels means we can (a) seek a lower level
   sstable to the end of a range delete from a higher level, (b) for a key
   lookup, stop searching in lower levels once a range delete is encountered
   for that key in a higher level.
  
2. Sequence number zeroing logic can change the sequence number of
  `"g"#15` to `"g"#0` (for better compression) once it realizes that
   there are no older versions of `"g"`. It would be incorrect for this
  `"g"#0` to be deleted.  


#### Loss of Power

This act-within behavior introduces some "loss of power" for
the original range delete `["c", "h")#10`. By acting within sst2...sst4
it can no longer delete keys `"f"#6`, `"f"#5`, `"f"#2`.

Luckily for us, this is harmless since these keys cannot have existed
in the system due to the levelling behavior: we cannot be writing
sst2...sst4 to level `i` if versions of `"f"` younger than `"f"#4` are
already in level `i` or version older than `"f"#7` have been left in
level i - 1. There is some trickery possible to prevent this "loss of
power" for queries (see the "Putting it together" section), but given
the history of bugs in this area, we should be cautious.

### Improperly truncated Range Deletes

We refer to range deletes that have experienced this "loss of power"
as **improper**. In the above example the range deletions in sst2, sst3, sst4
are improper. The problem with improper range deletions occurs
when they need to participate in a future compaction: even though we
have restricted them to act-within their current sstable boundary, we
don't have a way of **"writing"** this restriction to a new sstable,
since they still need to be written in the `[s, e)#n` format.

For example, sst2 has delete `["f", "h")#10` that must act-within
the bound `["f"#12, "f"#7]`. If sst2 was compacted down to the next
level into a new sstable (whose bounds we cannot predict because they
depend on other data written to that sstable) we need to be able to
write a range delete entry that follows the original restriction. But
the narrowest we can write is `["f", ImmediateSuccessor("f"))#10`. This
is an expansion of the act-within restriction with potentially
unintended consequences. In this case the expansion happened in the suffix.
For sst4, the range deletion `["f", "h")#10` must act-within `["f"#1, "g"#inf]`,
and we can precisely represent the constraint on the suffix by writing
`["f", "g")#10` but it does not precisely represent that this range delete
should not apply to `"f"#9`...`"f"#2`.

In comparison, the sst1 range delete `["c", "h")#10` that must act-within
the bound `["c"#10, "f"#inf]` is not improper. This restriction can
be applied precisely to get a range delete `["c", "f")#10`. 

The solution to this is to note that while individual sstables have
improper range deletes, if we look at a collection of sstables we
can restore the improper range deletes spread across them to their proper self
(and their full power). To accurately find these improper range
deletes would require looking into the contents of a file, which is
expensive. But we can construct a pessimistic set based on
looking at the sequence of all files in a level and partitioning them:
adjacent files `f1`, `f2` with largest and smallest bound `k1#n1`,
`k2#n2` must be in the same partition if

```
k1 = k2 and n1 != inf
```

In the above example sst2, sst3, sst4 are one partition. The
**spanning bound** of this partition is `["f"#12, "g"#inf]` and the
range delete `["f", "h")#10` when constrained to act-within this
spanning bound is precisely the range delete `["f",
"g")#10`. Intuitively, the "loss of power" of this range delete has
been restored for the sake of making it proper, so it can be
accurately "written" in the output of the compaction (it may be
improperly fragmented again in the output, but we have already
discussed that). Such partitions are called "atomic compaction groups"
and must participate as a whole in a compaction (and a
compaction can use multiple atomic compaction groups as input).

Consider another example:

```
          sst1              sst2
points:  "e"#12         |  "e"#10
delete: ["c", "g")#8    | ["c", "g")#8
bounds  ["c"#8, "e"#12] | ["e"#10, "g"#inf]
```

sst1, sst2 are an atomic compaction group. Say we violated the
requirement that both be inputs in a compaction and only compacted
sst2 down to level `i + 1` and then down to level `i + 2`. Then we add
sst3 with bounds `["h"#10, "j"#5]` to level `i` and sst1 and sst3 are
compacted to level `i + 1` into a single sstable. This new sstable
will have bounds `["c"#8, "j"#5]` so these bounds do not help with the
original apply-witin constraint on `["c", "g")#8` (that it should
apply-within `["c"#8, "e"#12]`). The narrowest we can construct (if we had
`ImmediateSuccessor`) would be `["c", ImmediateSuccessor("e"))#8`.  Now we
can incorrectly apply this range delete that is in level `i + 1` to `"e"#10`
sitting in level `i + 2`. Note that this example can be made worse using
sequence number zeroing -- `"e"#10` may have been rewritten to `"e"#0`.  

If a range delete `[s, e)#n` is in an atomic compaction group with
spanning bounds `[k1#n1, k2#n2]` our construction above guarantees the
following properties

- `k1#n1 <= s#n`, so the bounds do not constrain the start of the
  range delete.

- `k2 >= e` or `n2 = inf`, so if `k2` is constraining the range delete
  it will properly truncate the range delete.


#### New sstable at sequence number 0

A new sstable can be assigned sequence number 0 (and be written to L0)
if the keys in the sstable are not in any other sstable. This
comparison uses the keys and not key#seqnum, so the loss and
restoration of power does not cause problems since that occurs within
the versions of a single key.

#### Flawed optimizations

For the case where the atomic compaction group correspond to the lower
level of a compaction, it may initially seem to be correct to use only
a prefix or suffix of that group in a compaction. In this case the
prefix (suffix) will correspond to the largest key (smallest key) in
the input sstables in the compaction and so can continue to constrain
the range delete.  For example, sst1 and sst2 are in the same atomic
compaction group

```
          sst1               sst2
points: "c"#10 "e"#12    |  "e"#10
delete: ["c", "g")#8     | ["c", "g")#8
bounds  ["c"#10, "e"#12] | ["e"#10, "g"#inf]
```

and this is the lower level of a compaction with

```
          sst3
points: "a"#14 "d"#15
bounds  ["a"#14, "d"#15]
```

we could allow for a compaction involving sst1 and sst3 which would produce

```
          sst4
points: "a"#14 "c"#10 "d"#15 "e"#12
delete: ["c", "g")#8
bounds  ["a"#14, "e"#12]
```

and the range delete is still improper but its act-within constraint has
not expanded.

But we have to be very careful to not have a more significant loss of power
of this range delete. Consider a situation where sst3 had a single delete
`"e"#16`. It still does not overlap in bounds with sst2 and we again pick
sst1 and sst3 for compaction. This single delete will cause `"e"#12` to be deleted
and sst4 bounds would be (unless we had complicated code preventing it):

```
          sst4
points: "a"#14 "c"#10 "d"#15
delete: ["c", "g")#8
bounds  ["a"#14, "d"#15]
```

Now this delete cannot delete `"dd"#6` and we have lost the ability to know
that sst4 and sst2 are in the same atomic compaction group.


### Putting it together

Summarizing the above, we have:

- SStable bounds logic that ensures sstables are not
overlapping. These sstables contain range deletes that extend outside
these bounds. But these range deletes should **apply-within** the
sstable bounds.

- Compactions: they need to constrain the range deletes in the inputs
to **apply-within**, but this can create problems with **writing** the
**improper** range deletes. The solution is to include the full
**atomic compaction group** in a compaction so we can restore the
**improper** range deletes to their **proper** self and then apply the
constraints of the atomic compaction group.

- Queries: We need to act-within the file bound constraint on the range delete.
  Say the range delete is `[s, e)#n` and the file bound is `[b1#n1,
  b2#n2]`. We are guaranteed that `b1#n1 <= s#n` so the only
  constraint can come from `b2#n2`.
  
  - Deciding whether a range delete covers a key in the same or lower levels.

    - `b2 >= e`: there is no act-within constraint.
    - `b2 < e`: to be precise we cannot let it delete `b2#n2-1` or
      later keys. But it is likely that allowing it to delete up to
      `b2#0` would be ok due to the atomic compaction group. This
      would prevent the so-called "loss of power" discussed earlier if
      one also includes the argument that the gap in the file bounds
      that also represents the loss of power is harmless (the gap
      exists within versions of key, and anyone doing a query for that
      key will start from the sstable to the left of the gap). But it
      may be better to be cautious.

  - For using the range delete to seek sstables at lower levels.
    - `b2 >= e`: seek to `e` since there is no act-within constraint.
    - `b2 < e`: seek to `b2`. We are ignoring that this range delete
      is allowed to  delete some versions of `b2` since this is just a
      performance optimization.