a more consistent implementation for diffs across the board

Author: ztellman

README.md

@@ -4,7 +4,7 @@
 <dependency>
   <groupId>io.lacuna</groupId>
   <artifactId>bifurcan</artifactId>
-  <version>0.2.0-alpha1</version>
+  <version>0.2.0-alpha4</version>
 </dependency>
 ```
 
@@ -16,13 +16,14 @@ This library provides high-quality Java implementations of mutable and immutable
 * customizable equality semantics
 * contiguous memory used wherever possible
 * performance equivalent to, or better than, existing alternatives
+* changes to a collection can be tracked in a **diff** data structure, which can be subsequently rebased onto a different collection
 * [ALPHA] durable (disk-backed) representations which share the API and asymptotic performance of their in-memory counterparts
 
 Rather than using the existing collection interfaces in `java.util` such as `List` or `Map`, it provides its own interfaces (`IList`, `IMap`, `ISet`) that provide functional semantics - each update to a collection returns a reference to a new collection.  Each interface provides a method (`toList`, `toMap`, `toSet`) for coercing the collection to a read-only version of the standard Java interfaces.
 
 ### what makes this better?
 
-Some aspects of this library, like the inverted indices and durable collections, are unique.  
+Some aspects of this library, like the inverted indices, diffs, and durable collections, are unique.  
 
 There are, however, many existing implementations of "functional" (aka persistent, immutable) data structures on the JVM.  As shown in [these in-depth comparisons](https://github.com/lacuna/bifurcan/blob/master/doc/comparison.md), Bifurcan's performance is equivalent to the best existing implementations for basic operations, and significantly better for batch operations such as `union`, `intersection`, and `difference`.
 
@@ -80,28 +81,64 @@ for (int i = 0; i < 1000; i++) {
 }
 ```
 
-If we call `forked()` on this collection, it will be wrapped in an immutable "diff" facade which tracks changes without touching the underlying collection.  These facades have similar performance to typical collections, but do not support efficient set operations.
+If we call `forked()` on this collection, it will be wrapped in a **diff** facade, which is described below.
 
 ### virtual collections
 
-These facades also allow us to define collections programmatically:
+Bifurcan offers a variety of collection implementations, but you can also create your own by implementing a handful of methods.
+
+A list, at its base, is just a `size` and a function that, given an index, returns the corresponding element.  This can be constructed using the `Lists.from` method:
 
 ```java
-// a list of numbers within [0,1e6)
 IList<Long> list = Lists.from(1_000_000, i -> i);
-	
-// the set of numbers within [0,1e6)
-ISet<Long> set = Sets.from(list, i -> (0 <= i && i < list.size()) ? OptionalLong.of(i) : OptionalLong.empty());
-	
-// a map of numbers within [0,1e6) onto their square
-IMap<Long, Long> map = Maps.from(set, i -> i * i);
 ```
 
-These collections are not realized in-memory, and can be used as a translation layer for other data structure implementations.  Using our facades, however, we can still update them like any other collection, and only those changes will be directly represented in-memory.
+This creates a list of the numbers within `[0, 1e6)` without any of the elements being stored in memory.  All of the other operations associated with lists (adding and removing elements, updating elements, concatenating other lists, and so on) have efficient default implementations, which will be discussed in the next section.
+
+An unsorted set is just a list of elements, plus a function that, given an value, returns an `OptionalLong` describing the index of that element:
+
+```java
+Function<Long, OptionalLong> indexOf = n -> (0 <= n && n < list.size()) ? OptionalLong.of(i) : OptionalLong.empty();
+
+ISet<Long> set = Sets.from(list, indexOf)
+```
+
+A sorted set, conversely, is a list of elements, a comparator, and a function that, given a value, returns an `OptionalLong` describing the index of the closest element which equal to or less than that value (referred to as the "floor index"):
+
+```java
+Function<Double, OptionalLong> floorIndexOf = n -> indexOf.apply((long) n);
+
+ISet<Double> sortedSet = Sets.from(list, Comparator.naturalOrder(), floorIndexOf);
+```
+
+Sorted and unsorted maps are just their corresponding sets, plus a function from key to value.  These can be constructed using `Maps.from`, or by calling `zip` on a set:
+
+```java
+IMap<Long, Double> squareRoots = set.zip(n -> Math.sqrt(n))
+```
+
+### diffs
+
+These virtual collections can be modified just like any other Bifurcan collection:
+
+```java
+Lists.from(1, x -> 1).addLast(42)
+// [1, 42]
+```
+
+This is made possible by [diffs](https://lacuna.io/docs/bifurcan/io/lacuna/bifurcan/IDiff.html), which track changes on an immutable **underlying** collection.  Diff implementations exists for all variants of Bifurcan collections, and share the asymptotic performance of their normal counterparts.  By calling `diff()` on any collection, we create a diff wrapper whose changes can then be **rebased** onto a new underlying collection:
+
+```java
+IList<Integer> numDiff = List.of(1, 2, 3).diff().removeFirst().addLast(42)
+// [2, 3, 42]
+
+IList<Integer> rebased = numDiffs.rebase(List.of(4, 5, 6))
+// [5, 6, 42]
+```
 
 ### durable collections
 
-All in-memory structures can be saved to disk, while retaining the same API and asymptotic performance.  These durable collections are optimized for reads and batched writes, which means they are not a replacement for general-purpose databases, but they are still [useful in a variety of applications](doc/durable.md).
+All in-memory structures can be also saved to disk, while retaining the same API and asymptotic performance.  These durable collections are optimized for reads and batched writes, which means they are not a replacement for general-purpose databases, but they are still [useful in a variety of applications](doc/durable.md).
 
 ### no lazy collections
 

src/io/lacuna/bifurcan/FloatMap.java

@@ -122,7 +122,7 @@ public OptionalLong floorIndex(double key) {
   }
 
   @Override
-  public OptionalLong floorIndex(Double key) {
+  public OptionalLong inclusiveFloorIndex(Double key) {
     return floorIndex((double) key);
   }
 
@@ -139,8 +139,7 @@ public FloatMap<V> slice(double min, double max) {
     return new FloatMap<>(map.slice(doubleToLong(min), doubleToLong(max)));
   }
 
-  @Override
-  public FloatMap<V> slice(Double min, Double max) {
+  public FloatMap<V> sliceReal(Double min, Double max) {
     return slice((double) min, (double) max);
   }
 

src/io/lacuna/bifurcan/IDiff.java

@@ -1,5 +1,20 @@
 package io.lacuna.bifurcan;
 
-public interface IDiff<C extends ICollection<C, V>, V> {
+/**
+ * A generic interface for diffs, which represent changes to an underlying collection, and which can be rebased atop
+ * a new underlying collection.
+ */
+public interface IDiff<C> {
+
+  /**
+   * The underlying collection
+   */
   C underlying();
+
+  /**
+   * Returns a new diff, which is rebased atop the new underlying collection. The returned diff may not reflect any
+   * changes which cannot be applied to the new collection (removing an element which isn't present in the new
+   * collection, for instance), and so {@code a.rebase(b).rebase(c) } is not necessarily equivalent to {@code a.rebase(c) }.
+   */
+  IDiff<C> rebase(C newUnderlying);
 }

src/io/lacuna/bifurcan/IDiffList.java

@@ -4,34 +4,50 @@
 
 import java.util.Iterator;
 
-public interface IDiffList<V> extends IList<V>, IDiff<IList<V>, V> {
+public interface IDiffList<V> extends IList<V>, IDiff<IList<V>> {
 
-  class Range {
-    public final long start, end;
+  interface Durable<V> extends IDiffList<V>, IDurableCollection {
+  }
+
+  /**
+   * A descriptor for the number of elements removed from the front and back of the underlying list.
+   */
+  class Slice {
+    public final long fromFront, fromBack;
+
+    public Slice(long fromFront, long fromBack) {
+      this.fromFront = fromFront;
+      this.fromBack = fromBack;
+    }
 
-    public Range(long start, long end) {
-      this.start = start;
-      this.end = end;
+    public long size(IList<?> underlying) {
+      return Math.max(0, underlying.size() - (fromBack + fromFront));
     }
 
-    public long size() {
-      return end - start;
+    public <V> V nth(IList<V> underlying, long idx) {
+      return underlying.nth(idx + fromFront);
+    }
+
+    public <V> Iterator<V> iterator(IList<V> underlying, long startIdx) {
+      return Iterators.range(fromFront + startIdx, size(underlying), underlying::nth);
     }
   }
 
   IList<V> underlying();
 
-  Range underlyingSlice();
+  Slice slice();
 
   IList<V> prefix();
 
   IList<V> suffix();
 
+  IDiffList<V> rebase(IList<V> newUnderlying);
+
   @Override
   default IList<V> concat(IList<V> l) {
     IList<V> result = Lists.concat(
         prefix(),
-        underlying().slice(underlyingSlice().start, underlyingSlice().end),
+        underlying().slice(slice().fromFront, slice().fromBack),
         suffix(),
         l
     );
@@ -41,7 +57,7 @@ default IList<V> concat(IList<V> l) {
 
   @Override
   default long size() {
-    return prefix().size() + underlyingSlice().size() + suffix().size();
+    return (prefix().size() + suffix().size() + slice().size(underlying()));
   }
 
   @Override
@@ -51,20 +67,20 @@ default V nth(long idx) {
     }
     idx -= prefix().size();
 
-    if (idx < underlyingSlice().size()) {
-      return underlying().nth(underlyingSlice().start + idx);
+    long underlyingSize = slice().size(underlying());
+    if (idx < underlyingSize) {
+      return slice().nth(underlying(), idx);
     }
-    idx -= underlyingSlice().size();
+    idx -= underlyingSize;
 
     return suffix().nth(idx);
   }
 
   @Override
   default Iterator<V> iterator() {
-    Range r = underlyingSlice();
     return Iterators.concat(
         prefix().iterator(),
-        r.size() == underlying().size() ? underlying().iterator() : Iterators.range(r.start, r.end, underlying()::nth),
+        slice().iterator(underlying(), 0),
         suffix().iterator()
     );
   }

src/io/lacuna/bifurcan/IDiffMap.java

@@ -13,7 +13,10 @@
 import java.util.function.BiPredicate;
 import java.util.function.ToLongFunction;
 
-public interface IDiffMap<K, V> extends IMap<K, V>, IDiff<IMap<K, V>, IEntry<K, V>> {
+public interface IDiffMap<K, V> extends IMap<K, V>, IDiff<IMap<K, V>> {
+
+  interface Durable<K, V> extends IDiffMap<K, V>, IDurableCollection {
+  }
 
   /**
    * The baseline data structure.
@@ -30,6 +33,8 @@ public interface IDiffMap<K, V> extends IMap<K, V>, IDiff<IMap<K, V>, IEntry<K,
    */
   ISortedSet<Long> removedIndices();
 
+  IDiffMap<K, V> rebase(IMap<K, V> newUnderlying);
+
   @Override
   default ToLongFunction<K> keyHash() {
     return underlying().keyHash();
@@ -84,7 +89,7 @@ default Iterator<IEntry<K, V>> iterator() {
   }
 
   @Override
-  default Durable<K, V> save(IDurableEncoding encoding, Path directory) {
+  default IMap.Durable<K, V> save(IDurableEncoding encoding, Path directory) {
     if (removedIndices().size() == 0 && added().size() == 0) {
       return underlying().save(encoding, directory);
     } else {

src/io/lacuna/bifurcan/IDiffSet.java

@@ -4,10 +4,15 @@
 import java.util.function.BiPredicate;
 import java.util.function.ToLongFunction;
 
-public interface IDiffSet<V> extends ISet<V>, IDiff<IMap<V, Void>, IEntry<V, Void>> {
+public interface IDiffSet<V> extends ISet<V>, IDiff<IMap<V, Void>> {
+
+  interface Durable<V> extends IDiffSet<V>, IDurableCollection {
+  }
 
   IDiffMap<V, Void> diffMap();
 
+  IDiffSet<V> rebase(IMap<V, Void> newUnderlying);
+
   default IMap<V, Void> underlying() {
     return diffMap().underlying();
   }

src/io/lacuna/bifurcan/IDiffSortedMap.java

@@ -2,15 +2,28 @@
 
 import io.lacuna.bifurcan.utils.Iterators;
 
+import java.util.Comparator;
 import java.util.Iterator;
 import java.util.OptionalLong;
 
-public interface IDiffSortedMap<K, V> extends ISortedMap<K, V> {
+public interface IDiffSortedMap<K, V> extends IDiff<ISortedMap<K, V>>, ISortedMap<K, V> {
+
+  interface Durable<K, V> extends IDiffSortedMap<K, V>, IDurableCollection {
+  }
+
+  ISortedMap<K, V> underlying();
 
   ISortedMap<K, ISortedMap<K, V>> segments();
 
   ISortedSet<Long> segmentOffsets();
 
+  IDiffSortedMap<K, V> rebase(ISortedMap<K, V> newUnderlying);
+
+  @Override
+  default Comparator<K> comparator() {
+    return underlying().comparator();
+  }
+
   @Override
   default long size() {
     return segments().size() == 0
@@ -19,18 +32,18 @@ default long size() {
   }
 
   @Override
-  default OptionalLong floorIndex(K key) {
+  default OptionalLong inclusiveFloorIndex(K key) {
     ISortedMap<K, ISortedMap<K, V>> segments = segments();
     ISortedSet<Long> segmentOffsets = segmentOffsets();
 
-    OptionalLong oSegmentIdx = segments.floorIndex(key);
+    OptionalLong oSegmentIdx = segments.inclusiveFloorIndex(key);
     if (!oSegmentIdx.isPresent()) {
       return OptionalLong.empty();
     }
     long segmentIdx = oSegmentIdx.getAsLong();
 
     ISortedMap<K, V> segment = segments.nth(segmentIdx).value();
-    OptionalLong oIdx = segment.floorIndex(key);
+    OptionalLong oIdx = segment.inclusiveFloorIndex(key);
 
     if (oIdx.isPresent()) {
       return OptionalLong.of(segmentOffsets.nth(segmentIdx) + oIdx.getAsLong());

src/io/lacuna/bifurcan/IDiffSortedSet.java

@@ -6,10 +6,20 @@
 import java.util.Iterator;
 import java.util.OptionalLong;
 
-public interface IDiffSortedSet<V> extends ISortedSet<V> {
+public interface IDiffSortedSet<V> extends IDiff<ISortedSet<V>>, ISortedSet<V> {
+
+  interface Durable<V> extends IDiffSortedSet<V>, IDurableCollection {
+  }
 
   IDiffSortedMap<V, Void> diffMap();
 
+  IDiffSortedSet<V> rebase(ISortedSet<V> newUnderlying);
+
+  @Override
+  default ISortedSet<V> underlying() {
+    return diffMap().underlying().keys();
+  }
+
   @Override
   default Comparator<V> comparator() {
     return diffMap().comparator();
@@ -21,8 +31,8 @@ default long size() {
   }
 
   @Override
-  default OptionalLong floorIndex(V value) {
-    return diffMap().floorIndex(value);
+  default OptionalLong inclusiveFloorIndex(V value) {
+    return diffMap().inclusiveFloorIndex(value);
   }
 
   @Override