-
-
Notifications
You must be signed in to change notification settings - Fork 21
/
Copy pathtech.v3.datatype.reductions.html
31 lines (30 loc) · 20.6 KB
/
tech.v3.datatype.reductions.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
<!DOCTYPE html PUBLIC ""
"">
<html><head><meta charset="UTF-8" /><title>tech.v3.datatype.reductions documentation</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="highlight/solarized-light.css" /><script type="text/javascript" src="highlight/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a> with <a href="https://github.com/xsc/codox-theme-rdash">RDash UI</a> theme</h2><h1><a href="index.html"><span class="project-title"><span class="project-name">dtype-next</span> <span class="project-version">8.042</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Topics</span></h3><ul><li class="depth-1 "><a href="buffered-image.html"><div class="inner"><span>Buffered Image Support</span></div></a></li><li class="depth-1 "><a href="cheatsheet.html"><div class="inner"><span>Cheatsheet</span></div></a></li><li class="depth-1 "><a href="datatype-to-dtype-next.html"><div class="inner"><span>Why dtype-next?</span></div></a></li><li class="depth-1 "><a href="dimensions-bytecode-gen.html"><div class="inner"><span>Dimensions and Bytecode Generation</span></div></a></li><li class="depth-1 "><a href="overview.html"><div class="inner"><span>Overview</span></div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>tech</span></div></div></li><li class="depth-2"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>v3</span></div></div></li><li class="depth-3"><a href="tech.v3.datatype.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>datatype</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.argops.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>argops</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.bitmap.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>bitmap</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.convolve.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>convolve</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.datetime.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>datetime</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.errors.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>errors</span></div></a></li><li class="depth-4"><a href="tech.v3.datatype.ffi.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>ffi</span></div></a></li><li class="depth-5 branch"><a href="tech.v3.datatype.ffi.clang.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>clang</span></div></a></li><li class="depth-5"><a href="tech.v3.datatype.ffi.graalvm.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>graalvm</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.functional.html"><div class="inner"><span class="tree" style="top: -83px;"><span class="top" style="height: 92px;"></span><span class="bottom"></span></span><span>functional</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.gradient.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>gradient</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.jna.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>jna</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.jvm-map.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>jvm-map</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.list.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>list</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.locker.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>locker</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.mmap.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>mmap</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.mmap-writer.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>mmap-writer</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.native-buffer.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>native-buffer</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.nippy.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>nippy</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.packing.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>packing</span></div></a></li><li class="depth-4 branch current"><a href="tech.v3.datatype.reductions.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>reductions</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.rolling.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>rolling</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.sampling.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>sampling</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.struct.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>struct</span></div></a></li><li class="depth-4"><a href="tech.v3.datatype.wavelet.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>wavelet</span></div></a></li><li class="depth-3"><div class="no-link"><div class="inner"><span class="tree" style="top: -765px;"><span class="top" style="height: 774px;"></span><span class="bottom"></span></span><span>libs</span></div></div></li><li class="depth-4 branch"><a href="tech.v3.libs.buffered-image.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>buffered-image</span></div></a></li><li class="depth-4"><a href="tech.v3.libs.neanderthal.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>neanderthal</span></div></a></li><li class="depth-3"><div class="no-link"><div class="inner"><span class="tree" style="top: -83px;"><span class="top" style="height: 92px;"></span><span class="bottom"></span></span><span>parallel</span></div></div></li><li class="depth-4"><a href="tech.v3.parallel.for.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>for</span></div></a></li><li class="depth-3"><a href="tech.v3.tensor.html"><div class="inner"><span class="tree" style="top: -52px;"><span class="top" style="height: 61px;"></span><span class="bottom"></span></span><span>tensor</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.tensor.color-gradients.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>color-gradients</span></div></a></li><li class="depth-4"><a href="tech.v3.tensor.dimensions.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>dimensions</span></div></a></li></ul></div><div class="sidebar secondary"><h3><a href="#top"><span class="inner">Public Vars</span></a></h3><ul><li class="depth-1"><a href="tech.v3.datatype.reductions.html#var-commutative-binary-double"><div class="inner"><span>commutative-binary-double</span></div></a></li><li class="depth-1"><a href="tech.v3.datatype.reductions.html#var-commutative-binary-long"><div class="inner"><span>commutative-binary-long</span></div></a></li><li class="depth-1"><a href="tech.v3.datatype.reductions.html#var-commutative-binary-object"><div class="inner"><span>commutative-binary-object</span></div></a></li><li class="depth-1"><a href="tech.v3.datatype.reductions.html#var-commutative-binary-reduce"><div class="inner"><span>commutative-binary-reduce</span></div></a></li><li class="depth-1"><a href="tech.v3.datatype.reductions.html#var-double-reductions"><div class="inner"><span>double-reductions</span></div></a></li><li class="depth-1"><a href="tech.v3.datatype.reductions.html#var-double-summation"><div class="inner"><span>double-summation</span></div></a></li><li class="depth-1"><a href="tech.v3.datatype.reductions.html#var-nan-strategies"><div class="inner"><span>nan-strategies</span></div></a></li><li class="depth-1"><a href="tech.v3.datatype.reductions.html#var-nan-strategy-.3Edouble-predicate"><div class="inner"><span>nan-strategy->double-predicate</span></div></a></li><li class="depth-1"><a href="tech.v3.datatype.reductions.html#var-ordered-group-by-reduce"><div class="inner"><span>ordered-group-by-reduce</span></div></a></li><li class="depth-1"><a href="tech.v3.datatype.reductions.html#var-reader-.3Edouble-spliterator"><div class="inner"><span>reader->double-spliterator</span></div></a></li><li class="depth-1"><a href="tech.v3.datatype.reductions.html#var-reducer-value-.3Econsumer-fn"><div class="inner"><span>reducer-value->consumer-fn</span></div></a></li><li class="depth-1"><a href="tech.v3.datatype.reductions.html#var-staged-double-consumer-reduction"><div class="inner"><span>staged-double-consumer-reduction</span></div></a></li><li class="depth-1"><a href="tech.v3.datatype.reductions.html#var-unary-double-summation"><div class="inner"><span>unary-double-summation</span></div></a></li><li class="depth-1"><a href="tech.v3.datatype.reductions.html#var-unordered-group-by-reduce"><div class="inner"><span>unordered-group-by-reduce</span></div></a></li></ul></div><div class="namespace-docs" id="content"><h1 class="anchor" id="top">tech.v3.datatype.reductions</h1><div class="doc"><div class="markdown"><p>High performance reductions based on tech.v3.datatype concepts as well as java stream concepts.</p></div></div><div class="public anchor" id="var-commutative-binary-double"><h3>commutative-binary-double</h3><div class="usage"><code>(commutative-binary-double op options rdr)</code><code>(commutative-binary-double op rdr)</code></div><div class="doc"><div class="markdown"><p>Perform a commutative reduction using a binary operator to perform the reduction. The operator needs to be both commutative and associative.</p></div></div><div class="src-link"><a href="https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/reductions.clj#L176">view source</a></div></div><div class="public anchor" id="var-commutative-binary-long"><h3>commutative-binary-long</h3><div class="usage"><code>(commutative-binary-long op rdr)</code></div><div class="doc"><div class="markdown"><p>Perform a commutative reduction in int64 space using a binary operator. The operator needs to be both commutative and associative.</p></div></div><div class="src-link"><a href="https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/reductions.clj#L188">view source</a></div></div><div class="public anchor" id="var-commutative-binary-object"><h3>commutative-binary-object</h3><div class="usage"><code>(commutative-binary-object op rdr)</code></div><div class="doc"><div class="markdown"><p>Perform a commutative reductions in object space using a binary operator. The operator needs to be both commutative and associative.</p></div></div><div class="src-link"><a href="https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/reductions.clj#L208">view source</a></div></div><div class="public anchor" id="var-commutative-binary-reduce"><h3>commutative-binary-reduce</h3><div class="usage"><code>(commutative-binary-reduce op rdr)</code></div><div class="doc"><div class="markdown"><p>Perform a commutative binary reduction. The space of the reduction will be determined by the datatype of the reader.</p></div></div><div class="src-link"><a href="https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/reductions.clj#L226">view source</a></div></div><div class="public anchor" id="var-double-reductions"><h3>double-reductions</h3><div class="usage"><code>(double-reductions reducer-map options rdr)</code><code>(double-reductions reducer-map rdr)</code></div><div class="doc"><div class="markdown"><p>Perform a group of reductions on a single double reader.</p></div></div><div class="src-link"><a href="https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/reductions.clj#L115">view source</a></div></div><div class="public anchor" id="var-double-summation"><h3>double-summation</h3><div class="usage"><code>(double-summation options rdr)</code><code>(double-summation rdr)</code></div><div class="doc"><div class="markdown"><p>Double sum of data using <a href="https://en.wikipedia.org/wiki/Kahan_summation_algorithm">Kahan compensated summation</a>.</p></div></div><div class="src-link"><a href="https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/reductions.clj#L152">view source</a></div></div><div class="public anchor" id="var-nan-strategies"><h3>nan-strategies</h3><div class="usage"></div><div class="doc"><div class="markdown"></div></div><div class="src-link"><a href="https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/reductions.clj#L29">view source</a></div></div><div class="public anchor" id="var-nan-strategy-.3Edouble-predicate"><h3>nan-strategy->double-predicate</h3><div class="usage"><code>(nan-strategy->double-predicate nan-strategy)</code><code>(nan-strategy->double-predicate)</code></div><div class="doc"><div class="markdown"><p>Passing in either a keyword nan strategy #{:exception :keep :remove} or a DoublePredicate, return a DoublePredicate that filters double values.</p></div></div><div class="src-link"><a href="https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/reductions.clj#L33">view source</a></div></div><div class="public anchor" id="var-ordered-group-by-reduce"><h3>ordered-group-by-reduce</h3><div class="usage"><code>(ordered-group-by-reduce reducer batch-data rdr)</code><code>(ordered-group-by-reduce reducer rdr)</code></div><div class="doc"><div class="markdown"><p>Perform an ordered group-by operation using reader and placing results into the result-map. Expects that reducer’s batch-data method has already been called and Returns the non-finalized result-map. If result-map is nil then one is created. Each bucket’s results end up ordered by index iteration order. The original parallel pass goes through each index in order and then the reduction goes through the thread groups in order so if your index reduction merger just does (.addAll lhs rhs) then the final result ends up ordered.</p>
<p>Example:</p>
<pre><code class="clojure">user> (import '[tech.v3.datatype IndexReduction])
tech.v3.datatype.IndexReduction
user> (require '[tech.v3.datatype :as dtype])
nil
user> (require '[tech.v3.datatype.reductions :as dt-reduce])
nil
user> (def key-col [:a :b :a :a :c :d])
#'user/key-col
user> (def val-col (mapv name key-col))
#'user/val-col
user> (dt-reduce/ordered-group-by-reduce
(reify IndexReduction
(reduceIndex [this batch ctx idx]
(conj ctx (val-col idx)))
(reduceReductions [this lhs rhs]
(vec (concat lhs rhs))))
key-col)
{:a ("a" "a" "a"), :b ("b"), :c ("c"), :d ("d")}
user>
</code></pre></div></div><div class="src-link"><a href="https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/reductions.clj#L303">view source</a></div></div><div class="public anchor" id="var-reader-.3Edouble-spliterator"><h3>reader->double-spliterator</h3><div class="usage"><code>(reader->double-spliterator rdr nan-strategy)</code><code>(reader->double-spliterator rdr)</code></div><div class="doc"><div class="markdown"><p>Convert a primitiveIO object into a spliterator with an optional nan strategy [:keep :remove :exception] or an implementation of a java.util.functions.DoublePredicate that can filter the double stream (or throw an exception on an invalid value). Implementations of UnaryPredicate also implement DoublePredicate.</p></div></div><div class="src-link"><a href="https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/reductions.clj#L245">view source</a></div></div><div class="public anchor" id="var-reducer-value-.3Econsumer-fn"><h3>reducer-value->consumer-fn</h3><div class="usage"><code>(reducer-value->consumer-fn reducer-value)</code></div><div class="doc"><div class="markdown"><p>Produce a consumer from a generic reducer value.</p></div></div><div class="src-link"><a href="https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/reductions.clj#L67">view source</a></div></div><div class="public anchor" id="var-staged-double-consumer-reduction"><h3>staged-double-consumer-reduction</h3><div class="usage"><code>(staged-double-consumer-reduction staged-consumer-fn {:keys [nan-strategy]} rdr)</code><code>(staged-double-consumer-reduction staged-consumer-fn rdr)</code></div><div class="doc"><div class="markdown"><p>Given a function that produces an implementation of tech.v3.datatype.Consumers$StagedConsumer perform a reduction.</p>
<p>A staged consumer is a consumer can be used in a map-reduce pathway where during the map portion .consume is called and then during produces a ‘result’ on which .combine is called during the reduce pathway.</p></div></div><div class="src-link"><a href="https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/reductions.clj#L85">view source</a></div></div><div class="public anchor" id="var-unary-double-summation"><h3>unary-double-summation</h3><div class="usage"><code>(unary-double-summation op options rdr)</code><code>(unary-double-summation op rdr)</code></div><div class="doc"><div class="markdown"><p>Perform a double summation using a unary operator to transform the input stream into a new double stream.</p></div></div><div class="src-link"><a href="https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/reductions.clj#L164">view source</a></div></div><div class="public anchor" id="var-unordered-group-by-reduce"><h3>unordered-group-by-reduce</h3><div class="usage"><code>(unordered-group-by-reduce reducer batch-data rdr result-map)</code><code>(unordered-group-by-reduce reducer batch-data rdr)</code><code>(unordered-group-by-reduce reducer rdr)</code></div><div class="doc"><div class="markdown"><p>Perform an unordered group-by operation using reader and placing results into the result-map. Expects that reducer’s batch-data method has already been called and returns the non-finalized result-map.</p>
<ul>
<li>batch-data is the <em>result</em> of the reducer’s prepareBatch function.</li>
</ul>
<p>If a map is passed in then it’s compute operator needs to be threadsafe. If result-map is nil then one is created. This implementation takes advantage of the fact that for java8+ we have essentially a lock free concurrent hash map as long as there aren’t collisions so it performs surprisingly well considering the amount of pressure this can put on the concurrency model. Unordered has an advantage with very large datasets in that it does not require a merge step at the end of the parallelized group-by pass.</p>
<p>Returns the result map. It is the caller’s job to call the reducer’s finalize on each map value if necessary.</p></div></div><div class="src-link"><a href="https://github.com/cnuernber/dtype-next/blob/master/src/tech/v3/datatype/reductions.clj#L261">view source</a></div></div></div></body></html>