-
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
9 changed files
with
96 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| <!DOCTYPE html PUBLIC "" | ||
| ""> | ||
| <html><head><meta charset="UTF-8" /><title>Some Reduction Timings</title><script async="true" src="https://www.googletagmanager.com/gtag/js?id=G-CLH3CS7E1R"></script><script>window.dataLayer = window.dataLayer || []; | ||
| function gtag(){dataLayer.push(arguments);} | ||
| gtag('js', new Date()); | ||
|
|
||
| gtag('config', 'G-CLH3CS7E1R');</script><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="highlight/solarized-light.css" /><script type="text/javascript" src="highlight/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a> with <a href="https://github.com/xsc/codox-theme-rdash">RDash UI</a> theme</h2><h1><a href="index.html"><span class="project-title"><span class="project-name">tmdjs</span> <span class="project-version">1.014</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Topics</span></h3><ul><li class="depth-1 current"><a href="Reductions.html"><div class="inner"><span>Some Reduction Timings</span></div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>tech</span></div></div></li><li class="depth-2"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>v3</span></div></div></li><li class="depth-3"><a href="tech.v3.dataset.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>dataset</span></div></a></li><li class="depth-4"><a href="tech.v3.dataset.node.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>node</span></div></a></li><li class="depth-3"><a href="tech.v3.datatype.html"><div class="inner"><span class="tree" style="top: -52px;"><span class="top" style="height: 61px;"></span><span class="bottom"></span></span><span>datatype</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.argops.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>argops</span></div></a></li><li class="depth-4"><a href="tech.v3.datatype.functional.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>functional</span></div></a></li><li class="depth-3"><div class="no-link"><div class="inner"><span class="tree" style="top: -83px;"><span class="top" style="height: 92px;"></span><span class="bottom"></span></span><span>libs</span></div></div></li><li class="depth-4"><a href="tech.v3.libs.cljs-ajax.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>cljs-ajax</span></div></a></li></ul></div><div class="document" id="content"><div class="doc"><div class="markdown"><h1>Some Reduction Timings</h1> | ||
| <p>The datatype library has some helpers that work with datasets that can make certain types of | ||
| reductions much faster.</p> | ||
| <h2>Filter On One Column, Reduce Another</h2> | ||
| <p>This is a very common operation so let's take a closer look. The generic dataset | ||
| pathway would be:</p> | ||
| <pre><code class="language-clojure">cljs.user> (require '[tech.v3.dataset :as ds]) | ||
| nil | ||
| cljs.user> (def test-ds (ds/->dataset {:a (range 20000) | ||
| :b (repeatedly 20000 rand)})) | ||
| #'cljs.user/test-ds | ||
| cljs.user> ;;filter on a, sum b. | ||
| cljs.user> (reduce + 0.0 (-> (ds/filter-column test-ds :a #(> % 10000)) | ||
| (ds/column :b))) | ||
| 5000.898384571656 | ||
| cljs.user> (time (dotimes [idx 100] (reduce + 0.0 (-> (ds/filter-column test-ds :a #(> % 10000)) | ||
| (ds/column :b))))) | ||
| "Elapsed time: 282.714231 msecs" | ||
| "Elapsed time: 282.714231 msecs" | ||
| "Elapsed time: 282.714231 msecs" | ||
| </code></pre> | ||
| <p>Think transducers are fast? What about a generic transducer pathway?</p> | ||
| <pre><code class="language-clojure">cljs.user> (let [a (test-ds :a) | ||
| b (test-ds :b)] | ||
| (transduce (comp (filter #(> (nth a %) 10000)) | ||
| (map #(nth b %))) | ||
| (completing +) | ||
| (range (ds/row-count test-ds)))) | ||
| 5000.898384571656 | ||
| cljs.user> (time (dotimes [idx 100] | ||
| (let [a (test-ds :a) | ||
| b (test-ds :b)] | ||
| (transduce (comp (filter #(> (nth a %) 10000)) | ||
| (map #(nth b %))) | ||
| (completing +) | ||
| (range (ds/row-count test-ds)))))) | ||
| "Elapsed time: 436.235972 msecs" | ||
| "Elapsed time: 436.235972 msecs" | ||
| "Elapsed time: 436.235972 msecs" | ||
| nil | ||
| </code></pre> | ||
| <p>Transducers are fast - after looking at this pathway we cound it | ||
| pays a lot for each nth call. The datatype library has a way | ||
| to get the fastest access available for a given container. Columns overload | ||
| this pathway such that if there are no missing they use the fastest | ||
| access for their buffer, else they have to wrap a missing check. Regardless, | ||
| this gets us a solid improvement:</p> | ||
| <pre><code class="language-clojure">cljs.user> (require '[tech.v3.datatype :as dtype]) | ||
| nil | ||
| cljs.user> (time (dotimes [idx 100] | ||
| (let [a (dtype/->fast-nth (test-ds :a)) | ||
| b (dtype/->fast-nth (test-ds :b))] | ||
| (transduce (comp (filter #(> (a %) 10000)) | ||
| (map #(b %))) | ||
| (completing +) | ||
| (range (ds/row-count test-ds)))))) | ||
| "Elapsed time: 77.823553 msecs" | ||
| "Elapsed time: 77.823553 msecs" | ||
| "Elapsed time: 77.823553 msecs" | ||
| nil | ||
| </code></pre> | ||
| <p>OK - there is another more dangerous approach. dtype has another query, | ||
| as-agetable, that either returns something for which <code>aget</code> works or | ||
| nil. If you know your dataset's columns have no missing data and their | ||
| backing store data itself is agetable - then you can get an agetable. This | ||
| doesn't have a fallback so you risk null ptr issues - but it is the fastest | ||
| possible pathway.</p> | ||
| <pre><code class="language-clojure">cljs.user> (time (dotimes [idx 100] | ||
| (let [a (dtype/as-agetable (test-ds :a)) | ||
| b (dtype/as-agetable (test-ds :b))] | ||
| (transduce (comp (filter #(> (aget a %) 10000)) | ||
| (map #(aget b %))) | ||
| (completing +) | ||
| (range (ds/row-count test-ds)))))) | ||
| "Elapsed time: 57.404783 msecs" | ||
| "Elapsed time: 57.404783 msecs" | ||
| "Elapsed time: 57.404783 msecs" | ||
| nil | ||
| </code></pre> | ||
| <p>In this simple example we find that a transducing pathway is indeed a quite bit faster but only | ||
| when it is coupled with an efficient per-element access pattern.</p> | ||
| </div></div></div></body></html> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.