-
-
Notifications
You must be signed in to change notification settings - Fork 22
/
overview.html
227 lines (204 loc) · 20.5 KB
/
overview.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
<!DOCTYPE html PUBLIC ""
"">
<html><head><meta charset="UTF-8" /><title>Overview</title><link rel="stylesheet" type="text/css" href="css/default.css" /><link rel="stylesheet" type="text/css" href="highlight/solarized-light.css" /><script type="text/javascript" src="highlight/highlight.min.js"></script><script type="text/javascript" src="js/jquery.min.js"></script><script type="text/javascript" src="js/page_effects.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body><div id="header"><h2>Generated by <a href="https://github.com/weavejester/codox">Codox</a> with <a href="https://github.com/xsc/codox-theme-rdash">RDash UI</a> theme</h2><h1><a href="index.html"><span class="project-title"><span class="project-name">dtype-next</span> <span class="project-version">8.042</span></span></a></h1></div><div class="sidebar primary"><h3 class="no-link"><span class="inner">Project</span></h3><ul class="index-link"><li class="depth-1 "><a href="index.html"><div class="inner">Index</div></a></li></ul><h3 class="no-link"><span class="inner">Topics</span></h3><ul><li class="depth-1 "><a href="buffered-image.html"><div class="inner"><span>Buffered Image Support</span></div></a></li><li class="depth-1 "><a href="cheatsheet.html"><div class="inner"><span>Cheatsheet</span></div></a></li><li class="depth-1 "><a href="datatype-to-dtype-next.html"><div class="inner"><span>Why dtype-next?</span></div></a></li><li class="depth-1 "><a href="dimensions-bytecode-gen.html"><div class="inner"><span>Dimensions and Bytecode Generation</span></div></a></li><li class="depth-1 current"><a href="overview.html"><div class="inner"><span>Overview</span></div></a></li></ul><h3 class="no-link"><span class="inner">Namespaces</span></h3><ul><li class="depth-1"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>tech</span></div></div></li><li class="depth-2"><div class="no-link"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>v3</span></div></div></li><li class="depth-3"><a href="tech.v3.datatype.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>datatype</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.argops.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>argops</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.bitmap.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>bitmap</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.convolve.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>convolve</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.datetime.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>datetime</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.errors.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>errors</span></div></a></li><li class="depth-4"><a href="tech.v3.datatype.ffi.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>ffi</span></div></a></li><li class="depth-5 branch"><a href="tech.v3.datatype.ffi.clang.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>clang</span></div></a></li><li class="depth-5"><a href="tech.v3.datatype.ffi.graalvm.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>graalvm</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.functional.html"><div class="inner"><span class="tree" style="top: -83px;"><span class="top" style="height: 92px;"></span><span class="bottom"></span></span><span>functional</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.gradient.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>gradient</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.jna.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>jna</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.jvm-map.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>jvm-map</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.list.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>list</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.locker.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>locker</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.mmap.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>mmap</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.mmap-writer.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>mmap-writer</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.native-buffer.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>native-buffer</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.nippy.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>nippy</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.packing.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>packing</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.reductions.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>reductions</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.rolling.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>rolling</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.sampling.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>sampling</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.datatype.struct.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>struct</span></div></a></li><li class="depth-4"><a href="tech.v3.datatype.wavelet.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>wavelet</span></div></a></li><li class="depth-3"><div class="no-link"><div class="inner"><span class="tree" style="top: -765px;"><span class="top" style="height: 774px;"></span><span class="bottom"></span></span><span>libs</span></div></div></li><li class="depth-4 branch"><a href="tech.v3.libs.buffered-image.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>buffered-image</span></div></a></li><li class="depth-4"><a href="tech.v3.libs.neanderthal.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>neanderthal</span></div></a></li><li class="depth-3"><div class="no-link"><div class="inner"><span class="tree" style="top: -83px;"><span class="top" style="height: 92px;"></span><span class="bottom"></span></span><span>parallel</span></div></div></li><li class="depth-4"><a href="tech.v3.parallel.for.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>for</span></div></a></li><li class="depth-3"><a href="tech.v3.tensor.html"><div class="inner"><span class="tree" style="top: -52px;"><span class="top" style="height: 61px;"></span><span class="bottom"></span></span><span>tensor</span></div></a></li><li class="depth-4 branch"><a href="tech.v3.tensor.color-gradients.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>color-gradients</span></div></a></li><li class="depth-4"><a href="tech.v3.tensor.dimensions.html"><div class="inner"><span class="tree"><span class="top"></span><span class="bottom"></span></span><span>dimensions</span></div></a></li></ul></div><div class="document" id="content"><div class="doc"><div class="markdown"><h1><a href="#overview" name="overview"></a>Overview</h1>
<h2><a href="#setup" name="setup"></a>Setup</h2>
<pre><code class="clojure">(require '[tech.v3.datatype :as dtype])
</code></pre>
<h2><a href="#containers" name="containers"></a>Containers</h2>
<p>There are two different types of containers in tech.v3.datatype; jvm-heap containers and native-heap containers. Object datatypes are only supported in jvm-heap containers and native-heap containers have mmap support and offer zero-copy pathways to toolkits like Python’s numpy and Clojure’s neanderthal.</p>
<pre><code class="clojure">user> (require '[tech.v3.datatype :as dtype])
nil
user> (dtype/make-container :jvm-heap :float32 5)
#array-buffer<float32>[5]
[0.000, 0.000, 0.000, 0.000, 0.000, ]
user> (dtype/make-container :jvm-heap :float32 (range 5))
#array-buffer<float32>[5]
[0.000, 1.000, 2.000, 3.000, 4.000, ]
user> (dtype/make-container :native-heap :float32 (range 5))
#native-buffer@0x00007F665489BAE0<float32>[5]
[0.000, 1.000, 2.000, 3.000, 4.000, ]
user> ;;These containers support Indexed and read-only access via IFn
user> (def data (dtype/make-container :jvm-heap :float32 (range 5)))
#'user/data
user> (data 0)
0.0
user> (nth data 2)
2.0
</code></pre>
<h2><a href="#copy" name="copy"></a>Copy</h2>
<p>Heavily optimized copying for bulk transfers. Use this also for completing lazy operations.</p>
<pre><code class="clojure">;; You will not see major efficiency gains until both sides the copy
;; operation is backed by
user> (dtype/copy! (range 10) (float-array 10))
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
user> (type *1)
[F
;; Fastest (by factor of 100 discounting array creation)
user> (dtype/copy! (float-array (range 10)) (float-array 10))
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
;; Also extremely fast! native->jvm-array transfer is an optimized
;; operation supported by graal native pathways via the sun.misc.Unsafe pathway
user> (dtype/copy! (dtype/make-container :native-buffer :float32 (range 10)) (float-array 10))
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
;; Copy also supports marshalling data between containers using C casting rules - floating values
;; are floored if ceiled depending on which size of zero they are on.
user> (dtype/copy! (dtype/make-container :native-heap :float32 [0 1 254])
(dtype/make-container :jvm-heap :uint8 3))
#array-buffer<uint8>[3]
[0, 1, 254, ]
user>
</code></pre>
<h2><a href="#buffers-readers-writers" name="buffers-readers-writers"></a>Buffers, Readers, Writers</h2>
<p>Efficent primitive-datatype-aware access is provided via the <a href="https://github.com/cnuernber/dtype-next/blob/d04c309bd565292c1c3d9880b4bbb80b6ff9478e/java/tech/v3/datatype/Buffer.java">buffer interface</a>. A reader is a buffer that can be read from and a writer is a buffer that can be written to – note the buffer interface has ‘canRead’ and ‘canWrite’ methods.</p>
<p>Buffers have many transformations such as adding a scalar or element-wise adding the elements of another buffer. A few of these operations are exposed via the <a href="https://cnuernber.github.io/dtype-next/tech.v3.datatype.functional.html">tech.v3.datatype.functional</a> interface and those are discussed in depth below.</p>
<p>Most things are convertible to a reader of a specific type. So things like persistent vectors, java arrays, numpy arrays, etc. are convertible to readers. Some things are convertible to concrete buffers such as jvm heap containers (arrays) and native heap containers. Any reader can be <code>reshape</code>d into a tensor.</p>
<h2><a href="#math" name="math"></a>Math</h2>
<p>A small sample of what is available. Basic elementwise operations along with some statistics and a fixed rolling windowing facility.</p>
<h3><a href="#container-coersion-rules" name="container-coersion-rules"></a>Container Coersion Rules</h3>
<p>Rules are applied in order.</p>
<ol>
<li>If the arguments contain an iterable, an iterable is returned.</li>
<li>If the arguments contain a reader, a reader is returned. If the input reader had a shape with more than one dimension, a tensor is returned.</li>
<li>Else a scalar is returned.</li>
</ol>
<p>1,2,and many argument versions of these rules part of the public tech.v3.datatype api so they are reused for other generalized functions. These functions are:</p>
<ul>
<li><code>vectorized-dispatch-1</code> - single argument dispatch pathway.</li>
<li><code>vectorized-dispatch-2</code> - double argument dispatch pathway.</li>
<li><code>emap</code> - generalized n-dimension implementation for n arguments.</li>
</ul>
<h3><a href="#other-details" name="other-details"></a>Other Details</h3>
<ul>
<li>All arguments are casted to the widest datatype present.</li>
<li>Readers implement List and RandomAccess so they look like persistent vectors in the repl.</li>
</ul>
<pre><code class="clojure">user> (require '[tech.v2.datatype.functional :as dtype-fn])
nil
user> (def test-data (dtype-fn/+ (range 10 0 -1) 5))
#'user/test-data
user> test-data
(15 14 13 12 11 10 9 8 7 6)
user> ;;Working in index space is often ideal when dealing with datasets described this way.
user> ;;To help with that we have a few tools.
user> (require '[tech.v3.datatype.argops :as argops])
nil
user> (argops/argsort > test-data )
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
user> (argops/argsort < test-data )
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
user> ;;once you have a set of indexes, you can reindex your original reader
user> (dtype/indexed-buffer indexes test-data)
[6 7 8 9 10 11 12 13 14 15]
user> ;;dfn has several functions for descriptive statistics
user> (dfn/variance test-data)
9.166666666666666
user> (dfn/descriptive-statistics test-data)
{:min 6.0,
:mean 10.5,
:standard-deviation 3.0276503540974917,
:max 15.0,
:n-values 10}
</code></pre>
<h2><a href="#dates-times" name="dates-times"></a>Dates & Times</h2>
<p>All of the constructors for the types are in <code>tech.v3.datatype.datetime</code>. All your favorite <code>java.time</code> types are there.</p>
<pre><code class="clojure">user> (require '[tech.v3.datatype.datetime :as dtype-dt])
user> (dtype-dt/zoned-date-time)
#object[java.time.ZonedDateTime 0x190aea6a "2020-09-24T16:00:27.211-06:00[America/Denver]"]
user> (dtype-dt/local-date)
#object[java.time.LocalDate 0x39080aab "2020-09-24")
user> (dtype-dt/plus-temporal-amount (dtype-dt/local-date) 2 :days)
#object[java.time.LocalDate 0x680586f8 "2020-09-26"]
user> (dtype-dt/plus-temporal-amount (dtype-dt/local-date) (range 10) :days)
[#object[java.time.LocalDate 0x56bd7a17 "2020-09-24"] #object[java.time.LocalDate 0x47edcb55 "2020-09-25"] #object[java.time.LocalDate 0x3ab1a911 "2020-09-26"] #object[java.time.LocalDate 0x5a75b76 "2020-09-27"] #object[java.time.LocalDate 0x38f7e417 "2020-09-28"] #object[java.time.LocalDate 0x1169273f "2020-09-29"] #object[java.time.LocalDate 0x26a7ed22 "2020-09-30"] #object[java.time.LocalDate 0x19646e6b "2020-10-01"] #object[java.time.LocalDate 0x1eb4e60c "2020-10-02"] #object[java.time.LocalDate 0x463c6a3a "2020-10-03"]]
user> (def offset-local-dates (dtype-dt/plus-temporal-amount (dtype-dt/local-date) (range 10) :days))
#'user/offset-local-dates
user> (dtype-dt/long-temporal-field :days offset-local-dates)
[24 25 26 27 28 29 30 1 2 3]
user> (dtype-dt/long-temporal-field :day-of-week offset-local-dates)
[4 5 6 7 1 2 3 4 5 6]
user> (dtype-dt/long-temporal-field :day-of-year offset-local-dates)
[268 269 270 271 272 273 274 275 276 277]
;;there is also a generalized conversion mechanism to milliseconds-since-epoch and back. This conversion
;;can be timezone aware when one is passed in and it uses UTC when no timezone is passed.
user> (dtype-dt/datetime->milliseconds offset-local-dates)
[1600905600000 1600992000000 1601078400000 1601164800000 1601251200000 1601337600000 1601424000000 1601510400000 1601596800000 1601683200000]
user> (dtype-dt/system-zone-id)
#object[java.time.ZoneRegion 0x10bdee8e "America/Denver"]
user> (dtype-dt/datetime->milliseconds (dtype-dt/system-zone-id) offset-local-dates)
[1600927200000 1601013600000 1601100000000 1601186400000 1601272800000 1601359200000 1601445600000 1601532000000 1601618400000 1601704800000]
</code></pre>
<h2><a href="#nd-buffers" name="nd-buffers"></a>ND Buffers</h2>
<p>Generic N-dimensional support is provied both based on raw containers and based on functions from N indexes to a value. These objects implement the <a href="https://github.com/cnuernber/dtype-next/blob/d04c309bd565292c1c3d9880b4bbb80b6ff9478e/java/tech/v3/datatype/NDBuffer.java">NDBuffer interface</a> and the support is provided primarily via the ‘tensor’ namespace. The basic ways of dealing with tensor are:</p>
<ul>
<li><code>->tensor</code> - Make a new tensor by copying data into a container. This can take sequences, persistent vectors, etc. and will copy the data into a contiguous buffer and return a new implementation of NDBuffer based on that data.</li>
<li><code>ensure-tensor</code>- Attempt in-place transformation of data falling back to <code>->tensor</code> when not available.</li>
<li><code>compute-tensor</code> - Create a new tensor via a function that takes <code>rank</code> long indexes and returns a value.</li>
<li><code>reshape</code> - reshape a buffer or reader into a different dimension and rank via linear reinterpretation of the data. Use this to convert a normal bit of data such as a java array or a persistent vector into a tensor without copying the data.</li>
<li><code>transpose</code> - Permute data via reordering dimensions.</li>
<li><code>select</code> - Select a subrect of data. Select can also take indexes in order to do ad-hoc reordering of the data.</li>
<li><code>broadcast</code> - Create a larger tensor via duplication along one or more dimensions.</li>
</ul>
<p>These tech.v3.datatype functions are also useful for tensors:</p>
<ul>
<li><code>elemwise-cast</code> - change datatype of tensor.</li>
<li><code>clone</code> - Create a new tensor of same shape and data. This can be used to realize lazy operations.</li>
<li><code>->buffer,->reader</code> - Get the data in 1D row-major vector.</li>
</ul>
<pre><code class="clojure">user> (require '[tech.v3.tensor :as dtt])
nil
user> (dtt/->tensor (partition 3 (range 9)))
#tech.v3.tensor<object>[3 3]
[[0 1 2]
[3 4 5]
[6 7 8]]
</code></pre>
<p>Tensor implement various java/clojure interfaces they interact naturally with the language.</p>
<pre><code class="clojure">user> (def tens (dtt/->tensor (partition 3 (range 9))))
#'user/tens
user> (tens 0)
#tech.v3.tensor<object>[3]
[0 1 2]
user> (tens 1)
#tech.v3.tensor<object>[3]
[3 4 5]
user> (nth tens 2)
#tech.v3.tensor<object>[3]
[6 7 8]
[#tech.v3.tensor<object>[3]
[0 3 6] #tech.v3.tensor<object>[3]
[1 4 7] #tech.v3.tensor<object>[3]
[2 5 8]]
user> (dtt/transpose tens [1 0])
#tech.v3.tensor<object>[3 3]
[[0 3 6]
[1 4 7]
[2 5 8]]
</code></pre>
<p>You can get/set subrects at a given time using mget/mset! pathways. If you provide fewer indexes than the required indexes they will either return or assign values to those sections.</p>
<pre><code class="clojure">user> (def tens (dtt/->tensor (partition 3 (range 9))))
#'user/tens
user> (dtt/mget tens 0)
#tech.v3.tensor<object>[3]
[0 1 2]
user> (dtt/mset! tens 0 25)
#tech.v3.tensor<object>[3 3]
[[25 25 25]
[ 3 4 5]
[ 6 7 8]]
user> (dtt/mset! tens 0 [25 26 27])
#tech.v3.tensor<object>[3 3]
[[25 26 27]
[ 3 4 5]
[ 6 7 8]]
user> (dtt/transpose tens [1 0])
#tech.v3.tensor<object>[3 3]
[[25 3 6]
[26 4 7]
[27 5 8]]
user> (dtt/mset! (dtt/transpose tens [1 0]) 0 [25 26 27])
#tech.v3.tensor<object>[3 3]
[[25 26 27]
[26 4 7]
[27 5 8]]
</code></pre>
<p>If you got this far be sure to checkout the <a href="https://cnuernber.github.io/dtype-next/cheatsheet.html">cheatseet</a>.</p></div></div></div></body></html>