Buffered Image Support
+Generated by Codox with RDash UI theme
dtype-next 8.034
Buffered Image Support
dtype-next contains support for loading/saving buffered images and creating tensors from buffered images.
Usage
Buffered images implement the protocols required to be part of the datatype system.
diff --git a/docs/cheatsheet.html b/docs/cheatsheet.html index a57e6869..56990663 100644 --- a/docs/cheatsheet.html +++ b/docs/cheatsheet.html @@ -1,6 +1,6 @@ -Generated by Codox with RDash UI theme
dtype-next 8.033
Cheatsheet
+Generated by Codox with RDash UI theme
dtype-next 8.034
Cheatsheet
The old cheatsheet has been moved to overview.
Most of these functions are accessible via the [tech.v3.datatype :as dtype] namespace. When another namespace is required, it is specified separately.
Containers
diff --git a/docs/datatype-to-dtype-next.html b/docs/datatype-to-dtype-next.html index 3a93dba2..4e0e8e4f 100644 --- a/docs/datatype-to-dtype-next.html +++ b/docs/datatype-to-dtype-next.html @@ -1,6 +1,6 @@ -Generated by Codox with RDash UI theme
dtype-next 8.033
Why dtype-next?
+Generated by Codox with RDash UI theme
dtype-next 8.034
Why dtype-next?
tech.datatype
tech.datatype as a numerics stack fulfills our technical needs at TechAscent in regards to scientific computing, data science, and machine learning. It enables a completely unified interface between native heap and JVM heap datastructures with a base level of datatype support and various simple accelerated operations.
In order to fulfill unified support for efficient random access across many datatypes it uses many type specific constructions that enable primitive reading/writing to buffers with disparate storage backends.
diff --git a/docs/dimensions-bytecode-gen.html b/docs/dimensions-bytecode-gen.html index fdaa20af..2ed42c21 100644 --- a/docs/dimensions-bytecode-gen.html +++ b/docs/dimensions-bytecode-gen.html @@ -1,6 +1,6 @@ -Generated by Codox with RDash UI theme
dtype-next 8.033
Dimensions and Bytecode Generation
+Generated by Codox with RDash UI theme
dtype-next 8.034
Dimensions and Bytecode Generation
Translating from N-Dimensional Space to 1 Dimensional Space
We want to be able to take 1 dimensional buffers of data and represent N-dimensional hyper-rectangular spaces. Lots of things are n-dimensional and hyperrectangular; things like images and datasets and volumes and 3d objects so let’s take a moment and talk about the abstractions required efficiently present this interface.
dimensions objects describe the N-dimensional addressing scheme and perform the translation from our n-dimensional ‘global’ space into our 1-dimensional ‘local’ dense address spaces. In other words they describe how to randomly addressed data buffers using an n-dimensional addressing scheme that supports numpy style in-place slicing, transpose, broadcasting, reshape, and APL style rotations. In this way they are designed to add ‘ND’ operations to any randomly addressed system.
Generated by Codox with RDash UI theme
dtype-next 8.033
dtype-next 8.033
A Clojure library designed to aid in the implementation of high performance algorithms and systems.
Topics
Namespaces
tech.v3.datatype
Base namespace for container creation and elementwise access of data
Public variables and functions:
- ->array
- ->array-buffer
- ->buffer
- ->byte-array
- ->char-array
- ->double-array
- ->float-array
- ->int-array
- ->iterable
- ->long-array
- ->native-buffer
- ->reader
- ->short-array
- ->vector
- ->writer
- arg-type
- as-array-buffer
- as-buffer
- as-concrete-buffer
- as-native-buffer
- as-nd-buffer-descriptor
- as-persistent-vector
- as-reader
- as-roaring-bitmap
- as-writer
- cast
- clone
- coalesce!
- coalesce-blocks!
- concat-buffers
- const-reader
- copy!
- copy-raw->item!
- datatype
- ecount
- elemwise-cast
- elemwise-datatype
- emap
- ensure-reader
- ensure-serializeable
- get-datatype
- get-value
- indexed-buffer
- iterable?
- make-container
- make-list
- make-reader
- reader-like?
- reader?
- set-constant!
- set-datatype
- set-value!
- shape
- sub-buffer
- unchecked-cast
- vectorized-dispatch-1
- vectorized-dispatch-2
- writer?
tech.v3.datatype.argops
Efficient functions for operating in index space. Take-off of the argsort, argmin, etc. type functions from Matlab. These functions generally only work on readers and all return some version of an index or list of indexes.
Public variables and functions:
- ->binary-operator
- ->binary-predicate
- ->comparator
- ->double-comparator
- ->long-comparator
- ->unary-operator
- ->unary-predicate
- arg-min-n
- argfilter
- arggroup
- arggroup-by
- arglast-every
- argmax
- argmin
- argpartition
- argpartition-by
- argshuffle
- argsort
- binary-argfilter
- binary-search
- binary-search-impl
- ensure-reader
- find-base-comparator
- impl-index-of
- index-comparator
- index-of
- index-reducer
- last-index-of
- tech-numerics-kwd?
tech.v3.datatype.bitmap
Functions for working with RoaringBitmaps. These are integrated deeply into several tech.v3.datatype algorithms and have many potential applications in high performance computing applications as they are both extremely fast and storage-space efficient.
Public variables and functions:
tech.v3.datatype.convolve
Namespace for implementing various basic convolutions. Currently only 1d convolutions are supported.
Public variables and functions:
tech.v3.datatype.datetime
Thorough bindings to java.time.datetime. Includes packed datatypes and lifting various datetime datatypes into the datatype system.
Public variables and functions:
- between
- datetime->epoch
- datetime->milliseconds
- datetime-datatype?
- days-since-epoch->local-date
- duration
- duration->milliseconds
- duration->nanoseconds
- duration-datatype?
- epoch->datetime
- epoch-days->epoch-months
- epoch-months->epoch-days
- epoch-months->local-date
- instant
- instant->local-date-time
- instant->microseconds-since-epoch
- instant->milliseconds-since-epoch
- instant->seconds-since-epoch
- instant->zoned-date-time
- local-date
- local-date->days-since-epoch
- local-date->epoch-months
- local-date->instant
- local-date->local-date-time
- local-date->milliseconds-since-epoch
- local-date->zoned-date-time
- local-date-time
- local-date-time->instant
- local-date-time->local-date
- local-date-time->local-time
- local-date-time->milliseconds-since-epoch
- local-date-time->zoned-date-time
- local-time
- local-time->instant
- local-time->local-date-time
- local-time->milliseconds
- local-time->seconds
- long-temporal-field
- microseconds-since-epoch->instant
- millisecond-descriptive-statistics
- milliseconds->datetime
- milliseconds->duration
- milliseconds->local-time
- milliseconds-in-day
- milliseconds-in-hour
- milliseconds-in-minute
- milliseconds-in-second
- milliseconds-in-week
- milliseconds-since-epoch->instant
- milliseconds-since-epoch->local-date
- milliseconds-since-epoch->local-date-time
- milliseconds-since-epoch->local-time
- milliseconds-since-epoch->zoned-date-time
- minus-temporal-amount
- nanoseconds->duration
- nanoseconds-in-day
- nanoseconds-in-hour
- nanoseconds-in-millisecond
- nanoseconds-in-minute
- nanoseconds-in-second
- nanoseconds-in-week
- plus-temporal-amount
- seconds->local-time
- seconds-in-day
- seconds-in-hour
- seconds-in-minute
- seconds-since-epoch->instant
- system-zone-id
- system-zone-offset
- utc-zone-id
- utc-zone-offset
- variable-rolling-window-ranges
- zoned-date-time
- zoned-date-time->instant
- zoned-date-time->milliseconds-since-epoch
tech.v3.datatype.errors
Generic, basic error handling. No dependencies aside from apache commons math for NAN exception.
Public variables and functions:
tech.v3.datatype.ffi
Generalized C Foreign Function Interface (FFI) that unifies JNA and JDK-16 FFI architectures.
Public variables and functions:
- c->string
- clear-string->c-stats!
- define-foreign-interface
- define-library
- define-library-functions
- define-library-interface
- enable-string->c-stats!
- find-library
- find-symbol
- foreign-interface-instance->c
- if-class
- instantiate-class
- instantiate-foreign-interface
- instantiate-library
- jdk-ffi?
- jdk-mmodel?
- jna-ffi?
- library-loadable?
- library-singleton
- make-ptr
- PLibrarySingleton
- PToPointer
- ptr->struct
- set-ffi-impl!
- string->c
- string->c-data-histogram
- struct-member-ptr
- utf16-platform-encoding
- utf32-platform-encoding
- windows-platform?
tech.v3.datatype.ffi.clang
General utilites to work with clang in order to define things like structs or enumerations
Public variables and functions:
tech.v3.datatype.ffi.graalvm
Graalvm-specific namespace that implements the dtype-next ffi system and allows users to build stand-alone shared libraries that other languages can use to call Clojure code via a binary C interface.
Public variables and functions:
tech.v3.datatype.functional
Arithmetic and statistical operations based on the Buffer interface. These operators and functions all implement vectorized interfaces so passing in something convertible to a reader will return a reader. Arithmetic operations are done lazily. These functions generally incur a large dispatch cost so for example each call to ‘+’ checks all the arguments to decide if it should dispatch to an iterable implementation or to a reader implementation. For tight loops or operations like map and filter, using the specific operators will result in far faster code than using the ‘+’ function itself.
Public variables and functions:
- *
- +
- -
- /
- <
- <=
- >
- >=
- abs
- acos
- and
- asin
- atan
- atan2
- bit-and
- bit-and-not
- bit-clear
- bit-flip
- bit-not
- bit-or
- bit-set
- bit-shift-left
- bit-shift-right
- bit-test
- bit-xor
- bool-reader->indexes
- cbrt
- ceil
- cos
- cosh
- cummax
- cummin
- cumprod
- cumsum
- descriptive-statistics
- distance
- distance-squared
- dot-product
- eq
- equals
- even?
- exp
- expm1
- fill-range
- finite?
- fixed-rolling-window
- floor
- get-significand
- hypot
- identity
- ieee-remainder
- infinite?
- kendalls-correlation
- kurtosis
- linear-regressor
- log
- log10
- log1p
- logistic
- magnitude
- magnitude-squared
- mathematical-integer?
- max
- mean
- median
- min
- nan?
- neg?
- next-down
- next-up
- not
- not-eq
- odd?
- or
- pearsons-correlation
- percentiles
- pos?
- pow
- quartile-1
- quartile-3
- quartile-outlier-fn
- quartiles
- quot
- reduce-*
- reduce-+
- reduce-max
- reduce-min
- rem
- rint
- round
- shift
- signum
- sin
- sinh
- skew
- spearmans-correlation
- sq
- sqrt
- standard-deviation
- sum
- tan
- tanh
- to-degrees
- to-radians
- ulp
- unsigned-bit-shift-right
- variance
- zero?
tech.v3.datatype.gradient
Calculate the numeric gradient of successive elements. Contains simplified versions of numpy.gradient and numpy.diff.
Public variables and functions:
tech.v3.datatype.jna
Protocol bindings to the tech.jna system. Most users will not need to require this namespace.
Public variables and functions:
tech.v3.datatype.jvm-map
Creation and higher-level manipulation for java.util.HashMap and java.util.concurrent.ConcurrentHashMap.
Public variables and functions:
- ->bi-consumer
- ->bi-function
- ->function
- add!
- bi-consumer
- bi-function
- compute!
- compute-fn!
- compute-if-absent!
- compute-if-absent-fn!
- compute-if-present!
- compute-if-present-fn!
- concurrent-hash-map
- concurrent-hash-map?
- concurrent-set
- contains?
- entry-key
- entry-value
- foreach!
- function
- get
- hash-map
- into-map!
- merge-with!
- put!
tech.v3.datatype.mmap-writer
Provides the ability to efficiently write binary data to a file with an automatic conversion to a native-buffer using mmap.
Public variables and functions:
tech.v3.datatype.native-buffer
Support for malloc/free and generalized support for reading/writing typed data to a long integer address of memory.
tech.v3.datatype.nippy
Nippy bindings for datatype base types and tensor types
Public variables and functions:
tech.v3.datatype.packing
Implements the ‘packed/unpacked’ concept in datatype. This allows us to take objects like LocalDate’s and store them in int32 storage which dramatically decreases their size.
Public variables and functions:
tech.v3.datatype.reductions
High performance reductions based on tech.v3.datatype concepts as well as java stream concepts.
Public variables and functions:
- commutative-binary-double
- commutative-binary-long
- commutative-binary-object
- commutative-binary-reduce
- double-reductions
- double-summation
- nan-strategies
- nan-strategy->double-predicate
- ordered-group-by-reduce
- reader->double-spliterator
- reducer-value->consumer-fn
- staged-double-consumer-reduction
- unary-double-summation
- unordered-group-by-reduce
tech.v3.datatype.sampling
Implementation of reservoir sampling designed to be used in other systems. Provides a low-level sampler object and a double-reservoir that implements DoubleConsumer and derefs to a buffer of doubles.
Public variables and functions:
tech.v3.datatype.struct
Structs are datatypes composed of primitive datatypes or other structs. Similar to records except they do not support string or object columns, only numeric values. They have memset-0 initialization, memcpy copy semantics. For correct equals, hashing, convert struct into a normal persistent map via into.
Public variables and functions:
tech.v3.datatype.wavelet
Implementation of the scipy continuous wavelet transform. See fastmath for implementations of the discrete wavelet transform.
tech.v3.libs.buffered-image
Bindings to buffered images for the datatype system
tech.v3.libs.neanderthal
Implementation of the various datatype protocols for neanderthal datatypes. Users must require this to enable ->reader and as-tensor functionality for neanderthal datatypes.
Public variables and functions:
tech.v3.parallel.for
Serial and parallel iteration strategies across iterators and index spaces.
tech.v3.tensor
ND bindings for the tech.v3.datatype system. A Tensor is conceptually just a tuple of a buffer and an index operator that is capable of converting indexes in ND space into a single long index into the buffer. Tensors implement the tech.v3.datatype.NDBuffer interface and outside this file ND objects are expected to simply implement that interface.
Public variables and functions:
- ->jvm
- ->tensor
- as-tensor
- broadcast
- clone
- columns
- compute-tensor
- const-tensor
- construct-tensor
- dimensions-dense?
- dims-suitable-for-desc?
- ensure-native
- ensure-nd-buffer-descriptor
- ensure-tensor
- mget
- mset!
- native-tensor
- nd-buffer->buffer-reader
- nd-buffer-descriptor->tensor
- nd-copy!
- new-tensor
- reduce-axis
- reshape
- rotate
- rows
- select
- simple-dimensions?
- slice
- slice-right
- tensor->buffer
- tensor->dimensions
- tensor-copy!
- tensor?
- transpose
- typed-compute-tensor
tech.v3.tensor.color-gradients
Implement a mapping from double->color for each entry in a tensor. Produces an image of the same dimensions in pixels as the input tensor.
Public variables and functions:
tech.v3.tensor.dimensions
Dimensions implement a projection from ND space in the the address space of the buffer along with a reverse projection from the address space back into ND space.
Public variables and functions:
- ->2d-shape
- ->batch-shape
- ->global->local
- ->least-rapidly-changing-dimension
- ->local->global
- ->most-rapidly-changing-dimension
- access-increasing?
- broadcast
- buffer-ecount
- create-dimension-transforms
- dense?
- dimensions
- dimensions->column-stride
- direct?
- ecount
- get-elem-dims-local->global
- in-place-reshape
- indirect?
- local-address->local-shape
- matrix-column-stride
- matrix-element-stride
- native?
- rotate
- select
- shape
- slice
- slice-right
- strides
- trans-2d-shape
- transpose
Generated by Codox with RDash UI theme
dtype-next 8.034
dtype-next 8.034
A Clojure library designed to aid in the implementation of high performance algorithms and systems.
Topics
Namespaces
tech.v3.datatype
Base namespace for container creation and elementwise access of data
Public variables and functions:
- ->array
- ->array-buffer
- ->buffer
- ->byte-array
- ->char-array
- ->double-array
- ->float-array
- ->int-array
- ->iterable
- ->long-array
- ->native-buffer
- ->reader
- ->short-array
- ->vector
- ->writer
- arg-type
- as-array-buffer
- as-buffer
- as-concrete-buffer
- as-native-buffer
- as-nd-buffer-descriptor
- as-persistent-vector
- as-reader
- as-roaring-bitmap
- as-writer
- cast
- clone
- coalesce!
- coalesce-blocks!
- concat-buffers
- const-reader
- copy!
- copy-raw->item!
- datatype
- ecount
- elemwise-cast
- elemwise-datatype
- emap
- ensure-reader
- ensure-serializeable
- get-datatype
- get-value
- indexed-buffer
- iterable?
- make-container
- make-list
- make-reader
- reader-like?
- reader?
- set-constant!
- set-datatype
- set-value!
- shape
- sub-buffer
- unchecked-cast
- vectorized-dispatch-1
- vectorized-dispatch-2
- writer?
tech.v3.datatype.argops
Efficient functions for operating in index space. Take-off of the argsort, argmin, etc. type functions from Matlab. These functions generally only work on readers and all return some version of an index or list of indexes.
Public variables and functions:
- ->binary-operator
- ->binary-predicate
- ->comparator
- ->double-comparator
- ->long-comparator
- ->unary-operator
- ->unary-predicate
- arg-min-n
- argfilter
- arggroup
- arggroup-by
- arglast-every
- argmax
- argmin
- argpartition
- argpartition-by
- argshuffle
- argsort
- binary-argfilter
- binary-search
- binary-search-impl
- ensure-reader
- find-base-comparator
- impl-index-of
- index-comparator
- index-of
- index-reducer
- last-index-of
- tech-numerics-kwd?
tech.v3.datatype.bitmap
Functions for working with RoaringBitmaps. These are integrated deeply into several tech.v3.datatype algorithms and have many potential applications in high performance computing applications as they are both extremely fast and storage-space efficient.
Public variables and functions:
tech.v3.datatype.convolve
Namespace for implementing various basic convolutions. Currently only 1d convolutions are supported.
Public variables and functions:
tech.v3.datatype.datetime
Thorough bindings to java.time.datetime. Includes packed datatypes and lifting various datetime datatypes into the datatype system.
Public variables and functions:
- between
- datetime->epoch
- datetime->milliseconds
- datetime-datatype?
- days-since-epoch->local-date
- duration
- duration->milliseconds
- duration->nanoseconds
- duration-datatype?
- epoch->datetime
- epoch-days->epoch-months
- epoch-months->epoch-days
- epoch-months->local-date
- instant
- instant->local-date-time
- instant->microseconds-since-epoch
- instant->milliseconds-since-epoch
- instant->seconds-since-epoch
- instant->zoned-date-time
- local-date
- local-date->days-since-epoch
- local-date->epoch-months
- local-date->instant
- local-date->local-date-time
- local-date->milliseconds-since-epoch
- local-date->zoned-date-time
- local-date-time
- local-date-time->instant
- local-date-time->local-date
- local-date-time->local-time
- local-date-time->milliseconds-since-epoch
- local-date-time->zoned-date-time
- local-time
- local-time->instant
- local-time->local-date-time
- local-time->milliseconds
- local-time->seconds
- long-temporal-field
- microseconds-since-epoch->instant
- millisecond-descriptive-statistics
- milliseconds->datetime
- milliseconds->duration
- milliseconds->local-time
- milliseconds-in-day
- milliseconds-in-hour
- milliseconds-in-minute
- milliseconds-in-second
- milliseconds-in-week
- milliseconds-since-epoch->instant
- milliseconds-since-epoch->local-date
- milliseconds-since-epoch->local-date-time
- milliseconds-since-epoch->local-time
- milliseconds-since-epoch->zoned-date-time
- minus-temporal-amount
- nanoseconds->duration
- nanoseconds-in-day
- nanoseconds-in-hour
- nanoseconds-in-millisecond
- nanoseconds-in-minute
- nanoseconds-in-second
- nanoseconds-in-week
- plus-temporal-amount
- seconds->local-time
- seconds-in-day
- seconds-in-hour
- seconds-in-minute
- seconds-since-epoch->instant
- system-zone-id
- system-zone-offset
- utc-zone-id
- utc-zone-offset
- variable-rolling-window-ranges
- zoned-date-time
- zoned-date-time->instant
- zoned-date-time->milliseconds-since-epoch
tech.v3.datatype.errors
Generic, basic error handling. No dependencies aside from apache commons math for NAN exception.
Public variables and functions:
tech.v3.datatype.ffi
Generalized C Foreign Function Interface (FFI) that unifies JNA and JDK-16 FFI architectures.
Public variables and functions:
- c->string
- clear-string->c-stats!
- define-foreign-interface
- define-library
- define-library-functions
- define-library-interface
- enable-string->c-stats!
- find-library
- find-symbol
- foreign-interface-instance->c
- if-class
- instantiate-class
- instantiate-foreign-interface
- instantiate-library
- jdk-ffi?
- jdk-mmodel?
- jna-ffi?
- library-loadable?
- library-singleton
- make-ptr
- PLibrarySingleton
- PToPointer
- ptr->struct
- set-ffi-impl!
- string->c
- string->c-data-histogram
- struct-member-ptr
- utf16-platform-encoding
- utf32-platform-encoding
- windows-platform?
tech.v3.datatype.ffi.clang
General utilites to work with clang in order to define things like structs or enumerations
Public variables and functions:
tech.v3.datatype.ffi.graalvm
Graalvm-specific namespace that implements the dtype-next ffi system and allows users to build stand-alone shared libraries that other languages can use to call Clojure code via a binary C interface.
Public variables and functions:
tech.v3.datatype.functional
Arithmetic and statistical operations based on the Buffer interface. These operators and functions all implement vectorized interfaces so passing in something convertible to a reader will return a reader. Arithmetic operations are done lazily. These functions generally incur a large dispatch cost so for example each call to ‘+’ checks all the arguments to decide if it should dispatch to an iterable implementation or to a reader implementation. For tight loops or operations like map and filter, using the specific operators will result in far faster code than using the ‘+’ function itself.
Public variables and functions:
- *
- +
- -
- /
- <
- <=
- >
- >=
- abs
- acos
- and
- asin
- atan
- atan2
- bit-and
- bit-and-not
- bit-clear
- bit-flip
- bit-not
- bit-or
- bit-set
- bit-shift-left
- bit-shift-right
- bit-test
- bit-xor
- bool-reader->indexes
- cbrt
- ceil
- cos
- cosh
- cummax
- cummin
- cumprod
- cumsum
- descriptive-statistics
- distance
- distance-squared
- dot-product
- eq
- equals
- even?
- exp
- expm1
- fill-range
- finite?
- fixed-rolling-window
- floor
- get-significand
- hypot
- identity
- ieee-remainder
- infinite?
- kendalls-correlation
- kurtosis
- linear-regressor
- log
- log10
- log1p
- logistic
- magnitude
- magnitude-squared
- mathematical-integer?
- max
- mean
- median
- min
- nan?
- neg?
- next-down
- next-up
- not
- not-eq
- odd?
- or
- pearsons-correlation
- percentiles
- pos?
- pow
- quartile-1
- quartile-3
- quartile-outlier-fn
- quartiles
- quot
- reduce-*
- reduce-+
- reduce-max
- reduce-min
- rem
- rint
- round
- shift
- signum
- sin
- sinh
- skew
- spearmans-correlation
- sq
- sqrt
- standard-deviation
- sum
- tan
- tanh
- to-degrees
- to-radians
- ulp
- unsigned-bit-shift-right
- variance
- zero?
tech.v3.datatype.gradient
Calculate the numeric gradient of successive elements. Contains simplified versions of numpy.gradient and numpy.diff.
Public variables and functions:
tech.v3.datatype.jna
Protocol bindings to the tech.jna system. Most users will not need to require this namespace.
Public variables and functions:
tech.v3.datatype.jvm-map
Creation and higher-level manipulation for java.util.HashMap and java.util.concurrent.ConcurrentHashMap.
Public variables and functions:
- ->bi-consumer
- ->bi-function
- ->function
- add!
- bi-consumer
- bi-function
- compute!
- compute-fn!
- compute-if-absent!
- compute-if-absent-fn!
- compute-if-present!
- compute-if-present-fn!
- concurrent-hash-map
- concurrent-hash-map?
- concurrent-set
- contains?
- entry-key
- entry-value
- foreach!
- function
- get
- hash-map
- into-map!
- merge-with!
- put!
tech.v3.datatype.mmap-writer
Provides the ability to efficiently write binary data to a file with an automatic conversion to a native-buffer using mmap.
Public variables and functions:
tech.v3.datatype.native-buffer
Support for malloc/free and generalized support for reading/writing typed data to a long integer address of memory.
tech.v3.datatype.nippy
Nippy bindings for datatype base types and tensor types
Public variables and functions:
tech.v3.datatype.packing
Implements the ‘packed/unpacked’ concept in datatype. This allows us to take objects like LocalDate’s and store them in int32 storage which dramatically decreases their size.
Public variables and functions:
tech.v3.datatype.reductions
High performance reductions based on tech.v3.datatype concepts as well as java stream concepts.
Public variables and functions:
- commutative-binary-double
- commutative-binary-long
- commutative-binary-object
- commutative-binary-reduce
- double-reductions
- double-summation
- nan-strategies
- nan-strategy->double-predicate
- ordered-group-by-reduce
- reader->double-spliterator
- reducer-value->consumer-fn
- staged-double-consumer-reduction
- unary-double-summation
- unordered-group-by-reduce
tech.v3.datatype.sampling
Implementation of reservoir sampling designed to be used in other systems. Provides a low-level sampler object and a double-reservoir that implements DoubleConsumer and derefs to a buffer of doubles.
Public variables and functions:
tech.v3.datatype.struct
Structs are datatypes composed of primitive datatypes or other structs. Similar to records except they do not support string or object columns, only numeric values. They have memset-0 initialization, memcpy copy semantics. For correct equals, hashing, convert struct into a normal persistent map via into.
Public variables and functions:
tech.v3.datatype.wavelet
Implementation of the scipy continuous wavelet transform. See fastmath for implementations of the discrete wavelet transform.
tech.v3.libs.buffered-image
Bindings to buffered images for the datatype system
tech.v3.libs.neanderthal
Implementation of the various datatype protocols for neanderthal datatypes. Users must require this to enable ->reader and as-tensor functionality for neanderthal datatypes.
Public variables and functions:
tech.v3.parallel.for
Serial and parallel iteration strategies across iterators and index spaces.
tech.v3.tensor
ND bindings for the tech.v3.datatype system. A Tensor is conceptually just a tuple of a buffer and an index operator that is capable of converting indexes in ND space into a single long index into the buffer. Tensors implement the tech.v3.datatype.NDBuffer interface and outside this file ND objects are expected to simply implement that interface.
Public variables and functions:
- ->jvm
- ->tensor
- as-tensor
- broadcast
- clone
- columns
- compute-tensor
- const-tensor
- construct-tensor
- dimensions-dense?
- dims-suitable-for-desc?
- ensure-native
- ensure-nd-buffer-descriptor
- ensure-tensor
- mget
- mset!
- native-tensor
- nd-buffer->buffer-reader
- nd-buffer-descriptor->tensor
- nd-copy!
- new-tensor
- reduce-axis
- reshape
- rotate
- rows
- select
- simple-dimensions?
- slice
- slice-right
- tensor->buffer
- tensor->dimensions
- tensor-copy!
- tensor?
- transpose
- typed-compute-tensor
tech.v3.tensor.color-gradients
Implement a mapping from double->color for each entry in a tensor. Produces an image of the same dimensions in pixels as the input tensor.
Public variables and functions:
tech.v3.tensor.dimensions
Dimensions implement a projection from ND space in the the address space of the buffer along with a reverse projection from the address space back into ND space.
Public variables and functions:
- ->2d-shape
- ->batch-shape
- ->global->local
- ->least-rapidly-changing-dimension
- ->local->global
- ->most-rapidly-changing-dimension
- access-increasing?
- broadcast
- buffer-ecount
- create-dimension-transforms
- dense?
- dimensions
- dimensions->column-stride
- direct?
- ecount
- get-elem-dims-local->global
- in-place-reshape
- indirect?
- local-address->local-shape
- matrix-column-stride
- matrix-element-stride
- native?
- rotate
- select
- shape
- slice
- slice-right
- strides
- trans-2d-shape
- transpose
Generated by Codox with RDash UI theme
dtype-next 8.033
Overview
+Generated by Codox with RDash UI theme
dtype-next 8.034
Overview
Setup
(require '[tech.v3.datatype :as dtype])
Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.argops
Efficient functions for operating in index space. Take-off of the argsort, argmin, etc. type functions from Matlab. These functions generally only work on readers and all return some version of an index or list of indexes.
->binary-operator
(->binary-operator op)Convert a thing to a binary operator. Thing can be a keyword or an implementation of IFn or an implementation of a BinaryOperator.
->binary-predicate
(->binary-predicate op)Convert a thing to a binary predicate. Thing can be a keyword or an implementation of IFn or an implementation of a BinaryPredicate.
->double-comparator
(->double-comparator src-comparator)Convert a thing to a it.unimi.dsi.fastutil.doubles.DoubleComparator.
->long-comparator
(->long-comparator src-comparator)Convert a thing to a it.unimi.dsi.fastutil.longs.LongComparator.
->unary-operator
(->unary-operator op)Convert a thing to a unary operator. Thing can be a keyword or an implementation of IFn or an implementation of a UnaryOperator.
->unary-predicate
(->unary-predicate op)Convert a thing to a unary predicate. Thing can be a keyword or an implementation of IFn or an implementation of a UnaryPredicate.
arg-min-n
(arg-min-n N comparator {:keys [nan-strategy], :or {nan-strategy :last}} values)(arg-min-n N comparator values)(arg-min-n N values)Return the indexes of the top minimum items. Values must be countable and random access. Same options,arguments as argsort.
argfilter
(argfilter pred options rdr)(argfilter pred rdr)Filter out values returning either an iterable of indexes or a reader of indexes.
arggroup
(arggroup {:keys [storage-datatype unordered?]} rdr)(arggroup rdr)Group by elemens in the reader returning a map of value->list of indexes.
+Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.argops
Efficient functions for operating in index space. Take-off of the argsort, argmin, etc. type functions from Matlab. These functions generally only work on readers and all return some version of an index or list of indexes.
->binary-operator
(->binary-operator op)Convert a thing to a binary operator. Thing can be a keyword or an implementation of IFn or an implementation of a BinaryOperator.
->binary-predicate
(->binary-predicate op)Convert a thing to a binary predicate. Thing can be a keyword or an implementation of IFn or an implementation of a BinaryPredicate.
->double-comparator
(->double-comparator src-comparator)Convert a thing to a it.unimi.dsi.fastutil.doubles.DoubleComparator.
->long-comparator
(->long-comparator src-comparator)Convert a thing to a it.unimi.dsi.fastutil.longs.LongComparator.
->unary-operator
(->unary-operator op)Convert a thing to a unary operator. Thing can be a keyword or an implementation of IFn or an implementation of a UnaryOperator.
->unary-predicate
(->unary-predicate op)Convert a thing to a unary predicate. Thing can be a keyword or an implementation of IFn or an implementation of a UnaryPredicate.
arg-min-n
(arg-min-n N comparator {:keys [nan-strategy], :or {nan-strategy :last}} values)(arg-min-n N comparator values)(arg-min-n N values)Return the indexes of the top minimum items. Values must be countable and random access. Same options,arguments as argsort.
argfilter
(argfilter pred options rdr)(argfilter pred rdr)Filter out values returning either an iterable of indexes or a reader of indexes.
arggroup
(arggroup {:keys [storage-datatype unordered?]} rdr)(arggroup rdr)Group by elemens in the reader returning a map of value->list of indexes.
Options:
- storage-datatype - :int32, :int64, or :bitmap, defaults to whatever will fit based on the element count of the reader. diff --git a/docs/tech.v3.datatype.bitmap.html b/docs/tech.v3.datatype.bitmap.html index 4ebccf13..6f090596 100644 --- a/docs/tech.v3.datatype.bitmap.html +++ b/docs/tech.v3.datatype.bitmap.html @@ -1,3 +1,3 @@ -
-
diff --git a/docs/tech.v3.datatype.errors.html b/docs/tech.v3.datatype.errors.html
index a81bbaa4..365fd871 100644
--- a/docs/tech.v3.datatype.errors.html
+++ b/docs/tech.v3.datatype.errors.html
@@ -1,3 +1,3 @@
-
tech.v3.datatype.errors documentation Generated by Codox with RDash UI theme
dtype-next 8.033
\ No newline at end of file +tech.v3.datatype.errors
Generic, basic error handling. No dependencies aside from apache commons math for NAN exception.
check-idx
macro
(check-idx idx n-elems)Check that an index is less than n-elems. Throw an index-out-of-bounds exception if that isn’t the case.
check-nan-error
macro
(check-nan-error arg)Check if arg is double NaN and throw an exception if so.
check-offset-length
macro
(check-offset-length offset length elem-ecount)Check that a combined offset and length fit within elem-ecount. Throw an exception with a nice message in the case where bounds were exceeded.
throw-index-out-of-boundsf
macro
(throw-index-out-of-boundsf msg & args)Throw an index out of bounds exception with a nice message.
when-not-error
macro
(when-not-error expr error-msg)Throw an error in the case where expr isn’t true.
when-not-errorf
macro
(when-not-errorf expr error-msg & args)Throw an exception with formatting in the case where expr isn’t true.
tech.v3.datatype.errors documentation Generated by Codox with RDash UI theme
dtype-next 8.034
\ No newline at end of file diff --git a/docs/tech.v3.datatype.ffi.clang.html b/docs/tech.v3.datatype.ffi.clang.html index 88a1a631..f4439b6d 100644 --- a/docs/tech.v3.datatype.ffi.clang.html +++ b/docs/tech.v3.datatype.ffi.clang.html @@ -1,6 +1,6 @@ -tech.v3.datatype.errors
Generic, basic error handling. No dependencies aside from apache commons math for NAN exception.
check-idx
macro
(check-idx idx n-elems)Check that an index is less than n-elems. Throw an index-out-of-bounds exception if that isn’t the case.
check-nan-error
macro
(check-nan-error arg)Check if arg is double NaN and throw an exception if so.
check-offset-length
macro
(check-offset-length offset length elem-ecount)Check that a combined offset and length fit within elem-ecount. Throw an exception with a nice message in the case where bounds were exceeded.
throw-index-out-of-boundsf
macro
(throw-index-out-of-boundsf msg & args)Throw an index out of bounds exception with a nice message.
when-not-error
macro
(when-not-error expr error-msg)Throw an error in the case where expr isn’t true.
when-not-errorf
macro
(when-not-errorf expr error-msg & args)Throw an exception with formatting in the case where expr isn’t true.
tech.v3.datatype.ffi.clang documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.ffi.clang
General utilites to work with clang in order to define things like structs or enumerations
defstruct-from-layout
(defstruct-from-layout struct-dtype layout {:keys [failed-line-parser]})(defstruct-from-layout struct-dtype layout)Function to take a partial clang record dump and make a struct defintion. In order to generate a file containing dumps we do something like:
+tech.v3.datatype.ffi.clang documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.ffi.clang
General utilites to work with clang in order to define things like structs or enumerations
defstruct-from-layout
(defstruct-from-layout struct-dtype layout {:keys [failed-line-parser]})(defstruct-from-layout struct-dtype layout)Function to take a partial clang record dump and make a struct defintion. In order to generate a file containing dumps we do something like:
clang avcodec.cpp -I/usr/include/x86_64-linux-gnu/libavcodec -Xclang -fdump-record-layouts > recordlayouts64.txtNote the
diff --git a/docs/tech.v3.datatype.ffi.graalvm.html b/docs/tech.v3.datatype.ffi.graalvm.html index ab018691..8cc3608f 100644 --- a/docs/tech.v3.datatype.ffi.graalvm.html +++ b/docs/tech.v3.datatype.ffi.graalvm.html @@ -1,6 +1,6 @@ --fdump-record-layoutsargument.tech.v3.datatype.ffi.graalvm documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.ffi.graalvm
Graalvm-specific namespace that implements the dtype-next ffi system and allows users to build stand-alone shared libraries that other languages can use to call Clojure code via a binary C interface.
define-library
(define-library fn-defs symbols {:keys [classname instantiate?], :as options})Define a graal-native set functions bound to a particular native library. See the documentation for tech.v3.datatype.ffi/define-library.
expose-clojure-functions
(expose-clojure-functions fn-defs classname options)Expose a set of clojure functions as graal library entry points. In this case, the keys of fn-defs are the full namespaced symbols that point to the fns you want to define. These will be defined to a class named ‘classname’ and the resuling class file will be output to
+*compile-path*.tech.v3.datatype.ffi.graalvm documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.ffi.graalvm
Graalvm-specific namespace that implements the dtype-next ffi system and allows users to build stand-alone shared libraries that other languages can use to call Clojure code via a binary C interface.
define-library
(define-library fn-defs symbols {:keys [classname instantiate?], :as options})Define a graal-native set functions bound to a particular native library. See the documentation for tech.v3.datatype.ffi/define-library.
expose-clojure-functions
(expose-clojure-functions fn-defs classname options)Expose a set of clojure functions as graal library entry points. In this case, the keys of fn-defs are the full namespaced symbols that point to the fns you want to define. These will be defined to a class named ‘classname’ and the resuling class file will be output to
*compile-path*.One caveat - strings are passed as tech.v3.datatype.ffi.Pointer classes and you will have to use tech.v3.datatype.ffi/c->string in order to process them.
Any persistent state must be referenced also from your main class in your jarfile so it will have to reference systems used in your libfile. ‘def’, ‘defonce’ variables will show up as uninitialized exceptions at runtime when objects call into your library if they are not referenced in some manner from you jar’s main function. This can be achieved via several ways, one of which is to have your exposed namespace referenced from your main namespace or to have your library export file share a common namespace with your main namespace.
-
diff --git a/docs/tech.v3.datatype.ffi.html b/docs/tech.v3.datatype.ffi.html
index 8ba80f96..f4918b74 100644
--- a/docs/tech.v3.datatype.ffi.html
+++ b/docs/tech.v3.datatype.ffi.html
@@ -1,6 +1,6 @@
-
:nan-strategy- one of:keep,:remove,:exception. Defaults to:remove.
diff --git a/docs/tech.v3.datatype.gradient.html b/docs/tech.v3.datatype.gradient.html
index 4b33a928..55f7aca2 100644
--- a/docs/tech.v3.datatype.gradient.html
+++ b/docs/tech.v3.datatype.gradient.html
@@ -1,6 +1,6 @@
-:prepend- prepend a value to the returned sequence making it the same length as the passed in sequence.
diff --git a/docs/tech.v3.datatype.html b/docs/tech.v3.datatype.html
index f1756607..b85066ce 100644
--- a/docs/tech.v3.datatype.html
+++ b/docs/tech.v3.datatype.html
@@ -1,6 +1,6 @@
-- number? - Used to initializes the hashtable size. diff --git a/docs/tech.v3.datatype.list.html b/docs/tech.v3.datatype.list.html index 0639f676..20ee1d37 100644 --- a/docs/tech.v3.datatype.list.html +++ b/docs/tech.v3.datatype.list.html @@ -1,3 +1,3 @@ -
:resource-type- defaults to:gc- maps to:track-typeintech.v3.resourcebut can also be set to nil in which case the data is not tracked this library will not clean it up.
diff --git a/docs/tech.v3.datatype.nippy.html b/docs/tech.v3.datatype.nippy.html
index ba341c0e..161ea148 100644
--- a/docs/tech.v3.datatype.nippy.html
+++ b/docs/tech.v3.datatype.nippy.html
@@ -1,3 +1,3 @@
-:relative-window-position- Defaults to:center- controls the window’s relative positioning in the sequence.
diff --git a/docs/tech.v3.datatype.sampling.html b/docs/tech.v3.datatype.sampling.html
index aba7ca0d..d0c2523c 100644
--- a/docs/tech.v3.datatype.sampling.html
+++ b/docs/tech.v3.datatype.sampling.html
@@ -1,6 +1,6 @@
-:algorithm- either:lcgor:mersenne-twister. Defaults to:lcgwhich is the default java implementation.
diff --git a/docs/tech.v3.datatype.struct.html b/docs/tech.v3.datatype.struct.html
index e5b051ea..030e3d9c 100644
--- a/docs/tech.v3.datatype.struct.html
+++ b/docs/tech.v3.datatype.struct.html
@@ -1,6 +1,6 @@
-num-iters- Indexes are the values of(range num-iters).indexed-map-fn- Function that takes two integers, start-idx and group-len and returns a value. These values are then reduced usingreduce-fn.
diff --git a/docs/tech.v3.tensor.color-gradients.html b/docs/tech.v3.tensor.color-gradients.html
index bd704db9..5cf65e53 100644
--- a/docs/tech.v3.tensor.color-gradients.html
+++ b/docs/tech.v3.tensor.color-gradients.html
@@ -1,6 +1,6 @@
-
tech.v3.datatype.ffi documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.ffi
Generalized C Foreign Function Interface (FFI) that unifies JNA and JDK-16 FFI architectures.
+tech.v3.datatype.ffi documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.ffi
Generalized C Foreign Function Interface (FFI) that unifies JNA and JDK-16 FFI architectures.
Users can dynamically define and load libraries and define callbacks that C can then call.
This namespace is meant to work with the
structnamespace where you can define C-structs laid out precisely in memory.Available datatypes for the binding layer:
diff --git a/docs/tech.v3.datatype.functional.html b/docs/tech.v3.datatype.functional.html index abfa759b..b4a66846 100644 --- a/docs/tech.v3.datatype.functional.html +++ b/docs/tech.v3.datatype.functional.html @@ -1,6 +1,6 @@ -tech.v3.datatype.functional documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.functional
Arithmetic and statistical operations based on the Buffer interface. These operators and functions all implement vectorized interfaces so passing in something convertible to a reader will return a reader. Arithmetic operations are done lazily. These functions generally incur a large dispatch cost so for example each call to ‘+’ checks all the arguments to decide if it should dispatch to an iterable implementation or to a reader implementation. For tight loops or operations like map and filter, using the specific operators will result in far faster code than using the ‘+’ function itself.
bool-reader->indexes
(bool-reader->indexes {:keys [storage-type]} bool-item)(bool-reader->indexes bool-item)Given a reader, produce a filtered list of indexes filtering out ‘false’ values.
cummax
(cummax options data)(cummax data)Cumulative running max; returns result in double space.
+tech.v3.datatype.functional documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.functional
Arithmetic and statistical operations based on the Buffer interface. These operators and functions all implement vectorized interfaces so passing in something convertible to a reader will return a reader. Arithmetic operations are done lazily. These functions generally incur a large dispatch cost so for example each call to ‘+’ checks all the arguments to decide if it should dispatch to an iterable implementation or to a reader implementation. For tight loops or operations like map and filter, using the specific operators will result in far faster code than using the ‘+’ function itself.
bool-reader->indexes
(bool-reader->indexes {:keys [storage-type]} bool-item)(bool-reader->indexes bool-item)Given a reader, produce a filtered list of indexes filtering out ‘false’ values.
cummax
(cummax options data)(cummax data)Cumulative running max; returns result in double space.
Options:
tech.v3.datatype.gradient documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.gradient
Calculate the numeric gradient of successive elements. Contains simplified versions of numpy.gradient and numpy.diff.
diff1d
(diff1d data & [options])Returns a lazy reader of each successive element minus the previous element of length input-len - 1 unless the
+:prependoption is used.tech.v3.datatype.gradient documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.gradient
Calculate the numeric gradient of successive elements. Contains simplified versions of numpy.gradient and numpy.diff.
diff1d
(diff1d data & [options])Returns a lazy reader of each successive element minus the previous element of length input-len - 1 unless the
:prependoption is used.Options:
tech.v3.datatype documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype
Base namespace for container creation and elementwise access of data
->array
(->array datatype {:keys [nan-strategy], :or {nan-strategy :keep}, :as options} item)(->array datatype item)(->array item)Perform a NaN-aware conversion into an array. Default nan-strategy is :keep. Nan strategies can be: [:keep :remove :exception]
->array-buffer
(->array-buffer datatype {:keys [nan-strategy], :or {nan-strategy :keep}} item)(->array-buffer datatype item)(->array-buffer item)Perform a NaN-aware conversion into an array buffer. Default nan-strategy is :keep.
+tech.v3.datatype documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype
Base namespace for container creation and elementwise access of data
->array
(->array datatype {:keys [nan-strategy], :or {nan-strategy :keep}, :as options} item)(->array datatype item)(->array item)Perform a NaN-aware conversion into an array. Default nan-strategy is :keep. Nan strategies can be: [:keep :remove :exception]
->array-buffer
(->array-buffer datatype {:keys [nan-strategy], :or {nan-strategy :keep}} item)(->array-buffer datatype item)(->array-buffer item)Perform a NaN-aware conversion into an array buffer. Default nan-strategy is :keep.
Nan strategies can be: [:keep :remove :exception]
->buffer
(->buffer item)Convert this item to a buffer implementation or throw an exception.
->double-array
(->double-array options data)(->double-array data)Nan-aware conversion to double array. See documentation for ->array-buffer. Returns a double array. options - * nan-strategy - :keep (default) :remove :exception
->float-array
(->float-array options data)(->float-array data)Nan-aware conversion to double array. See documentation for ->array-buffer. Returns a double array. options - * nan-strategy - :keep (default) :remove :exception
->iterable
(->iterable item)Ensure this object is iterable. If item is iterable, return the item. If the item is convertible to a buffer object , convert to it and return the object. Else if the item is a scalar constant, return an infinite sequence of items.
->native-buffer
(->native-buffer item)Convert to a native buffer if possible, else throw an exception. See as-native-buffer
->reader
(->reader item)(->reader item read-datatype)If this object has a read-only or read-write conversion to a buffer object return the buffer object. Else throw an exception.
->writer
(->writer item)If this item is convertible to a PrimtiveIO writing the perform the conversion and return the interface. Else throw an excepion.
arg-type
(arg-type arg)Return the type of a thing. Types could be: :scalar :iterable :reader :tensor (reader with more than 1 dimension)
as-array-buffer
(as-array-buffer item)If this item is convertible to a tech.v3.datatype.array_buffer.ArrayBuffer then convert it and return the typed buffer
as-buffer
(as-buffer item)If this item is or has a conversion to an implementation of the Buffer interface then return the buffer interface for this object. Else return nil.
as-concrete-buffer
(as-concrete-buffer item)If this item is representable as a base buffer type, either native or array, return that buffer.
as-native-buffer
(as-native-buffer item)If this item is convertible to a tech.v3.datatype.native_buffer.NativeBuffer then convert it and return the typed buffer
as-nd-buffer-descriptor
(as-nd-buffer-descriptor src-item)If this item is convertible to a buffer descriptor, convert it. Else return nil.
Buffer descriptors are a ND description of data. For example, a native 3x3 tensor has a buffer description like thus:
{:ptr 140330005614816 diff --git a/docs/tech.v3.datatype.jna.html b/docs/tech.v3.datatype.jna.html index 4bf7ff35..eed289e4 100644 --- a/docs/tech.v3.datatype.jna.html +++ b/docs/tech.v3.datatype.jna.html @@ -1,3 +1,3 @@ -tech.v3.datatype.jna documentation Generated by Codox with RDash UI theme
dtype-next 8.033
\ No newline at end of file +tech.v3.datatype.jna
Protocol bindings to the tech.jna system. Most users will not need to require this namespace.
tech.v3.datatype.jna documentation Generated by Codox with RDash UI theme
dtype-next 8.034
\ No newline at end of file diff --git a/docs/tech.v3.datatype.jvm-map.html b/docs/tech.v3.datatype.jvm-map.html index 1359456e..1db0aee0 100644 --- a/docs/tech.v3.datatype.jvm-map.html +++ b/docs/tech.v3.datatype.jvm-map.html @@ -1,6 +1,6 @@ -tech.v3.datatype.jna
Protocol bindings to the tech.jna system. Most users will not need to require this namespace.
tech.v3.datatype.jvm-map documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.jvm-map
Creation and higher-level manipulation for
java.util.HashMapandjava.util.concurrent.ConcurrentHashMap.->bi-consumer
(->bi-consumer ifn)Take a thing and return a bi-consumer. If already a bi-consumer return as is, else calls
(bi-consumer x y (ifn x y))->bi-function
(->bi-function ifn)Take a thing and return a bi-function. If already a bi-function return as is, else calls
(bi-function x y (ifn x y))->function
(->function ifn)Convert an ifn to a java.util.function.Function. If ifn is already a Function, return with no changes. Else calls
(function x (ifn x))bi-consumer
macro
(bi-consumer karg varg code)Create a java.util.function.BiConsumer. karg will be the name of the first argument to the consumer, varg will be the name of the second. Code will be output inline into the consumer’s accept function.
bi-function
macro
(bi-function karg varg code)Create a java.util.function.BiFunction. karg will be the name of the first argument, varg will be the name of the second argument. code will be output inline into the function’s apply function.
compute!
(compute! map k val-fn)Compute a new value in-place. val-fn gets passed the key and existing value (if any). It is more efficient to pre-convert val-fn to a bi-function than to force this function to do it during execution.
compute-fn!
(compute-fn! map val-fn)Given a map and a function to call if the value is in the map, return a function that takes a k and returns the result of calling
(.compute map k (->bi-function val-fn)).val-fngets passed the key and existing value (if any).compute-if-absent!
(compute-if-absent! map k val-fn)Compute a new value in-place. val-fn gets passed the key. It is more efficient to pre-convert val-fn to a java.util.function.Function than to force this function to do it during its execution.
compute-if-absent-fn!
(compute-if-absent-fn! map val-fn)Given a map and a function to call if the value is not in the map, return a function that takes a k and returns the result of calling
(.compute map k (->function val-fn)).val-fngets passed the key when the value is not in the map.compute-if-present!
(compute-if-present! map k val-fn)Compute a new value in-place. val-fn gets passed the key and existing value. It is more efficient to pre-convert val-fn to a java.util.function.BiFunction than to force this function to do it during its execution.
compute-if-present-fn!
(compute-if-present-fn! map val-fn)Given a map and a function to call if the value is in the map, return a function that takes a k and returns the result of calling
(.computeIfPresent map k (->bi-function val-fn)).val-fngets passed the existing value and new value is in the map.concurrent-hash-map?
(concurrent-hash-map? item)Return true if this is a concurrent hash map.
concurrent-set
(concurrent-set)Create a concurrent-safe set. Wraps (.. (ConcurrentHashMap.) keySet)
contains?
(contains? item v)Much faster version of contains? if you are dealing with java.util sets or maps.
foreach!
(foreach! item op & [parallel?])Perform an operation for each entry in a map. Returns the op. This method will by default be serial unless parallel? is true in which case if the item in question is a concurrent hash map then the parallel version of foreach is used.
function
macro
(function karg code)Create a java.util.function.Function. karg will be the name of the argument, will be output inline into the function’s apply method.
get
(get map k)(get map k default-value)Get a value from a map with a potential default. Missing values will be nil otherwise.
hash-map
(hash-map)(hash-map init)Create a java.util.HashMap. Note the only optional argument is for the initial size of the underlying hashtable.
+tech.v3.datatype.jvm-map documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.jvm-map
Creation and higher-level manipulation for
java.util.HashMapandjava.util.concurrent.ConcurrentHashMap.->bi-consumer
(->bi-consumer ifn)Take a thing and return a bi-consumer. If already a bi-consumer return as is, else calls
(bi-consumer x y (ifn x y))->bi-function
(->bi-function ifn)Take a thing and return a bi-function. If already a bi-function return as is, else calls
(bi-function x y (ifn x y))->function
(->function ifn)Convert an ifn to a java.util.function.Function. If ifn is already a Function, return with no changes. Else calls
(function x (ifn x))bi-consumer
macro
(bi-consumer karg varg code)Create a java.util.function.BiConsumer. karg will be the name of the first argument to the consumer, varg will be the name of the second. Code will be output inline into the consumer’s accept function.
bi-function
macro
(bi-function karg varg code)Create a java.util.function.BiFunction. karg will be the name of the first argument, varg will be the name of the second argument. code will be output inline into the function’s apply function.
compute!
(compute! map k val-fn)Compute a new value in-place. val-fn gets passed the key and existing value (if any). It is more efficient to pre-convert val-fn to a bi-function than to force this function to do it during execution.
compute-fn!
(compute-fn! map val-fn)Given a map and a function to call if the value is in the map, return a function that takes a k and returns the result of calling
(.compute map k (->bi-function val-fn)).val-fngets passed the key and existing value (if any).compute-if-absent!
(compute-if-absent! map k val-fn)Compute a new value in-place. val-fn gets passed the key. It is more efficient to pre-convert val-fn to a java.util.function.Function than to force this function to do it during its execution.
compute-if-absent-fn!
(compute-if-absent-fn! map val-fn)Given a map and a function to call if the value is not in the map, return a function that takes a k and returns the result of calling
(.compute map k (->function val-fn)).val-fngets passed the key when the value is not in the map.compute-if-present!
(compute-if-present! map k val-fn)Compute a new value in-place. val-fn gets passed the key and existing value. It is more efficient to pre-convert val-fn to a java.util.function.BiFunction than to force this function to do it during its execution.
compute-if-present-fn!
(compute-if-present-fn! map val-fn)Given a map and a function to call if the value is in the map, return a function that takes a k and returns the result of calling
(.computeIfPresent map k (->bi-function val-fn)).val-fngets passed the existing value and new value is in the map.concurrent-hash-map?
(concurrent-hash-map? item)Return true if this is a concurrent hash map.
concurrent-set
(concurrent-set)Create a concurrent-safe set. Wraps (.. (ConcurrentHashMap.) keySet)
contains?
(contains? item v)Much faster version of contains? if you are dealing with java.util sets or maps.
foreach!
(foreach! item op & [parallel?])Perform an operation for each entry in a map. Returns the op. This method will by default be serial unless parallel? is true in which case if the item in question is a concurrent hash map then the parallel version of foreach is used.
function
macro
(function karg code)Create a java.util.function.Function. karg will be the name of the argument, will be output inline into the function’s apply method.
get
(get map k)(get map k default-value)Get a value from a map with a potential default. Missing values will be nil otherwise.
hash-map
(hash-map)(hash-map init)Create a java.util.HashMap. Note the only optional argument is for the initial size of the underlying hashtable.
Init can be of several forms:
tech.v3.datatype.list documentation Generated by Codox with RDash UI theme
dtype-next 8.033
\ No newline at end of file +tech.v3.datatype.list
make-list
(make-list initial-container ptr)(make-list datatype)Make a new primitive list out of a container and a ptr that indicates the current write position.
wrap-container
(wrap-container container)In-place wrap an existing container. Write ptr points to the end of the container so the next add* method will cause an allocation.
tech.v3.datatype.list documentation Generated by Codox with RDash UI theme
dtype-next 8.034
\ No newline at end of file diff --git a/docs/tech.v3.datatype.mmap-writer.html b/docs/tech.v3.datatype.mmap-writer.html index f89b1cdb..0265022e 100644 --- a/docs/tech.v3.datatype.mmap-writer.html +++ b/docs/tech.v3.datatype.mmap-writer.html @@ -1,6 +1,6 @@ -tech.v3.datatype.list
make-list
(make-list initial-container ptr)(make-list datatype)Make a new primitive list out of a container and a ptr that indicates the current write position.
wrap-container
(wrap-container container)In-place wrap an existing container. Write ptr points to the end of the container so the next add* method will cause an allocation.
tech.v3.datatype.mmap-writer documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.mmap-writer
Provides the ability to efficiently write binary data to a file with an automatic conversion to a native-buffer using mmap.
+tech.v3.datatype.mmap-writer documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.mmap-writer
Provides the ability to efficiently write binary data to a file with an automatic conversion to a native-buffer using mmap.
Endianness can be provided and strings are automatically saved as UTF-8.
mmap-writer
(mmap-writer fpath {:keys [endianness resource-type mmap-file-options open-options file-channel delete-on-close?], :or {resource-type :auto, endianness (dtype-proto/platform-endianness), open-options :append}, :as options})(mmap-writer fpath)Create a new data writer that has a conversion to a mmaped native buffer upon ->buffer or ->reader. Object metadata is passed to the mmap-file method so users can specify how the new file should interact with the resource system.
Binary data will be encoded to bytes using the endianness provided in the options. Strings are encoded to UTF-8 but may not be zero terminated!!.
Options:
diff --git a/docs/tech.v3.datatype.mmap.html b/docs/tech.v3.datatype.mmap.html index 14e0e0d4..050617dc 100644 --- a/docs/tech.v3.datatype.mmap.html +++ b/docs/tech.v3.datatype.mmap.html @@ -1,4 +1,4 @@ -tech.v3.datatype.mmap documentation Generated by Codox with RDash UI theme
dtype-next 8.033
\ No newline at end of file +tech.v3.datatype.mmap
mmap-file
(mmap-file fpath options)(mmap-file fpath)Memory map a file returning a native buffer. fpath must resolve to a valid java.io.File. Options * :resource-type - maps to tech.v3.resource
:track-type, defaults to :auto. * :mmap-mode * :read-only - default - map the data as shared read-only. * :read-write - map the data as shared read-write. * :private - map a private copy of the data and do not share.set-mmap-impl!
(set-mmap-impl! mmap-fn)Set the system mmap implementation to use. There are three provided under the mmap directory - *
-mmodel- Use the JDK-16 memory model. You have to have the module enabled. *larray- Use LArray mmap - default if available.If unset then the system will try to load the jdk-16 namespace, then larray.
tech.v3.datatype.mmap documentation Generated by Codox with RDash UI theme
dtype-next 8.034
\ No newline at end of file diff --git a/docs/tech.v3.datatype.native-buffer.html b/docs/tech.v3.datatype.native-buffer.html index 93f91296..2f2b4c1b 100644 --- a/docs/tech.v3.datatype.native-buffer.html +++ b/docs/tech.v3.datatype.native-buffer.html @@ -1,6 +1,6 @@ -tech.v3.datatype.mmap
mmap-file
(mmap-file fpath options)(mmap-file fpath)Memory map a file returning a native buffer. fpath must resolve to a valid java.io.File. Options * :resource-type - maps to tech.v3.resource
:track-type, defaults to :auto. * :mmap-mode * :read-only - default - map the data as shared read-only. * :read-write - map the data as shared read-write. * :private - map a private copy of the data and do not share.set-mmap-impl!
(set-mmap-impl! mmap-fn)Set the system mmap implementation to use. There are three provided under the mmap directory - *
+mmodel- Use the JDK-17 memory model. You have to have the module enabled. *larray- Use LArray mmap - default if available.If unset then the system will try to load the jdk-17 namespace, then larray.
tech.v3.datatype.native-buffer documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.native-buffer
Support for malloc/free and generalized support for reading/writing typed data to a long integer address of memory.
as-native-buffer
(as-native-buffer item)Convert a thing to a native buffer if possible. Calls tech.v3.datatype.protocols/->native-buffer if object indicates it is convertible to a native buffer.
free
(free data)Free a long ptr. Malloc will do this for you. Calling this is probably a mistake.
malloc
(malloc n-bytes {:keys [resource-type uninitialized? endianness log-level], :or {resource-type :gc}})(malloc n-bytes)Malloc memory. If a desired buffer type is needed follow up with set-native-datatype.
+tech.v3.datatype.native-buffer documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.native-buffer
Support for malloc/free and generalized support for reading/writing typed data to a long integer address of memory.
as-native-buffer
(as-native-buffer item)Convert a thing to a native buffer if possible. Calls tech.v3.datatype.protocols/->native-buffer if object indicates it is convertible to a native buffer.
free
(free data)Free a long ptr. Malloc will do this for you. Calling this is probably a mistake.
malloc
(malloc n-bytes {:keys [resource-type uninitialized? endianness log-level], :or {resource-type :gc}})(malloc n-bytes)Malloc memory. If a desired buffer type is needed follow up with set-native-datatype.
Options:
tech.v3.datatype.nippy documentation Generated by Codox with RDash UI theme
dtype-next 8.033
\ No newline at end of file +tech.v3.datatype.nippy
Nippy bindings for datatype base types and tensor types
tech.v3.datatype.nippy documentation Generated by Codox with RDash UI theme
dtype-next 8.034
\ No newline at end of file diff --git a/docs/tech.v3.datatype.packing.html b/docs/tech.v3.datatype.packing.html index 5caee00f..06213358 100644 --- a/docs/tech.v3.datatype.packing.html +++ b/docs/tech.v3.datatype.packing.html @@ -1,6 +1,6 @@ -tech.v3.datatype.nippy
Nippy bindings for datatype base types and tensor types
tech.v3.datatype.packing documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.packing
Implements the ‘packed/unpacked’ concept in datatype. This allows us to take objects like LocalDate’s and store them in int32 storage which dramatically decreases their size.
add-packed-datatype!
(add-packed-datatype! object-cls object-datatype packed-datatype primitive-datatype pack-fn unpack-fn)Add a datatype that you wish to be packed into a single scalar numeric value.
buffer-packing-pair
(buffer-packing-pair datatype)If datatype is a packed datatype, return a pair of functions, unpacking-read and packing-write. unpacking-read reads the appropriate datatype and calls the datatype’s unpack fn on it. packing-write packs the input and writes it to the appropriate place. Note that in this case those functions take the buffer and the index. returns {:unpacking-read :packing-write}
pack
(pack item)Pack a scalar, iterable, or a reader into a new scalar, iterable or a reader. If this isn’t a packable datatype this is a no-op.
pack-datatype
(pack-datatype datatype)Returns the packed datatype for this unpacked datatype.
pack-scalar
(pack-scalar value dtype)Pack a scalar value. Dtype is provided so nil can be packed.
+tech.v3.datatype.packing documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.packing
Implements the ‘packed/unpacked’ concept in datatype. This allows us to take objects like LocalDate’s and store them in int32 storage which dramatically decreases their size.
add-packed-datatype!
(add-packed-datatype! object-cls object-datatype packed-datatype primitive-datatype pack-fn unpack-fn)Add a datatype that you wish to be packed into a single scalar numeric value.
buffer-packing-pair
(buffer-packing-pair datatype)If datatype is a packed datatype, return a pair of functions, unpacking-read and packing-write. unpacking-read reads the appropriate datatype and calls the datatype’s unpack fn on it. packing-write packs the input and writes it to the appropriate place. Note that in this case those functions take the buffer and the index. returns {:unpacking-read :packing-write}
pack
(pack item)Pack a scalar, iterable, or a reader into a new scalar, iterable or a reader. If this isn’t a packable datatype this is a no-op.
pack-datatype
(pack-datatype datatype)Returns the packed datatype for this unpacked datatype.
pack-scalar
(pack-scalar value dtype)Pack a scalar value. Dtype is provided so nil can be packed.
Example:
tech.v3.datatype.packing> (pack-scalar nil :local-date) -2147483648 diff --git a/docs/tech.v3.datatype.reductions.html b/docs/tech.v3.datatype.reductions.html index 3bb09f04..a0700337 100644 --- a/docs/tech.v3.datatype.reductions.html +++ b/docs/tech.v3.datatype.reductions.html @@ -1,6 +1,6 @@ -tech.v3.datatype.reductions documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.reductions
High performance reductions based on tech.v3.datatype concepts as well as java stream concepts.
commutative-binary-double
(commutative-binary-double op options rdr)(commutative-binary-double op rdr)Perform a commutative reduction using a binary operator to perform the reduction. The operator needs to be both commutative and associative.
commutative-binary-long
(commutative-binary-long op rdr)Perform a commutative reduction in int64 space using a binary operator. The operator needs to be both commutative and associative.
commutative-binary-object
(commutative-binary-object op rdr)Perform a commutative reductions in object space using a binary operator. The operator needs to be both commutative and associative.
commutative-binary-reduce
(commutative-binary-reduce op rdr)Perform a commutative binary reduction. The space of the reduction will be determined by the datatype of the reader.
double-reductions
(double-reductions reducer-map options rdr)(double-reductions reducer-map rdr)Perform a group of reductions on a single double reader.
double-summation
(double-summation options rdr)(double-summation rdr)Double sum of data using Kahan compensated summation.
nan-strategy->double-predicate
(nan-strategy->double-predicate nan-strategy)(nan-strategy->double-predicate)Passing in either a keyword nan strategy #{:exception :keep :remove} or a DoublePredicate, return a DoublePredicate that filters double values.
ordered-group-by-reduce
(ordered-group-by-reduce reducer batch-data rdr)(ordered-group-by-reduce reducer rdr)Perform an ordered group-by operation using reader and placing results into the result-map. Expects that reducer’s batch-data method has already been called and Returns the non-finalized result-map. If result-map is nil then one is created. Each bucket’s results end up ordered by index iteration order. The original parallel pass goes through each index in order and then the reduction goes through the thread groups in order so if your index reduction merger just does (.addAll lhs rhs) then the final result ends up ordered.
+tech.v3.datatype.reductions documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.reductions
High performance reductions based on tech.v3.datatype concepts as well as java stream concepts.
commutative-binary-double
(commutative-binary-double op options rdr)(commutative-binary-double op rdr)Perform a commutative reduction using a binary operator to perform the reduction. The operator needs to be both commutative and associative.
commutative-binary-long
(commutative-binary-long op rdr)Perform a commutative reduction in int64 space using a binary operator. The operator needs to be both commutative and associative.
commutative-binary-object
(commutative-binary-object op rdr)Perform a commutative reductions in object space using a binary operator. The operator needs to be both commutative and associative.
commutative-binary-reduce
(commutative-binary-reduce op rdr)Perform a commutative binary reduction. The space of the reduction will be determined by the datatype of the reader.
double-reductions
(double-reductions reducer-map options rdr)(double-reductions reducer-map rdr)Perform a group of reductions on a single double reader.
double-summation
(double-summation options rdr)(double-summation rdr)Double sum of data using Kahan compensated summation.
nan-strategy->double-predicate
(nan-strategy->double-predicate nan-strategy)(nan-strategy->double-predicate)Passing in either a keyword nan strategy #{:exception :keep :remove} or a DoublePredicate, return a DoublePredicate that filters double values.
ordered-group-by-reduce
(ordered-group-by-reduce reducer batch-data rdr)(ordered-group-by-reduce reducer rdr)Perform an ordered group-by operation using reader and placing results into the result-map. Expects that reducer’s batch-data method has already been called and Returns the non-finalized result-map. If result-map is nil then one is created. Each bucket’s results end up ordered by index iteration order. The original parallel pass goes through each index in order and then the reduction goes through the thread groups in order so if your index reduction merger just does (.addAll lhs rhs) then the final result ends up ordered.
Example:
user> (import '[tech.v3.datatype IndexReduction]) tech.v3.datatype.IndexReduction diff --git a/docs/tech.v3.datatype.rolling.html b/docs/tech.v3.datatype.rolling.html index 66c7c3b0..c8d5fbe5 100644 --- a/docs/tech.v3.datatype.rolling.html +++ b/docs/tech.v3.datatype.rolling.html @@ -1,6 +1,6 @@ -tech.v3.datatype.rolling documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.rolling
expanding-window-ranges
(expanding-window-ranges n-elems)Return a reader of expanding window ranges used for cumsum type operations.
fixed-rolling-window
(fixed-rolling-window item window-size window-fn {:keys [relative-window-position edge-mode datatype], :or {relative-window-position :center, edge-mode :clamp}, :as options})(fixed-rolling-window item window-size window-fn)Return a lazily evaluated rolling window of window-fn applied to each window. The iterable or sequence is padded such that there are the same number of values in the result as in the input with repeated elements padding the beginning and end of the original sequence. If input is an iterator, output is an lazy sequence. If input is a reader, output is a reader.
+tech.v3.datatype.rolling documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.rolling
expanding-window-ranges
(expanding-window-ranges n-elems)Return a reader of expanding window ranges used for cumsum type operations.
fixed-rolling-window
(fixed-rolling-window item window-size window-fn {:keys [relative-window-position edge-mode datatype], :or {relative-window-position :center, edge-mode :clamp}, :as options})(fixed-rolling-window item window-size window-fn)Return a lazily evaluated rolling window of window-fn applied to each window. The iterable or sequence is padded such that there are the same number of values in the result as in the input with repeated elements padding the beginning and end of the original sequence. If input is an iterator, output is an lazy sequence. If input is a reader, output is a reader.
:Options
tech.v3.datatype.sampling documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.sampling
Implementation of reservoir sampling designed to be used in other systems. Provides a low-level sampler object and a double-reservoir that implements DoubleConsumer and derefs to a buffer of doubles.
->random
(->random {:keys [random seed algorithm], :or {algorithm :lcg}})(->random)Given an options map return an implementation of java.util.Random.
+tech.v3.datatype.sampling documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.sampling
Implementation of reservoir sampling designed to be used in other systems. Provides a low-level sampler object and a double-reservoir that implements DoubleConsumer and derefs to a buffer of doubles.
->random
(->random {:keys [random seed algorithm], :or {algorithm :lcg}})(->random)Given an options map return an implementation of java.util.Random.
Options:
tech.v3.datatype.struct documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.struct
Structs are datatypes composed of primitive datatypes or other structs. Similar to records except they do not support string or object columns, only numeric values. They have memset-0 initialization, memcpy copy semantics. For correct equals, hashing, convert struct into a normal persistent map via
+into.tech.v3.datatype.struct documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.struct
Structs are datatypes composed of primitive datatypes or other structs. Similar to records except they do not support string or object columns, only numeric values. They have memset-0 initialization, memcpy copy semantics. For correct equals, hashing, convert struct into a normal persistent map via
into.Example:
user> (require '[tech.v3.datatype :as dtype]) nil diff --git a/docs/tech.v3.datatype.wavelet.html b/docs/tech.v3.datatype.wavelet.html index e9921132..3f49e9d5 100644 --- a/docs/tech.v3.datatype.wavelet.html +++ b/docs/tech.v3.datatype.wavelet.html @@ -1,3 +1,3 @@ -tech.v3.datatype.wavelet documentation Generated by Codox with RDash UI theme
dtype-next 8.033
\ No newline at end of file +tech.v3.datatype.wavelet
Implementation of the scipy continuous wavelet transform. See fastmath for implementations of the discrete wavelet transform.
cwt
(cwt data wavelet-fn widths options)(cwt data wavelet-fn widths)Minimal version of scipy’s cwt. Only real datatypes are supported.
tech.v3.datatype.wavelet documentation Generated by Codox with RDash UI theme
dtype-next 8.034
\ No newline at end of file diff --git a/docs/tech.v3.libs.buffered-image.html b/docs/tech.v3.libs.buffered-image.html index 7906d450..3f11a2de 100644 --- a/docs/tech.v3.libs.buffered-image.html +++ b/docs/tech.v3.libs.buffered-image.html @@ -1,3 +1,3 @@ -tech.v3.datatype.wavelet
Implementation of the scipy continuous wavelet transform. See fastmath for implementations of the discrete wavelet transform.
cwt
(cwt data wavelet-fn widths options)(cwt data wavelet-fn widths)Minimal version of scipy’s cwt. Only real datatypes are supported.
tech.v3.libs.buffered-image documentation Generated by Codox with RDash UI theme
dtype-next 8.033
\ No newline at end of file +tech.v3.libs.buffered-image
Bindings to buffered images for the datatype system
as-ubyte-tensor
(as-ubyte-tensor img & _options)Get the buffered image as a uint8 tensor. Works for byte and integer-buffer backed images.
buffered-image->data-buffer
(buffered-image->data-buffer img)Given a buffered image, return it’s data buffer.
downsample-bilinear
(downsample-bilinear src-img & {:keys [dst-img-width dst-img-height dst-img-type]})Reduce an image size using bilinear filtering.
draw-image!
(draw-image! src-img dst-image & {:keys [src-x-offset src-y-offset src-rect-width src-rect-height dst-x-offset dst-y-offset dst-rect-width dst-rect-height interpolation-type]})Draw a source image onto a destination image. This can be used for scaling, cropping, or copying images.
image-channel-format
(image-channel-format img)Get the image channel format of the buffered image. Formats returned may be: :gray :bgr :rgb :abgr :argb :abgr-pre :argb-pre
image-channel-map
(image-channel-map img)Get a map from keyword channel name to channel index in a ubyte tensor
new-image
(new-image height width img-type)Create a new buffered image. img-type is a keyword and must be one of the keys in the image-types map. ** Arguments are reverse of the buffered image constructor **
resize
(resize src-img new-width new-height {:keys [resize-algorithm dst-img-type]})Resize an image. Options - * resize-algorithm - One of #{:bilinear :cubic :nearest}. Defaults to - if the new width is larger than then old width, bilinear is chosen else nearest is chosen. * dst-img-type - Defaults to the src image type. Should be one of the keys of image-types.
save!
(save! img format-str fname-or-stream)(save! img fname-str)Save an image. Format-str can be things like “PNG” or “JPEG”. There are better versions of this in tech.io.
tech.v3.libs.buffered-image documentation Generated by Codox with RDash UI theme
dtype-next 8.034
\ No newline at end of file diff --git a/docs/tech.v3.libs.neanderthal.html b/docs/tech.v3.libs.neanderthal.html index 03089cd2..0a498f1e 100644 --- a/docs/tech.v3.libs.neanderthal.html +++ b/docs/tech.v3.libs.neanderthal.html @@ -1,3 +1,3 @@ -tech.v3.libs.buffered-image
Bindings to buffered images for the datatype system
as-ubyte-tensor
(as-ubyte-tensor img & _options)Get the buffered image as a uint8 tensor. Works for byte and integer-buffer backed images.
buffered-image->data-buffer
(buffered-image->data-buffer img)Given a buffered image, return it’s data buffer.
downsample-bilinear
(downsample-bilinear src-img & {:keys [dst-img-width dst-img-height dst-img-type]})Reduce an image size using bilinear filtering.
draw-image!
(draw-image! src-img dst-image & {:keys [src-x-offset src-y-offset src-rect-width src-rect-height dst-x-offset dst-y-offset dst-rect-width dst-rect-height interpolation-type]})Draw a source image onto a destination image. This can be used for scaling, cropping, or copying images.
image-channel-format
(image-channel-format img)Get the image channel format of the buffered image. Formats returned may be: :gray :bgr :rgb :abgr :argb :abgr-pre :argb-pre
image-channel-map
(image-channel-map img)Get a map from keyword channel name to channel index in a ubyte tensor
new-image
(new-image height width img-type)Create a new buffered image. img-type is a keyword and must be one of the keys in the image-types map. ** Arguments are reverse of the buffered image constructor **
resize
(resize src-img new-width new-height {:keys [resize-algorithm dst-img-type]})Resize an image. Options - * resize-algorithm - One of #{:bilinear :cubic :nearest}. Defaults to - if the new width is larger than then old width, bilinear is chosen else nearest is chosen. * dst-img-type - Defaults to the src image type. Should be one of the keys of image-types.
save!
(save! img format-str fname-or-stream)(save! img fname-str)Save an image. Format-str can be things like “PNG” or “JPEG”. There are better versions of this in tech.io.
tech.v3.libs.neanderthal documentation Generated by Codox with RDash UI theme
dtype-next 8.033
\ No newline at end of file +tech.v3.libs.neanderthal
Implementation of the various datatype protocols for neanderthal datatypes. Users must require this to enable ->reader and as-tensor functionality for neanderthal datatypes.
tech.v3.libs.neanderthal documentation Generated by Codox with RDash UI theme
dtype-next 8.034
\ No newline at end of file diff --git a/docs/tech.v3.parallel.for.html b/docs/tech.v3.parallel.for.html index f6c89ea7..9ca15838 100644 --- a/docs/tech.v3.parallel.for.html +++ b/docs/tech.v3.parallel.for.html @@ -1,6 +1,6 @@ -tech.v3.libs.neanderthal
Implementation of the various datatype protocols for neanderthal datatypes. Users must require this to enable ->reader and as-tensor functionality for neanderthal datatypes.
tech.v3.parallel.for documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.parallel.for
Serial and parallel iteration strategies across iterators and index spaces.
->consumer
(->consumer consumer)Convert a generic object into a consumer. Works for any java.util.consumer and a Clojure IFn. Returns an implementation of java.util.Consumer
batch-iter
(batch-iter batch-size item)Given an iterator, batch it up into an implementation of
java.util.Listthat contains batches of the data. Return a sequence of batches.common-pool-parallelism
(common-pool-parallelism)Integer number of threads used in the ForkJoinPool’s common pool
consume!
(consume! consumer item)Consume (terminate) a sequence or stream. If the stream is parallel then the consumer had better be threadsafe. Returns the consumer.
convertible-to-iterator?
(convertible-to-iterator? item)True when this is an iterable or a stream.
cpu-pool-map-reduce
(cpu-pool-map-reduce map-fn reduce-fn fork-join-pool)Execute map-fn in the separate threads of ForkJoinPool’s common pool. Map-fn takes a single long which is it’s task index.
doiter
macro
(doiter varname iterable & body)Execute body for every item in the iterable. Expecting side effects, returns nil.
indexed-consume!
(indexed-consume! consumer item)Consume (terminate) an iterable. Consumer in this case is expected to be an clojure IFn and it will receive two arguments, a long and the value.
indexed-map-reduce
(indexed-map-reduce num-iters indexed-map-fn reduce-fn options)(indexed-map-reduce num-iters indexed-map-fn reduce-fn)(indexed-map-reduce num-iters indexed-map-fn)Execute
+indexed-map-fnovern-groupssubranges of(range num-iters). Then callreduce-fnpassing in entire in order result value sequence.tech.v3.parallel.for documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.parallel.for
Serial and parallel iteration strategies across iterators and index spaces.
->consumer
(->consumer consumer)Convert a generic object into a consumer. Works for any java.util.consumer and a Clojure IFn. Returns an implementation of java.util.Consumer
batch-iter
(batch-iter batch-size item)Given an iterator, batch it up into an implementation of
java.util.Listthat contains batches of the data. Return a sequence of batches.common-pool-parallelism
(common-pool-parallelism)Integer number of threads used in the ForkJoinPool’s common pool
consume!
(consume! consumer item)Consume (terminate) a sequence or stream. If the stream is parallel then the consumer had better be threadsafe. Returns the consumer.
convertible-to-iterator?
(convertible-to-iterator? item)True when this is an iterable or a stream.
cpu-pool-map-reduce
(cpu-pool-map-reduce map-fn reduce-fn fork-join-pool)Execute map-fn in the separate threads of ForkJoinPool’s common pool. Map-fn takes a single long which is it’s task index.
doiter
macro
(doiter varname iterable & body)Execute body for every item in the iterable. Expecting side effects, returns nil.
indexed-consume!
(indexed-consume! consumer item)Consume (terminate) an iterable. Consumer in this case is expected to be an clojure IFn and it will receive two arguments, a long and the value.
indexed-map-reduce
(indexed-map-reduce num-iters indexed-map-fn reduce-fn options)(indexed-map-reduce num-iters indexed-map-fn reduce-fn)(indexed-map-reduce num-iters indexed-map-fn)Execute
indexed-map-fnovern-groupssubranges of(range num-iters). Then callreduce-fnpassing in entire in order result value sequence.tech.v3.tensor.color-gradients documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.tensor.color-gradients
Implement a mapping from double->color for each entry in a tensor. Produces an image of the same dimensions in pixels as the input tensor.
+tech.v3.tensor.color-gradients documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.tensor.color-gradients
Implement a mapping from double->color for each entry in a tensor. Produces an image of the same dimensions in pixels as the input tensor.
Default color schemes are found here: https://reference.wolfram.com/language/guide/ColorSchemes.html
colorize
(colorize src-tens gradient-name & {:keys [data-min data-max alpha? check-invalid? invert-gradient? gradient-default-n], :or {gradient-default-n 200}})Apply a color gradient to a tensor returning an image. Takes A 1 or 2d tensor. If data-min, data-max aren’t provided they are found in the data. A buffered image is returned.
src-tens - Source tensor whose shape determines the shape of the final image.
gradient-name - may be a keyword, in which it must be a key in @gradient-map and these gradients come from: https://reference.wolfram.com/language/guide/ColorSchemes.html. gradient-name may be a tensor of dimensions [n 3]. gradient-name may be a function that takes a value from 0-1 and returns a tuple of length 3.
diff --git a/docs/tech.v3.tensor.dimensions.html b/docs/tech.v3.tensor.dimensions.html index 3b887441..8954cbee 100644 --- a/docs/tech.v3.tensor.dimensions.html +++ b/docs/tech.v3.tensor.dimensions.html @@ -1,6 +1,6 @@ -tech.v3.tensor.dimensions documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.tensor.dimensions
Dimensions implement a projection from ND space in the the address space of the buffer along with a reverse projection from the address space back into ND space.
->2d-shape
(->2d-shape {:keys [shape]})Given dimensions, return new dimensions with the lowest (fastest-changing) dimension unchanged and the rest of the dimensions multiplied into the higher dimension.
->batch-shape
(->batch-shape {:keys [shape]})Given dimensions, return new dimensions with the lowest (fastest-changing) dimension unchanged and the rest of the dimensions multiplied into the higher dimension.
->global->local
(->global->local dims)Get the ND->N addressing operator for these dimensions.
->least-rapidly-changing-dimension
(->least-rapidly-changing-dimension {:keys [shape-ecounts]})Get the size of the least rapidly changing dimension
->local->global
(->local->global dims)Get an inverse operator for these dims. Inverting the global->local dimensions means that for some dimensions (like broadcasted dimensions) each local index may correspond to a set of global indexes.
->most-rapidly-changing-dimension
(->most-rapidly-changing-dimension {:keys [shape-ecounts]})Get the size of the most rapidly changing dimension
access-increasing?
(access-increasing? {:keys [access-increasing?]})Are these dimensions setup such a naive seq through the data will be accessing memory in order. This is necessary for external library interfaces (blas, cudnn). An example would be after any nontrivial transpose that is not made concrete (copied) this condition will not hold.
broadcast
(broadcast {:keys [shape shape-ecounts strides], :as dims} new-shape)Broadcast one or more dimensions. End result is shape of target matches new-shape
buffer-ecount
(buffer-ecount {:keys [buffer-ecount]})What is the necessary ecount for a given buffer. Maybe nil if this could not be detected from the dimension arguments.
dense?
(dense? {:keys [native?]})Do the strides indicate a packed data buffer with no holes. Currently mapped to native?
dimensions
(dimensions shape strides shape-ecounts shape-ecount-strides)(dimensions shape strides)(dimensions shape)Dimensions contain information about how to map logical global indexes to local buffer addresses.
direct?
(direct? {:keys [shape-direct?]})Is this dimension object a direct dimensions meaning the shape has no indirect (indexed) access.
get-elem-dims-local->global
(get-elem-dims-local->global dims global->local*)Harder translation than above. May return nil in the case where the inverse operation hasn’t yet been derived. In this case, the best you can do is a O(N) iteration similar to dense math.
in-place-reshape
(in-place-reshape existing-dims shape)Return new dimensions that correspond to an in-place reshape. This is a very difficult algorithm to get correct as it needs to take into account changing strides and dense vs non-dense dimensions.
indirect?
(indirect? dims)Are these dimensions indirect meaning one of the shape entries is an index buffer (or some other index abstraction like a range with an increment other than 1)?
local-address->local-shape
(local-address->local-shape shape offsets strides shape-mins addr)Shape and strides are not transposed. Returns [valid? local-shape-as-list]
matrix-column-stride
(matrix-column-stride {:keys [shape strides]})Returns the larger of the 2 strides
native?
(native? {:keys [native?]})Does this dimension object describe packed, in-order access of the underlying data buffer?
rotate
(rotate {:keys [shape], :as dims} new-offset-vec)Dimensional rotations are applied via offsetting.
select
(select dims & args)Expanded implementation of the core.matrix select function call. Each dimension must have an entry and each entry may be: :all (identity) :lla (reverse) persistent-vector: [0 1 2 3 4 4 5] (not supported by all backends) map: {:type [:+ :-] :min-item 0 :max-item 50} Monotonically increasing/decreasing bounded (inclusive) sequences
+tech.v3.tensor.dimensions documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.tensor.dimensions
Dimensions implement a projection from ND space in the the address space of the buffer along with a reverse projection from the address space back into ND space.
->2d-shape
(->2d-shape {:keys [shape]})Given dimensions, return new dimensions with the lowest (fastest-changing) dimension unchanged and the rest of the dimensions multiplied into the higher dimension.
->batch-shape
(->batch-shape {:keys [shape]})Given dimensions, return new dimensions with the lowest (fastest-changing) dimension unchanged and the rest of the dimensions multiplied into the higher dimension.
->global->local
(->global->local dims)Get the ND->N addressing operator for these dimensions.
->least-rapidly-changing-dimension
(->least-rapidly-changing-dimension {:keys [shape-ecounts]})Get the size of the least rapidly changing dimension
->local->global
(->local->global dims)Get an inverse operator for these dims. Inverting the global->local dimensions means that for some dimensions (like broadcasted dimensions) each local index may correspond to a set of global indexes.
->most-rapidly-changing-dimension
(->most-rapidly-changing-dimension {:keys [shape-ecounts]})Get the size of the most rapidly changing dimension
access-increasing?
(access-increasing? {:keys [access-increasing?]})Are these dimensions setup such a naive seq through the data will be accessing memory in order. This is necessary for external library interfaces (blas, cudnn). An example would be after any nontrivial transpose that is not made concrete (copied) this condition will not hold.
broadcast
(broadcast {:keys [shape shape-ecounts strides], :as dims} new-shape)Broadcast one or more dimensions. End result is shape of target matches new-shape
buffer-ecount
(buffer-ecount {:keys [buffer-ecount]})What is the necessary ecount for a given buffer. Maybe nil if this could not be detected from the dimension arguments.
dense?
(dense? {:keys [native?]})Do the strides indicate a packed data buffer with no holes. Currently mapped to native?
dimensions
(dimensions shape strides shape-ecounts shape-ecount-strides)(dimensions shape strides)(dimensions shape)Dimensions contain information about how to map logical global indexes to local buffer addresses.
direct?
(direct? {:keys [shape-direct?]})Is this dimension object a direct dimensions meaning the shape has no indirect (indexed) access.
get-elem-dims-local->global
(get-elem-dims-local->global dims global->local*)Harder translation than above. May return nil in the case where the inverse operation hasn’t yet been derived. In this case, the best you can do is a O(N) iteration similar to dense math.
in-place-reshape
(in-place-reshape existing-dims shape)Return new dimensions that correspond to an in-place reshape. This is a very difficult algorithm to get correct as it needs to take into account changing strides and dense vs non-dense dimensions.
indirect?
(indirect? dims)Are these dimensions indirect meaning one of the shape entries is an index buffer (or some other index abstraction like a range with an increment other than 1)?
local-address->local-shape
(local-address->local-shape shape offsets strides shape-mins addr)Shape and strides are not transposed. Returns [valid? local-shape-as-list]
matrix-column-stride
(matrix-column-stride {:keys [shape strides]})Returns the larger of the 2 strides
native?
(native? {:keys [native?]})Does this dimension object describe packed, in-order access of the underlying data buffer?
rotate
(rotate {:keys [shape], :as dims} new-offset-vec)Dimensional rotations are applied via offsetting.
select
(select dims & args)Expanded implementation of the core.matrix select function call. Each dimension must have an entry and each entry may be: :all (identity) :lla (reverse) persistent-vector: [0 1 2 3 4 4 5] (not supported by all backends) map: {:type [:+ :-] :min-item 0 :max-item 50} Monotonically increasing/decreasing bounded (inclusive) sequences
tensor : int32, dense vector only. Not supported by all backends.
;;Some examples https://cloojure.github.io/doc/core.matrix/clojure.core.matrix.html#var-select
slice
(slice {:keys [shape strides], :as original-dims} n-elems)Slice off the leftmost n-elems dimensions. Return a sub-dim object that doesn’t change and a long reader of offsets. Returns {:dimension - dimensions for every sub object :offsets - long reader of offsets for the buffer. }
slice-right
(slice-right {:keys [shape strides], :as original-dims} n-elems)Slice off the rightmost n-elems dimensions. Return a sub-dim object that doesn’t change and a long reader of offsets. Returns {:dimension - dimensions for every sub object :offsets - long reader of offsets for the buffer. }
transpose
(transpose {:keys [shape strides shape-ecounts], :as dims} reorder-vec)Transpose the dimensions. Returns a new dimensions that will access memory in a transposed order. Dimension 0 is the leftmost (greatest) dimension:
(transpose tens (range (count (shape tens))))
diff --git a/docs/tech.v3.tensor.html b/docs/tech.v3.tensor.html index b3134cf1..e98400fc 100644 --- a/docs/tech.v3.tensor.html +++ b/docs/tech.v3.tensor.html @@ -1,6 +1,6 @@ -tech.v3.tensor documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.tensor
ND bindings for the tech.v3.datatype system. A Tensor is conceptually just a tuple of a buffer and an index operator that is capable of converting indexes in ND space into a single long index into the buffer. Tensors implement the tech.v3.datatype.NDBuffer interface and outside this file ND objects are expected to simply implement that interface.
+tech.v3.tensor documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.tensor
ND bindings for the tech.v3.datatype system. A Tensor is conceptually just a tuple of a buffer and an index operator that is capable of converting indexes in ND space into a single long index into the buffer. Tensors implement the tech.v3.datatype.NDBuffer interface and outside this file ND objects are expected to simply implement that interface.
This system relies heavily on the tech.v3.tensor.dimensions namespace to provide the optimized indexing operator from ND space to buffer space and back.
There is an ABI in the form of nd-buffer-descriptors that is a map containing:
-
diff --git a/project.clj b/project.clj
index 5f3d9c73..b5cd719b 100644
--- a/project.clj
+++ b/project.clj
@@ -1,4 +1,4 @@
-(defproject cnuernber/dtype-next "8.034-SNAPSHOT"
+(defproject cnuernber/dtype-next "8.034"
:description "A Clojure library designed to aid in the implementation of high performance algorithms and systems."
:url "http://github.com/cnuernber/dtype-next"
:license {:name "EPL-2.0"
Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.bitmap
Functions for working with RoaringBitmaps. These are integrated deeply into several tech.v3.datatype algorithms and have many potential applications in high performance computing applications as they are both extremely fast and storage-space efficient.
->bitmap
(->bitmap item)(->bitmap)Create a roaring bitmap. If this object has a conversion to a roaring bitmap use that, else copy the data into a new roaring bitmap.
->unique-bitmap
(->unique-bitmap item)(->unique-bitmap)Perform a conversion to a bitmap. If this thing is already a bitmap, clone it.
bitmap->efficient-random-access-reader
(bitmap->efficient-random-access-reader bitmap)Bitmaps do not implement efficient random access although we do provide access of inefficient random access for them. This converts a bitmap into a flat buffer of data that does support efficient random access.
bitmap-as-primitive-list
(bitmap-as-primitive-list bitmap)(bitmap-as-primitive-list)Return a bitmap as an implementation of a primitive list. This allows code dependent upon primitive lists to be write to bitmaps.
bitmap-value->bitmap-map
(bitmap-value->bitmap-map bitmap value)Given a bitmap and a value return an efficient implementation of clojure.lang.IPersistentMap that has the given value at the given indexes.
Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.bitmap
Functions for working with RoaringBitmaps. These are integrated deeply into several tech.v3.datatype algorithms and have many potential applications in high performance computing applications as they are both extremely fast and storage-space efficient.
->bitmap
(->bitmap item)(->bitmap)Create a roaring bitmap. If this object has a conversion to a roaring bitmap use that, else copy the data into a new roaring bitmap.
->unique-bitmap
(->unique-bitmap item)(->unique-bitmap)Perform a conversion to a bitmap. If this thing is already a bitmap, clone it.
bitmap->efficient-random-access-reader
(bitmap->efficient-random-access-reader bitmap)Bitmaps do not implement efficient random access although we do provide access of inefficient random access for them. This converts a bitmap into a flat buffer of data that does support efficient random access.
bitmap-as-primitive-list
(bitmap-as-primitive-list bitmap)(bitmap-as-primitive-list)Return a bitmap as an implementation of a primitive list. This allows code dependent upon primitive lists to be write to bitmaps.
bitmap-value->bitmap-map
(bitmap-value->bitmap-map bitmap value)Given a bitmap and a value return an efficient implementation of clojure.lang.IPersistentMap that has the given value at the given indexes.
Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.convolve
Namespace for implementing various basic convolutions. Currently only 1d convolutions are supported.
convolve1d
(convolve1d data win options)(convolve1d data win)Convolve a window across a signal. The only difference from correlate is the window is reversed and then correlate is called. See options for correlate, this has the same defaults.
+Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.convolve
Namespace for implementing various basic convolutions. Currently only 1d convolutions are supported.
convolve1d
(convolve1d data win options)(convolve1d data win)Convolve a window across a signal. The only difference from correlate is the window is reversed and then correlate is called. See options for correlate, this has the same defaults.
Examples:
user> (require '[tech.v3.datatype.convolve :as dt-conv])
nil
diff --git a/docs/tech.v3.datatype.datetime.html b/docs/tech.v3.datatype.datetime.html
index 06aecbf4..4ec17a42 100644
--- a/docs/tech.v3.datatype.datetime.html
+++ b/docs/tech.v3.datatype.datetime.html
@@ -1,6 +1,6 @@
-tech.v3.datatype.datetime documentation Generated by Codox with RDash UI theme
dtype-next 8.033
tech.v3.datatype.datetime
Thorough bindings to java.time.datetime. Includes packed datatypes and lifting various datetime datatypes into the datatype system.
+tech.v3.datatype.datetime documentation Generated by Codox with RDash UI theme
dtype-next 8.034
tech.v3.datatype.datetime
Thorough bindings to java.time.datetime. Includes packed datatypes and lifting various datetime datatypes into the datatype system.
A general outline is: