Skip to content

Commit

Permalink
Allow to Set Separate WAL Dirs
Browse files Browse the repository at this point in the history
  • Loading branch information
@alexanderkiel
alexanderkiel committed Sep 14, 2022
1 parent 4614051 commit d33d340
Show file tree
Hide file tree
Showing 5 changed files with 30 additions and 17 deletions.
4 changes: 4 additions & 0 deletions docs/deployment/environment-variables.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,11 @@ The three database directories must not exist on the first start of Blaze and wi
| Name | Default | Since | Depr ¹ | Description |
|:-------------------------------|:--------------|:------|:-------|:-------------------------------------------------------------------------------------------------------------------------------------------------|
| INDEX_DB_DIR | index ² | v0.8 | | The directory were the index database files are stored. |
| INDEX_DB_WAL_DIR | \<empty\> | v0.18 | | The directory were the index database write ahead log (WAL) files are stored. Empty means same dir as database files. |
| TRANSACTION_DB_DIR | transaction ² | v0.8 | | The directory were the transaction log files are stored. This directory must not exist on the first start of Blaze and will be created by Blaze. |
| TRANSACTION_DB_WAL_DIR | \<empty\> | v0.18 | | The directory were the transaction log write ahead log (WAL) files are stored. Empty means same dir as database files. |
| RESOURCE_DB_DIR | resource ² | v0.8 | | The directory were the resource files are stored. This directory must not exist on the first start of Blaze and will be created by |
| RESOURCE_DB_WAL_DIR | \<empty\> | v0.18 | | The directory were the resource write ahead log (WAL) files are stored. Empty means same dir as database files. |
| DB_BLOCK_CACHE_SIZE | 128 | v0.8 | | The size of the [block cache][2] of the DB in MB. |
| DB_RESOURCE_CACHE_SIZE | 100000 | v0.8 | | The size of the resource cache of the DB in number of resources. |
| DB_MAX_BACKGROUND_JOBS | 4 | v0.8 | | The maximum number of the [background jobs][3] used for DB compactions. |
Expand All @@ -38,6 +41,7 @@ The distributed storage variant only uses the index database locally.
| Name | Default | Since | Depr ¹ | Description |
|:-----------------------------------|:---------------|:------|:-------|:-----------------------------------------------------------------------------------------------------------------------------------------------------|
| INDEX_DB_DIR | index ² | v0.8 | | The directory were the index database files are stored. |
| INDEX_DB_WAL_DIR | \<empty\> | v0.18 | | The directory were the index database write ahead log (WAL) files are stored. Empty means same dir as database files. |
| DB_BLOCK_CACHE_SIZE | 128 | v0.8 | | The size of the [block cache][2] of the DB in MB. |
| DB_RESOURCE_CACHE_SIZE | 100000 | v0.8 | | The size of the resource cache of the DB in number of resources. |
| DB_MAX_BACKGROUND_JOBS | 4 | v0.8 | | The maximum number of the [background jobs][3] used for DB compactions. |
Expand Down
7 changes: 5 additions & 2 deletions modules/rocksdb/src/blaze/db/kv/rocksdb/impl.clj
View file
Original file line number Diff line number Diff line change
Expand Up @@ -78,12 +78,15 @@
(defn db-options
^DBOptions
[stats
{:keys [max-background-jobs
{:keys [wal-dir
max-background-jobs
compaction-readahead-size]
:or {max-background-jobs 2
:or {wal-dir ""
max-background-jobs 2
compaction-readahead-size 0}}]
(doto (DBOptions.)
(.setStatistics ^Statistics stats)
(.setWalDir (str wal-dir))
(.setMaxBackgroundJobs (long max-background-jobs))
(.setCompactionReadaheadSize (long compaction-readahead-size))
(.setEnablePipelinedWrite true)
Expand Down
7 changes: 6 additions & 1 deletion modules/rocksdb/src/blaze/db/kv/rocksdb/spec.clj
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@
#(instance? Statistics %))


(s/def ::db-options/wal-dir
string?)


(s/def ::db-options/max-background-jobs
nat-int?)

Expand All @@ -32,7 +36,8 @@


(s/def :blaze.db.kv.rocksdb/db-options
(s/keys :opt-un [::db-options/max-background-jobs
(s/keys :opt-un [::db-options/wal-dir
::db-options/max-background-jobs
::db-options/compaction-readahead-size]))


Expand Down
5 changes: 4 additions & 1 deletion modules/rocksdb/test/blaze/db/kv/rocksdb/impl_test.clj
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,8 @@

DBOptions
(datafy [options]
{:max-background-jobs (.maxBackgroundJobs options)
{:wal-dir (.walDir options)
:max-background-jobs (.maxBackgroundJobs options)
:compaction-readahead-size (.compactionReadaheadSize options)
:enable-pipelined-write (.enablePipelinedWrite options)
:create-if-missing (.createIfMissing options)
Expand Down Expand Up @@ -144,6 +145,7 @@
(deftest db-options-test
(testing "with defaults"
(given (datafy/datafy (impl/db-options (Statistics.) nil))
:wal-dir := ""
:max-background-jobs := 2
:compaction-readahead-size := 0
:enable-pipelined-write := true
Expand All @@ -154,6 +156,7 @@
(given (datafy/datafy (impl/db-options (Statistics.) {key value}))
key := value)

:wal-dir "wal"
:max-background-jobs 4
:compaction-readahead-size 10)))

Expand Down
24 changes: 11 additions & 13 deletions resources/blaze.edn
View file
Original file line number Diff line number Diff line change
Expand Up @@ -351,7 +351,8 @@
:block-cache #blaze/ref :blaze.db.kv.rocksdb/block-cache
:stats #blaze/ref :blaze.db.index-kv-store/stats
:opts
{:max-background-jobs
{:wal-dir #blaze/cfg ["INDEX_DB_WAL_DIR" string? ""]
:max-background-jobs
#blaze/cfg ["DB_MAX_BACKGROUND_JOBS" int? 4]
:compaction-readahead-size
#blaze/cfg ["DB_COMPACTION_READAHEAD_SIZE" int? 0]}
Expand Down Expand Up @@ -457,10 +458,9 @@
:block-cache #blaze/ref :blaze.db.kv.rocksdb/block-cache
:stats #blaze/ref :blaze.db.transaction-kv-store/stats
:opts
{:max-background-jobs
#blaze/cfg ["DB_MAX_BACKGROUND_JOBS" int? 4]
:compaction-readahead-size
#blaze/cfg ["DB_COMPACTION_READAHEAD_SIZE" int? 0]}
{:wal-dir #blaze/cfg ["TRANSACTION_DB_WAL_DIR" string? ""]
:max-background-jobs #blaze/cfg ["DB_MAX_BACKGROUND_JOBS" int? 4]
:compaction-readahead-size #blaze/cfg ["DB_COMPACTION_READAHEAD_SIZE" int? 0]}
:column-families
{:default
{:write-buffer-size-in-mb 8
Expand Down Expand Up @@ -493,10 +493,9 @@
:block-cache #blaze/ref :blaze.db.kv.rocksdb/block-cache
:stats #blaze/ref :blaze.db.resource-kv-store/stats
:opts
{:max-background-jobs
#blaze/cfg ["DB_MAX_BACKGROUND_JOBS" int? 4]
:compaction-readahead-size
#blaze/cfg ["DB_COMPACTION_READAHEAD_SIZE" int? 0]}
{:wal-dir #blaze/cfg ["RESOURCE_DB_WAL_DIR" string? ""]
:max-background-jobs #blaze/cfg ["DB_MAX_BACKGROUND_JOBS" int? 4]
:compaction-readahead-size #blaze/cfg ["DB_COMPACTION_READAHEAD_SIZE" int? 0]}
:column-families
{:default
{:write-buffer-size-in-mb 64
Expand Down Expand Up @@ -545,10 +544,9 @@
:block-cache #blaze/ref :blaze.db.kv.rocksdb/block-cache
:stats #blaze/ref :blaze.db.index-kv-store/stats
:opts
{:max-background-jobs
#blaze/cfg ["DB_MAX_BACKGROUND_JOBS" int? 4]
:compaction-readahead-size
#blaze/cfg ["DB_COMPACTION_READAHEAD_SIZE" int? 0]}
{:wal-dir #blaze/cfg ["INDEX_DB_WAL_DIR" string? ""]
:max-background-jobs #blaze/cfg ["DB_MAX_BACKGROUND_JOBS" int? 4]
:compaction-readahead-size #blaze/cfg ["DB_COMPACTION_READAHEAD_SIZE" int? 0]}
:column-families
{:search-param-value-index
{:write-buffer-size-in-mb 64
Expand Down

0 comments on commit d33d340

Please sign in to comment.