Skip to content

Commit

Permalink
Tidy up the RocksDB File Sizes
Browse files Browse the repository at this point in the history
For no good reason we had a 26 MiB file size in the resource store.
Also some smaller Column Families had a target file size of 3 MiB which
is smaller than the write buffer size. I did run an import with 100 k
Synthea patients and the performance was over 9000 patients per second.
  • Loading branch information
alexanderkiel committed Jul 16, 2024
1 parent 457ae53 commit 4c7d31e
Showing 1 changed file with 15 additions and 15 deletions.
30 changes: 15 additions & 15 deletions resources/blaze.edn
Original file line number Diff line number Diff line change
Expand Up @@ -567,19 +567,19 @@
:column-families
{:search-param-value-index
{:write-buffer-size-in-mb 64
:max-bytes-for-level-base-in-mb 512
:max-bytes-for-level-base-in-mb 256
:target-file-size-base-in-mb 64
:block-size #blaze/cfg ["DB_BLOCK_SIZE" int? 16384]}

:resource-value-index
{:write-buffer-size-in-mb 64
:max-bytes-for-level-base-in-mb 512
:max-bytes-for-level-base-in-mb 256
:target-file-size-base-in-mb 64
:block-size #blaze/cfg ["DB_BLOCK_SIZE" int? 16384]}

:compartment-search-param-value-index
{:write-buffer-size-in-mb 64
:max-bytes-for-level-base-in-mb 512
:max-bytes-for-level-base-in-mb 256
:target-file-size-base-in-mb 64
:block-size #blaze/cfg ["DB_BLOCK_SIZE" int? 16384]}

Expand Down Expand Up @@ -673,19 +673,19 @@
:column-families
{:search-param-value-index
{:write-buffer-size-in-mb 64
:max-bytes-for-level-base-in-mb 512
:max-bytes-for-level-base-in-mb 256
:target-file-size-base-in-mb 64
:block-size #blaze/cfg ["DB_BLOCK_SIZE" int? 16384]}

:resource-value-index
{:write-buffer-size-in-mb 64
:max-bytes-for-level-base-in-mb 512
:max-bytes-for-level-base-in-mb 256
:target-file-size-base-in-mb 64
:block-size #blaze/cfg ["DB_BLOCK_SIZE" int? 16384]}

:compartment-search-param-value-index
{:write-buffer-size-in-mb 64
:max-bytes-for-level-base-in-mb 512
:max-bytes-for-level-base-in-mb 256
:target-file-size-base-in-mb 64
:block-size #blaze/cfg ["DB_BLOCK_SIZE" int? 16384]}

Expand Down Expand Up @@ -804,7 +804,7 @@
{:default
{:write-buffer-size-in-mb 8
:max-bytes-for-level-base-in-mb 32
:target-file-size-base-in-mb 3
:target-file-size-base-in-mb 8
:block-size #blaze/cfg ["DB_BLOCK_SIZE" int? 16384]}}}

[:blaze.db.kv.rocksdb/stats :blaze.db.transaction-kv-store.main/stats] {}
Expand All @@ -818,7 +818,7 @@
{:default
{:write-buffer-size-in-mb 8
:max-bytes-for-level-base-in-mb 32
:target-file-size-base-in-mb 3
:target-file-size-base-in-mb 8
:block-size #blaze/cfg ["DB_BLOCK_SIZE" int? 16384]}}}

;;
Expand Down Expand Up @@ -850,7 +850,7 @@
{:default
{:write-buffer-size-in-mb 64
:max-bytes-for-level-base-in-mb 256
:target-file-size-base-in-mb 26
:target-file-size-base-in-mb 64
;; According to Synthea test data, a resource has an average size of
;; 1300 bytes. So a block with a size of 16 kb will hold about 10
;; resources in average.
Expand Down Expand Up @@ -921,19 +921,19 @@
:column-families
{:search-param-value-index
{:write-buffer-size-in-mb 64
:max-bytes-for-level-base-in-mb 512
:max-bytes-for-level-base-in-mb 256
:target-file-size-base-in-mb 64
:block-size #blaze/cfg ["DB_BLOCK_SIZE" int? 16384]}

:resource-value-index
{:write-buffer-size-in-mb 64
:max-bytes-for-level-base-in-mb 512
:max-bytes-for-level-base-in-mb 256
:target-file-size-base-in-mb 64
:block-size #blaze/cfg ["DB_BLOCK_SIZE" int? 16384]}

:compartment-search-param-value-index
{:write-buffer-size-in-mb 64
:max-bytes-for-level-base-in-mb 512
:max-bytes-for-level-base-in-mb 256
:target-file-size-base-in-mb 64
:block-size #blaze/cfg ["DB_BLOCK_SIZE" int? 16384]}

Expand Down Expand Up @@ -1027,19 +1027,19 @@
:column-families
{:search-param-value-index
{:write-buffer-size-in-mb 64
:max-bytes-for-level-base-in-mb 512
:max-bytes-for-level-base-in-mb 256
:target-file-size-base-in-mb 64
:block-size #blaze/cfg ["DB_BLOCK_SIZE" int? 16384]}

:resource-value-index
{:write-buffer-size-in-mb 64
:max-bytes-for-level-base-in-mb 512
:max-bytes-for-level-base-in-mb 256
:target-file-size-base-in-mb 64
:block-size #blaze/cfg ["DB_BLOCK_SIZE" int? 16384]}

:compartment-search-param-value-index
{:write-buffer-size-in-mb 64
:max-bytes-for-level-base-in-mb 512
:max-bytes-for-level-base-in-mb 256
:target-file-size-base-in-mb 64
:block-size #blaze/cfg ["DB_BLOCK_SIZE" int? 16384]}

Expand Down

0 comments on commit 4c7d31e

Please sign in to comment.