Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Project Cleaning #2187

Merged
merged 1 commit into from
Nov 14, 2024
Merged

Improve Project Cleaning #2187

merged 1 commit into from
Nov 14, 2024

Conversation

allentiak
Copy link
Member

In order to initialize properly, the server needs the Job IG resource files in specific locations (see below).
Since those files are prepped only if they did not exist before, we need do delete them every time we make a make clean.

Context:

  • the Job IG resource files are generated by make -C job-ig build, and saved in the folder job-ig/fsh-generated/resources.

  • then, when clojure -X:deps prep is executed, the resource files are copied to the individual resources
    path's (e.g.
    modules/job-re-index/resources/blaze/job/re_index/CodeSystem-ReIndexJobOutput.json) via build/copy-profiles. But this only happens if the folders don't already exist. See
    https://clojure.org/reference/deps_edn#prep_lib

If the server cannot find the files in their right locations, it will fail to initialize with the following error:

2024-11-11T16:30:08.301Z d2e461cd82fc main ERROR [blaze.core:21] - Error
while initializing Blaze.

    Error on key :blaze/admin-api when building system

    Cause:

    Config:
      ADMIN_INDEX_DB_DIR = /app/data/admin-index,
      ADMIN_TRANSACTION_DB_DIR = /app/data/admin-transaction,
      HOME = /,
      HOSTNAME = d2e461cd82fc,
      INDEX_DB_DIR = /app/data/index,
      JAVA_HOME = /opt/java/openjdk,
      JAVA_VERSION = jdk-21.0.5+11,
      JRE_CACERTS_PATH = /opt/java/openjdk/lib/security/cacerts,
      LANG = en_US.UTF-8,
      LANGUAGE = en_US:en,
      LC_ALL = en_US.UTF-8,
      LD_PRELOAD = libjemalloc.so.2,
PATH =
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,
      PWD = /app,
      RESOURCE_DB_DIR = /app/data/resource,
      SHLVL = 0,
      STORAGE = standalone,
      TRANSACTION_DB_DIR = /app/data/transaction

@allentiak allentiak changed the title Fix Server Initialization Error Fix Server Initialization Error (WIP) Nov 14, 2024
Copy link

codecov bot commented Nov 14, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.99%. Comparing base (8dc3378) to head (f07384c).
Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #2187   +/-   ##
=======================================
  Coverage   94.99%   94.99%           
=======================================
  Files         330      330           
  Lines       20249    20249           
  Branches      487      486    -1     
=======================================
  Hits        19236    19236           
  Misses        527      527           
  Partials      486      486           

@allentiak allentiak marked this pull request as ready for review November 14, 2024 12:54
Copy link
Member

@alexanderkiel alexanderkiel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deleting resources can't be done always.

modules/db/Makefile Outdated Show resolved Hide resolved
modules/db/src/blaze/db/node/resource_indexer.clj Outdated Show resolved Hide resolved
@allentiak allentiak changed the title Fix Server Initialization Error (WIP) Fix Server Initialization Error Nov 14, 2024
@allentiak allentiak force-pushed the fix-make-clean branch 2 times, most recently from fac2c95 to 1d5b0f6 Compare November 14, 2024 15:48
@allentiak allentiak changed the title Fix Server Initialization Error Improve Project Cleaning (fixes Local Server Initialization Error) Nov 14, 2024
This PR fixes a local server startup error due to missing Job IG
resource files.

Context:

When locally building the project more than once in a row, we need to
make sure `make clean` removes every file that is automatically
generated during the build.

More specifically, in order to initialize properly, the server needs the
Job IG resource
files in specific locations (see below).
Since those files are prepped only if they did not exist before, we need
do delete them every time we make a `make clean`.

Details:

* the Job IG resource files are generated by `make -C job-ig build`, and
saved in the folder `job-ig/fsh-generated/resources`.

* then, when `clojure -X:deps prep` is executed, the resource files are
copied to the individual resources
path's (e.g.
`modules/job-re-index/resources/blaze/job/re_index/CodeSystem-ReIndexJobOutput.json`)
via `build/copy-profiles`. But this only happens if the folders don't
already exist. See
https://clojure.org/reference/deps_edn#prep_lib

Before this, when building the project more than once in a row, the
server could not find the Job IG resource files in their right
locations, and it would
fail to initialize with the following error:
```
2024-11-11T16:30:08.301Z d2e461cd82fc main ERROR [blaze.core:21] - Error
while initializing Blaze.

    Error on key :blaze/admin-api when building system

    Cause:

    Config:
      ADMIN_INDEX_DB_DIR = /app/data/admin-index,
      ADMIN_TRANSACTION_DB_DIR = /app/data/admin-transaction,
      HOME = /,
      HOSTNAME = d2e461cd82fc,
      INDEX_DB_DIR = /app/data/index,
      JAVA_HOME = /opt/java/openjdk,
      JAVA_VERSION = jdk-21.0.5+11,
      JRE_CACERTS_PATH = /opt/java/openjdk/lib/security/cacerts,
      LANG = en_US.UTF-8,
      LANGUAGE = en_US:en,
      LC_ALL = en_US.UTF-8,
      LD_PRELOAD = libjemalloc.so.2,
PATH =
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,
      PWD = /app,
      RESOURCE_DB_DIR = /app/data/resource,
      SHLVL = 0,
      STORAGE = standalone,
      TRANSACTION_DB_DIR = /app/data/transaction
```
@alexanderkiel alexanderkiel changed the title Improve Project Cleaning (fixes Local Server Initialization Error) Improve Project Cleaning Nov 14, 2024
@alexanderkiel
Copy link
Member

Thanks, so root of the problem was that make clean did not work properly and so building Blaze on a non fresh repo checkout will not work. That's also the reason, the issue did not surface on the CI pipeline, because it always uses fest checkouts.

@alexanderkiel alexanderkiel added this pull request to the merge queue Nov 14, 2024
Merged via the queue into main with commit 6f94657 Nov 14, 2024
133 checks passed
@alexanderkiel alexanderkiel deleted the fix-make-clean branch November 14, 2024 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants