feat: add background processing jobs

[jfcalvo]

Description

This PR add the following changes:

• Add rq to help us execute background jobs.
• Add a background job to update all records for a dataset when the dataset distribution strategy is updated.
• Change HuggingFace Dockerfile to install Redis and run rq workers inside honcho Procfile.
• Add documentation about new ARGILLA_REDIS_URL environment variable.
• Add ping to Redis so Argilla server is not started if Redis is not ready.
• Change Argilla docker compose file to include a container with Redis and rq workers.
• Update Argilla server README.md file adding Redis as dependency to install.
• Add documentation about Redis being a new Argilla server dependency.
• Add BACKGROUND_NUM_WORKERS environment variable to specify the number of workers in the HF Space container.
• Modify Dockerfile template on HF to include the environment variable [TASK] Modify HF Space template to include a comment about BACKGROUND_NUM_WORKERS environment variable #5443:

# (since: v2.2.0) Uncomment the next line to specify the number of background job workers to run (default: 2).
# ENV BACKGROUND_NUM_WORKERS=2

• Remove some TODO sections before merging.
• Review K8s documentation (maybe delete it?).
• If we want to persist Redis data on HF Spaces we can change our Procfile Redis process to the following:

redis: /usr/bin/redis-server --dbfilename argilla-redis.rdb --dir ${ARGILLA_HOME_PATH}

• Allow tests job workers synchronously (with pytest) It's not working due to asyncio stuff (running an asynchronous loop inside another one, more info here: Task got Future attached to a different loop rq/rq#1986).

Closes #5431

Benchmarks

The following timings were obtained updating the distribution strategy of a dataset with 100 and 10.000 records, using a basic and an upgraded CPU on HF Spaces, with and without persistent storage and measuring how much time the background job takes to complete:

CPU basic: 2 vCPU, 16GB RAM
CPU upgrade: 8 vCPU, 32GB RAM

• CPU basic (with persistent storage):
  • 100 records dataset: ~8 seconds.
  • 10.000 records dataset: ~9 minutes.
• CPU upgrade (with persistent storage):
  • 100 records dataset: ~5 seconds.
  • 10.000 records dataset: ~6 minutes.
• CPU basic (no persistent storage):
  • 10.000 records dataset: ~101 seconds.
• CPU upgrade (no persistent storage):
  • 10.000 records dataset: ~62 seconds.

Type of change

• New feature (non-breaking change which adds functionality)

How Has This Been Tested

• Testing it on HF Spaces.

Checklist

• I added relevant documentation
• I followed the style guidelines of this project
• I did a self-review of my code
• I made corresponding changes to the documentation
• I confirm My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)

[codecov]

Codecov Report

Attention: Patch coverage is 63.63636% with 20 lines in your changes missing coverage. Please review.

Project coverage is 91.00%. Comparing base (fee1f5a) to head (96c96dc).
Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
argilla-server/src/argilla_server/_app.py	27.27%	8 Missing ⚠️
...lla-server/src/argilla_server/jobs/dataset_jobs.py	63.63%	8 Missing ⚠️
argilla-server/src/argilla_server/cli/worker.py	55.55%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #5432      +/-   ##
===========================================
- Coverage    91.32%   91.00%   -0.33%     
===========================================
  Files          141      144       +3     
  Lines         5869     5915      +46     
===========================================
+ Hits          5360     5383      +23     
- Misses         509      532      +23     

Flag	Coverage Δ
argilla-server	91.00% <63.63%> (-0.33%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

The URL of the deployed environment for this PR is https://argilla-quickstart-pr-5432-ki24f765kq-no.a.run.app

[frascuchon]

Suggested change

	- `ARGILLA_REDIS_URL`: A URL string that contains the necessary information to connect to a Redis instance (Default: `redis://localhost:6379/0`).
	- `ARGILLA_REDIS_URL`: A URL string that contains the necessary information to connect to a Redis instance (Default: `redis://localhost:6379`).

[jfcalvo]

The 0 it's the Redis database. We can remove it and it will use by default the db 0 but I though it was better to be explicit.