3.9.4 has a threading issue

[lejmr]

After upgrading to 3.9.4 from 3.9.2, I started to experience freezing of my application. With the downgrade, I can confirm the same issue happens with 3.9.3 but not with 3.9.2!

My application utilizes threads for receiving messages from multiple topics. When I receive two messages almost simultaneously (below 8ms time difference), the application freezes on orjson.loads call. Freezes = thread that stucks on json.loads consumes 100% of CPU, and all other threads stop. Nothing happens at all - even prometheus_client is not providing any metrics, i.e., HTTP interface timeouts.

For troubleshooting, I used strace which shows only repeating lines:

futex(0x7f98d4338d74, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=79294, tv_nsec=4595386}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f98d4338d78, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f98d4338d74, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=79294, tv_nsec=9691159}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x7f98d4338d78, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f98d4338d74, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=79294, tv_nsec=14785932}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)

full strace: strace.txt

[adlaube]

+1
We suffered from the same symptoms in our cluster (orjson 3.9.4) and downgrading to 3.9.2 solved this issue for us.
All application freezes we analyzed were caused by orjson.loads.

[ijl]

Thanks for the report. If you have time to produce a test case that would help. It could modify or fork from integration/wsgi.py or integration/thread.

What environment is this?

3.9.3 but not 3.9.2 suggests f94318a, e9b745e.

[jarus]

Hi @ijl,

I'm in the same team as @adlaube. First of all Thank you for orjson! We experienced a great speedup from using orjson and it is nowadays our main JSON implementation that we use within our applications.

Before the downgrade to orjson 3.9.2 (orjson-3.9.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl sha256:275b5a18fd9ed60b2720543d3ddac170051c43d680e47d04ff5203d2c6d8ebf1) we used orjson 3.9.4 (orjson-3.9.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl sha256:1e4b20164809b21966b63e063f894927bc85391e60d0a96fa0bb552090f1319c). In both cases we use it in CPython 3.11.3 runtime. Our application is a combination of Flask 2.3.2 + Cheroot 8.4.1 where HTTP requests and other async tasks are processed via Python multithreading and asyncio. We will try to provide a proper reproducible.

Beside the already mentioned application freezes, we also saw sporadic segfaults which seem to come from orjson but as also other C and Rust extensions are used in our application, we can't 100% confirm that yet. In case of application freezes we can prove that the application froze as the thread calling orjson didn't release the GIL anymore as it was stuck somewhere in orjson.

[sirfz]

Can confirm, experienced hanging in python 3.11 and segfaults in python 3.9. Downgrading to 3.9.2 as others mentioned resolved the issue.

[ijl]

3.9.5 removes the futex. I think going further requires a test exercising what you expect to work.

[adlaube]

Hi @ijl,

unfortunately I have not been able to reproduce this issue locally yet. Interestingly this issue did not occur in our test landscape indicating that a consistent high load is required.
I have rebuilt 3.9.4 with debug symbols to produce some traces from our core dumps [0].

For future issues it might be helpful to include a minimal set of symbols: debug settings

Let me know if there is any further useful information I could be able to retrieve from the dumps for you.
We will assess if the issue is still present in production with 3.9.5 and give you a feedback.

[0]

Case 0: hanging thread which holds the GIL

[Switching to thread 104 (Thread 0x7effcf7fe700 (LWP 118))]
#0  0x00007f01897c9876 in pool_malloc (size=8, ctx_ptr=<optimized out>) at include/yyjson/yyjson.c:1027
1027	include/yyjson/yyjson.c: No such file or directory.
#0  0x00007f01897c9876 in pool_malloc (size=8, ctx_ptr=<optimized out>) at include/yyjson/yyjson.c:1027
#1  pool_realloc (ctx_ptr=<optimized out>, ptr=0x7eff39072740, old_size=<optimized out>, size=8) at include/yyjson/yyjson.c:1115
#2  0x00007f01896e8700 in ?? ()
#3  0x00007f018e630e80 in ?? ()
#4  0x0000000000000000 in ?? ()
prev = <optimized out>
cur = 0x7effce7ff710
next = 0x7effce7fd050

Case 1: Segmentation fault

segfaults_orjson_3.9.4.txt

[lejmr]

Sadly for us, the bug is still present. I wrote a minimal test but didn't raise the issue... will give it another shot.

[andersk]

I can reproduce this crash in the Zulip development environment by reloading the page a few dozen times. I’m seeing similar symptoms: various segfaults and 100% CPU freezes. The environment is Python 3.8.10 on Ubuntu 20.04.6 LTS x86-64.

The first bad commit seems to be e9b745e. Moreover, I can reproduce the crash by cherry-picking only those buffer refactoring changes in deserialize_yyjson, YYJSON_ALLOC, yyjson_init onto 3.9.2 (here’s the exact code for that). Adding caae033 doesn’t help.

Obviously Zulip is a pretty complicated application and far from an ideal minimal test case, but at least it’s open source and you can run it yourself; alternatively, I’m happy to try out any debugging suggestions you might have.