BUG: free(): invalid pointer #471
Comments
|
Thanks for reporting, I can confirm this is a serious bug that applies to all versions. It seems to be some race condition when resetting the curl instance: curl_cffi/curl_cffi/requests/models.py Line 198 in 8cfad55 |
|
I'm glad an issue was created, I was intending to after confirming it was a problem with this library and creating a consistent reproduction script. We use curl-cffi in production, and have been dealing with segfault
abortlatest curl_cffi ver |
|
Thanks for the additional information. I saw |
|
I've been using curl_cffi through yt-dlp, and I've encountered frequent memory-related errors (listed below in descending order of frequency): All of these issues disappeared immediately once I monkey-patched curl_cffi to make the reset method a no-op: Of course, this is just a temporary, hacky fix that works for my specific deployment scenario, where a separate yt-dlp process is spun up, makes queries, and then terminates quickly. This approach likely won't be suitable for long-running processes because memory may accumulate (leading to a leak) if From what I can tell, the calls to |
|
Thanks for the report. I can almost reach a conclusion that |
|
Have you ever encountered this issue when |
|
The current implementation of We have 2 options here:
However, I don't know how long it takes, especially for option 2. |
|
I've also been encountering this issue with stream, but only in multithreaded code. The same usage patterns have been very stable if there's only one thread. I agree that this points to a race condition in the stream code, but I haven't been able to track it down. |
|
I'm posting my finding here, in case anyone have more interest and time in fixing this. Basically, our problem is that And it's actually a simple function: static CURLcode easy_transfer(struct Curl_multi *multi)
{
bool done = FALSE;
CURLMcode mcode = CURLM_OK;
CURLcode result = CURLE_OK;
while(!done && !mcode) {
int still_running = 0;
mcode = curl_multi_poll(multi, NULL, 0, 1000, NULL);
if(!mcode)
mcode = curl_multi_perform(multi, &still_running);
/* only read 'still_running' if curl_multi_perform() return OK */
if(!mcode && !still_running) {
int rc;
CURLMsg *msg = curl_multi_info_read(multi, &rc);
if(msg) {
result = msg->data.result;
done = TRUE;
}
}
}
/* Make sure to return some kind of error if there was a multi problem */
if(mcode) {
result = (mcode == CURLM_OUT_OF_MEMORY) ? CURLE_OUT_OF_MEMORY :
/* The other multi errors should never happen, so return
something suitably generic */
CURLE_BAD_FUNCTION_ARGUMENT;
}
return result;
}So, what we can do is: static CURLcode easy_transfer(struct Curl_multi *multi)
{
bool done = FALSE;
CURLMcode mcode = CURLM_OK;
CURLcode result = CURLE_OK;
while(!done && !mcode) {
int still_running = 0;
mcode = curl_multi_poll(multi, NULL, 0, 1000, NULL);
if(!mcode)
mcode = curl_multi_perform(multi, &still_running);
+ // go back to python
+ // data = read_buffer()
+ // go back to c
/* only read 'still_running' if curl_multi_perform() return OK */
if(!mcode && !still_running) {
int rc;
CURLMsg *msg = curl_multi_info_read(multi, &rc);
if(msg) {
result = msg->data.result;
done = TRUE;
}
}
}
/* Make sure to return some kind of error if there was a multi problem */
if(mcode) {
result = (mcode == CURLM_OUT_OF_MEMORY) ? CURLE_OUT_OF_MEMORY :
/* The other multi errors should never happen, so return
something suitably generic */
CURLE_BAD_FUNCTION_ARGUMENT;
}
return result;
} |
No, I didn't get any errors if I don't use stream. yt-dlp uses curl-cffi with |
Describe the bug
I create a Session instance and use streaming responses. But after a while I get memory related errors, for example free(): invalid pointer.
Error message content
To Reproduce
Versions
pip freezefreeze.txtThe text was updated successfully, but these errors were encountered: