Memory leak when using CurlOpt.RESOLVE

[Ehsan-U]

I was facing a memory leak in long-running spiders. After investigating for about a month, it turned out that only the spiders using CurlOpt.RESOLVE were affected and leak was originating from curl-cffi.

After adding this one line into reset() leak disappeared . I’m not entirely sure if this is the correct or best approach, but it does stop the leak.

curl_cffi/curl.py

def reset(self) -> None:
    self._is_cert_set = False

    lib.curl_slist_free_all(self._resolve)  # Add this line

    if self._curl is not None:
        lib.curl_easy_reset(self._curl)
        self._set_error_buffer()

    self._resolve = ffi.NULL

[lexiforest]

I guess you were passing CurlOpt.RESOLVE on each request, not to the session object, right?

[Ehsan-U]

I guess you were passing CurlOpt.RESOLVE on each request, not to the session object, right?

i was using AsyncSession and passing curl_options to that. not per-request

[lexiforest]

I got it, this is indeed a memory leak and your fix should be correct.

[Ehsan-U]

Hi @lexiforest
i have attached the PR to fix this bug. please feel free to review at your convenience

[lexiforest]

Were you using stream=True in your requests?

[Ehsan-U]

No

[lexiforest]

That's weird, I just found out we actually released the slist here:

curl_cffi/curl_cffi/curl.py

Lines 445 to 448 in 9f566fa

	if clear_resolve:
	if self._resolve != ffi.NULL:
	lib.curl_slist_free_all(self._resolve)
	self._resolve = ffi.NULL