How to properly hold onto an object in embedded mode? #267
Comments
Do you mean gevent handles it, or this technique is only employed when gevent information is required? What specific type of queue are using? collections.deqeue?
You can also do it continuation/callback style. Meaning, pass the consuming Clojure function as an argument to the Python function.
Have you considered using the actual tap function? i.e., Not positive it will help, just trying to help out with some additional options-- you are in uncharted waters, so I want to give you as many tools as I can.
I've written many production web applications and search engines with libpython-clj and I treat the the clojure/python "barrier" as if it were a serialized interface-- I am not shy about using ->jvm, ->py, etc as needed. I also tend to stick to core data structures that have strong transference such hashmaps/dicts and lists/vectors. If not you would need to implement some sort of memento/state pattern. (I will also mention if this a highly performant commercial application where absolute performance is critical, you may want to contact TechAcent directly) |
|
Hi, @jjtolton ! Thanks for helping out :)
Not sure I understood the question. In my setup, the python app is running in on the main thread, mostly unaware that clojure threads exist. The python app uses gevent internally for the non-blocking IO. For example, there are functions, that pull various data from databases, and those use connection pools. Python code from the main thread can call them freely, but trying the same from other threads (python or not) will lead to errors due to gevent hub being bound to the main thread. So, when trying to call those functions from clojure, I had to do it through the special queue (see below).
I had to use thread-safe
Returning a value from a task on a queue is one thing (that can be done via callback, indeed), another is waiting for the value to be ready on the other side. One-off queues solve both. from queue import Queue
from gevent import spawn, get_hub
from . import clojure # some cljbridge-inspired tooling
...
def spawn_real_thread(fn, *args, **kwargs):
'''Spawns real OS thread, while loops are running on gevent'''
pool = get_hub().threadpool
return pool.spawn(fn, *args, **kwargs)
def start_clojure_thread(aliases=None, with_repl=True, post_init=None):
'''Make sure repl deps are there, if you want to start it'''
# Must be done on the main thread
jvm_params = clojure.prepare_jvm_params(aliases)
def clj_thread():
clojure.init(jvm_params)
if with_repl:
clojure.start_repl("0.0.0.0", 50000)
if post_init:
post_init()
while True:
time.sleep(3600)
spawn_real_thread(clj_thread)
spawn(main_thread_worker)
__MAIN_THREAD_QUEUE = Queue()
def call_on_main_thread(fn, *args, **kwargs):
return do_on_main_thread(True, fn, *args, **kwargs)
def do_on_main_thread(wait, fn, *args, **kwargs):
retq = Queue() if wait else None
# logger.info("[dmt] attempting put: %r", __MAIN_THREAD_QUEUE)
__MAIN_THREAD_QUEUE.put((fn, None, args, kwargs, retq))
# logger.info("[dmt] put done")
if not wait:
return None
# logger.info("[dmt] waiting for result")
res, exc = retq.get()
retq.task_done()
if exc is not None:
raise exc
return res
def main_thread_worker():
while True:
# logger.warning("[dmq] waiting on queue: %r", __MAIN_THREAD_QUEUE)
fn, coro, args, kwargs, retq = __MAIN_THREAD_QUEUE.get()
# logger.info("got task: %s", (fn or coro, args, kwargs, retq))
if fn:
_run_function(fn, args, kwargs, retq)
else:
assert not (args or kwargs), "Not expecting args/kwargs for coros"
_run_coro(coro, retq)
# logger.info("[dmq] spawned glet")
__MAIN_THREAD_QUEUE.task_done()
def _run_function(fn, args, kwargs, retq):
'''Call this from main thread'''
def glet():
try:
res = fn(*args, **kwargs)
# logger.warning("res: %s", res)
if retq:
retq.put((res, None))
except Exception as e:
if retq:
retq.put((None, e))
else:
logger.exception("[glt] error: %s", e)
spawn(glet)And this is how I call it from clojure: (defmacro call-on-main-thread [& forms]
`(app.clojure_gevent_support/call_on_main_thread
#(do ~@forms)))
(defn get-location-titles [loc-id]
(-> (call-on-main-thread
(geo/titles_by_location loc-id)) ;; geo is a python module
->jvm))
I guess it won't be different, but I'll try. This is what I've done initially ( (defonce qparams (atom [])
(defn save-query-params [new-query-params]
(swap! qparams conj new-query-params))Accessing content of the atom leads to segfault. |
|
There are a lot of moving parts here (by design, of course) so it's hard to pinpoint the issue precisely. Would you please paste the segfault log, i.e., |
|
I believe the problem lies not in the interaction of these moving parts, but in my lack of understanding of how objects should be shared between python world and jvm world when using libpython-clj. I've cut down everything irrelevant (the queue I mentioned before just to draw a full picture of what I'm trying to do) and made a repo to reproduce the issue: libpython-repro Important files are just these two:
I've also added a log from one of the crashes: |
|
Thanks for that. I don't have a solution yet, but I was able to narrow the problem down considerably: (ns eeel.repro
(:require
[libpython-clj2.python :as py :refer [py..]]
[libpython-clj2.require :refer [require-python import-python]]
;;
))
(import-python)
(defonce taps (atom []))
(defn save-to-tap [obj]
(swap! taps conj obj))
(comment
(let [{{save-something "save_something"
set-tap "set_tap"}
:globals} (py/run-simple-string "
import json
from pprint import pformat
TAP = None
def set_tap(fn):
global TAP
TAP = fn
def save_something():
'''To be called from clojure'''
test_obj = json.loads('{\"test\": [1, 2, 3, 4]}')
print(\"saved:\", pformat(test_obj))
if TAP:
TAP(test_obj)
")]
(def save-something save-something)
(def set-tap set-tap)
)
(set-tap save-to-tap)
(save-something)
@taps ;;=> [{'__str__': [{...}, [...], 3, 4]}]
@taps ;; segfault
) |
|
I don't think this has anything to do with async, it looks like the string is being double-freed. |
|
If I change the test to the following: (ns eeel.repro
(:require
[libpython-clj2.python :as py :refer [py..]]
[libpython-clj2.require :refer [require-python import-python]]
;;
))
(import-python)
(defonce taps (atom []))
(defn save-to-tap [obj]
(swap! taps conj obj))
(comment
(let [{{save-something "save_something"
set-tap "set_tap"}
:globals} (py/run-simple-string "
import json
from pprint import pformat
TAP = None
def set_tap(fn):
global TAP
TAP = fn
def save_something():
'''To be called from clojure'''
test_obj = json.loads('{\"test\": [1, 2, 3, 4]}')
print(\"saved:\", test_obj)
if TAP:
TAP(test_obj)
")]
(def save-something save-something)
(def set-tap set-tap))
(set-tap tap>)
(add-tap println)
(save-something))repeatedly invoking saved: {'test': [1, 2, 3, 4]}
{'Py_Repr': [{...}, [...]]}
saved: {'test': [1, 2, 3, 4]}
saved: {'test': [1, 2, 3, 4]}
{'Py_Repr': [{...}, [...]]}
saved: {'test': [1, 2, 3, 4]}
{'Py_Repr': [{...}, [...]]}
saved: {'test': [1, 2, 3, 4]}
saved: {'test': [1, 2, 3, 4]}
{'Py_Repr': [{...}, [...]]} |
|
Thanks for looking into it. Before crashing it usually prints some garbage, when inspecting contents of What I do not understand is why test objects look corrupted before sending them to |
|
Yeah I understand. Ideally that wouldn't happen -- but I'd like to offer a practical workaround. I tend to follow a "render unto Caesar that which is Caesar's" approach with mixed runtime programming, and follow the path of least resistance. Perhaps consider the following: import json
from pprint import pformat
from collections import deque
taps = deque()
TAP = lambda x: taps.append(x)
def save_something():
'''To be called from clojure'''
test_obj = json.loads('{\"test\": [1, 2, 3, 4]}')
print(\"saved:\", test_obj)
if TAP:
TAP(test_obj)Full example: (ns eeel.repro
(:require
[libpython-clj2.python :as py :refer [py..]]
[libpython-clj2.require :refer [require-python import-python]]
;;
))
(import-python)
(defonce taps (atom []))
(defn save-to-tap [obj]
(swap! taps conj obj))
(comment
(let [{{save-something "save_something"
set-tap "set_tap"}
:globals} (py/run-simple-string "
import json
from pprint import pformat
from collections import deque
taps = deque()
TAP = lambda x: taps.append(x)
def save_something():
'''To be called from clojure'''
test_obj = json.loads('{\"test\": [1, 2, 3, 4]}')
print(\"saved:\", test_obj)
if TAP:
TAP(test_obj)
")]
(def save-something save-something)
(def set-tap set-tap))
(save-something)
(save-something)
(save-something)
(require-python '[__main__ :bind-ns true])
__main__/taps ;; => deque([{'test': [1, 2, 3, 4]}, {'test': [1, 2, 3, 4]}, {'test': [1, 2, 3, 4]}])
)We'd have to dig into the serialization FFI to find out why the string is not being marshalled correctly, but this will probably get you where you are going a little better since you are using Python's machinery instead of going through the data marshalling code. |
|
I think there is no crash with real |
|
The real |
I'm still not sure the root cause, but you seem to be correct that when the Python reference is lost, we are dereferencing a null pointer. When I hang on to the references, the (ns eeel.repro
(:require
[libpython-clj2.python :as py :refer [py..]]
[libpython-clj2.require :refer [require-python import-python]]
;;
))
(import-python)
(defonce taps (atom []))
(defn save-to-tap [obj]
(swap! taps conj obj))
(comment
(let [{{save-something "save_something"
set-tap "set_tap"}
:globals} (py/run-simple-string "
import json
from pprint import pformat
from collections import deque
taps = deque()
_TAP = None
TAP = lambda x: (taps.append(x), _TAP(x))
def set_tap(fn):
global _TAP
_TAP = fn
def save_something():
'''To be called from clojure'''
test_obj = json.loads('{\"test\": [1, 2, 3, 4]}')
print(\"saved:\", test_obj)
if TAP:
TAP(test_obj)
")]
(def save-something save-something)
(def set-tap set-tap))
(set-tap save-to-tap)
(save-something)
(save-something)
(save-something)
(require-python '[__main__ :bind-ns true])
__main__/taps ;; => deque([{'test': [1, 2, 3, 4]}, {'test': [1, 2, 3, 4]}, {'test': [1, 2, 3, 4]}])
@taps ;; [{'test': [1, 2, 3, 4]} {'test': [1, 2, 3, 4]} {'test': [1, 2, 3, 4]}]
) |
|
Yeah, there is a workaround (similar to your I've also tried doing |
|
Hmmm something tells me you are doing more than "tap for visual inspection" with this workflow, then, because otherwise manually clearing the |
|
The issue is that we might not have a control over the container, that holds objects. In my simple example we do have it, but imagine we would like to send tapped object to portal. If we're "borrowing" (or putting into dequeue) them (either on python side, or in clojure with incref), all is fine until the moment we've cleared the values from portal. After that objects are, essentially, "leaked". Because there is no way for us to know when values are released from the UI. |
|
Ahhh, I see. You need them to be in Clojure because they are hooked up to Clojure tools you don't control, such as portal, and you don't know when the reference will be released. I am curious why you would want to send a raw python object to something like Portal? I don't think it was designed to inspect those. You'd most likely be better off converting to a JVM object anyway. Are you using Portal for monitoring in production, or something? It's hard for me to imagine that in a dev environment you could accumulate so many references that you'd eat up all the system memory holding onto references, and you could use flags to disable the behavior in production. |
|
I'll be honest this is one area where I'm not exactly clear what the correct behavior is. Python is right to GC those strings, Clojure is right to hold a reference to them. I suppose the ideal would be to inform Python's garbage collector that Clojure is still holding references, then hook into the JVM garbage collector to notify Python's garbage collector when it GCs a JVM-managed python reference -- but I'm not sure that the "juice is worth the squeeze" there. |
|
I thought I remembered there being a somewhat simple trick to get around this 🤔 Something like setting a variable in Python, getting the data in Clojure, then clearing the variable in Python. |
|
Well, given the options, this still seems like the best bet: (ns eeel.repro
(:require
[libpython-clj2.python :as py :refer [py..]]
[libpython-clj2.require :refer [require-python import-python]]
;;
))
(import-python)
(defonce taps (atom []))
(defonce taps1 (atom []))
(defn save-to-tap [obj]
(swap! taps conj obj))
(comment
(let [{{save-something "save_something"
set-tap "set_tap"}
:globals} (py/run-simple-string "
import json
from pprint import pformat
from collections import deque
_TAP = None
to_jvm = None
def TAP(x):
_TAP(to_jvm(x))
def set_tap(fn):
global _TAP
_TAP = fn
def save_something():
'''To be called from clojure'''
test_obj = json.loads('{\"test\": [1, 2, 3, 4]}')
print(\"saved:\", test_obj)
if TAP:
TAP(test_obj)
")]
(def save-something save-something)
(def set-tap set-tap))
(require-python '[__main__ :bind-ns true])
(python/setattr __main__ "to_jvm" py/->jvm)
(set-tap save-to-tap)
(save-something)
(save-something)
(save-something)
@taps ;; => [{"test" [1 2 3 4]} {"test" [1 2 3 4]} {"test" [1 2 3 4]}]
)If you need something more complex than what |
Hello! First, I'd like to thank you for the amazing work on this library. It's a mind-bending level of inter-language interaction, IMO. 👏
I was experimenting with it in the embedded mode. Not sure if I was using it in the intended way, but... I took a considerably sized web-app and tried to add some kind of a debugging/introspection interface to it with clj & cljs.
The app uses gevent internally for non-blocking IO operations, so the setup was a bit involving, but in the end it worked out. I'll explain it to get a better picture:
-mainof the clojure's part of the appIt worked fine, I've even made a little introspection tool, that can peer into a running python app vars and call functions, toggle stuff and so on.
Then, I've tried to implement a
tap>like system (just putting python objects into an atom for later inspection), to send info from python to clojure... And was abruptly stopped by a SIGSEGV. :(It seems that objects I send through "tap" got GC'ed on python part. I've tried to find something about holding to objects in the documentation, but nothing was particularly fitting to my case. Digging through the code, I've stumbled upon
track-pyobjectandincref-and-track. Using the latter on the objects I send to clojure world did not help though... I still got a SIGSEGV, but in different internal python function. To validate my assumption about objects being GD'ed, I've implemented a dumb "borrowing" mechanics on python side (putting objects, to be sent to clojure, in a dict) an it worked okay.I'd prefer not to do
->jvmon objects, because the idea is to use them again in a python app to replay some actions that it performs.My question: Is there a way to hold onto python objects from clojure in embedded mode so that refcounts will be decreased when references to the pyobjs are GC'ed on the JVM?
Thank you for your time :)
Python 3.10.12
Clojure 1.12
openjdk 17.0.12 2024-07-16
OpenJDK Runtime Environment (build 17.0.12+7-Ubuntu-1ubuntu222.04)
OpenJDK 64-Bit Server VM (build 17.0.12+7-Ubuntu-1ubuntu222.04, mixed mode, sharing)
The text was updated successfully, but these errors were encountered: