Sasha/batching #11
Sasha/batching #11
Conversation
references/utils.py
Outdated
| items[i : i + batch_size] for i in range(0, len(items), batch_size) | ||
| ] | ||
| agg_response = UploadResponse(json={"dataset_id": dataset_id}) | ||
| for batch in tqdm.tqdm(batches): |
There was a problem hiding this comment.
Nastia, we had talked before about using two different versions of tqdm: one for command line and one for notebook. I tested the notebook use case by invoking the utils.py functions for an ipynb, and the standard tqdm renders fine. Take a look to double check this is the behavior you wanted, but my current view is that we should stick with just the basic version if it works.
There was a problem hiding this comment.
we still should use tqdm.notebook.tqdm: even though conventional tqdm.tqdm works the notebook version has better rendering in jupyter environment.
There was a problem hiding this comment.
I implemented this and WOW - you are right, so much better
references/utils.py
Outdated
| items[i : i + batch_size] for i in range(0, len(items), batch_size) | ||
| ] | ||
| agg_response = UploadResponse(json={"dataset_id": dataset_id}) | ||
| for batch in tqdm.tqdm(batches): |
There was a problem hiding this comment.
we still should use tqdm.notebook.tqdm: even though conventional tqdm.tqdm works the notebook version has better rendering in jupyter environment.
Added helper scripts in new references folder to give users an example for how to batch upload. The goal is to make it easier to debug uploading large payloads to Nucleus. Batching includes tqdm progress bar for feedback about estimated upload time.