feat: make storage purging default, add --resurrect#729
Conversation
| // TODO: deprecate these flags | ||
| if (this.flags.purgeQueue && !this.flags.purge) { | ||
| await purgeDefaultQueue(); | ||
| info({ message: 'Default local request queue was purged.' }); | ||
| } | ||
|
|
||
| if (this.flags.purgeDataset && !this.flags.purge) { | ||
| await purgeDefaultDataset(); | ||
| info({ message: 'Default local dataset was purged.' }); | ||
| } | ||
|
|
||
| if (this.flags.purgeKeyValueStore && !this.flags.purge) { | ||
| await purgeDefaultKeyValueStore(); | ||
| info({ message: 'Default local key-value store was purged.' }); | ||
| } |
There was a problem hiding this comment.
let's hope this won't be missed by anyone 🙃 worst case we can add it back
btw I hope oclif wont silently ignore undeclared flags?
There was a problem hiding this comment.
It will throw for unknown flags... Do we want to define them as hidden flags so it doesn't throw for users?
There was a problem hiding this comment.
that's good, if they are not doing anything we need to tell potential users
|
Hi guys, I've just realized that if we want the CLI to behave consistently with how the Apify platform works, we shouldn't purge entirely by default, but instead the CLI should just keep adding local runs for later introspection and comparison. I know you're making it on par with Crawlee's behavior, but I think this will introduce further divergence. |
|
I am not sure about that, regular actor developer runs, runs, and runs. They would end up with a lot of junk locally, I am not sure people would like such a default. Also, I am not sure if this is a "divergence", we are not doing that now either, the CLI never considered multiple storage folders, neither the actual consumers of the storage folder (crawlee/sdk). We could surely just rename the existing storage folder instead of wiping it, but should that really be a default? |
|
@B4nan Understood. Divergence from the platform behavior is what I'm concerned about. You run it there and the results always persist by default under a new run. But unifying the behavior of our local tooling is definitely a way to go. |
|
@vladfrangu lets describe the BCs in the PR description |
|
nit: should be |
|
also i double checked with others, python works the same as JS, auto purge for crawlee or SDK (even without crawlee) v2+ |
--resurrect
d8bd43b to
f0b5fe8
Compare
Closes #590
BREAKING CHANGE:
Purging in projects using Crawlee for SDK v3+ now purge by default, so using
--purgeflag is no longer necessary. Use--no-purgeto keep the storage folder intact.The
purge-queue,purge-datasetandpurge-key-value-storeflags have been removed, and the logic of all three was combined into thepurgeflag.