Skip to content

feat: add signature to ZIP tarball URL stored in KV store #875

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vladfrangu merged 5 commits into from
Aug 12, 2025
Merged

feat: add signature to ZIP tarball URL stored in KV store #875

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3b96703
feat: add signature to ZIP tarball URL stored in KV store
danpoletaev Aug 6, 2025
02c8db6
refactor: clean up
danpoletaev Aug 11, 2025
8d44b4f
Merge branch 'master' into feat/sign-tarball-kv-record-url
danpoletaev Aug 11, 2025
f00174f
refactor: clean up
danpoletaev Aug 11, 2025
cba48f9
fix: cast as ApifyClient
vladfrangu Aug 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@
"@apify/actor-templates": "^0.1.5",
"@apify/consts": "^2.36.0",
"@apify/input_schema": "^3.17.0",
"@apify/utilities": "^2.15.1",
"@apify/utilities": "^2.18.0",
"@crawlee/memory-storage": "^3.12.0",
"@inquirer/core": "^10.1.15",
"@inquirer/input": "^4.2.1",
Expand All @@ -79,7 +79,7 @@
"@skyra/jaro-winkler": "^1.1.1",
"adm-zip": "~0.5.15",
"ajv": "~8.17.1",
"apify-client": "^2.12.6",
"apify-client": "^2.14.0",
"archiver": "~7.0.1",
"axios": "^1.11.0",
"chalk": "~5.5.0",
Expand Down
18 changes: 5 additions & 13 deletions src/commands/actor/get-public-url.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import type { ApifyClient } from 'apify-client';

import { ACTOR_ENV_VARS, APIFY_ENV_VARS } from '@apify/consts';
import { createHmacSignature } from '@apify/utilities';

import { getApifyStorageClient } from '../../lib/actor.js';
import { ApifyCommand } from '../../lib/command-framework/apify-command.js';
Expand Down Expand Up @@ -27,8 +28,6 @@ export class ActorGetPublicUrlCommand extends ApifyCommand<typeof ActorGetPublic
process.exitCode = CommandExitCodes.NotImplemented;
return;
}

const apiBase = process.env[APIFY_ENV_VARS.API_PUBLIC_BASE_URL];
const storeId = process.env[ACTOR_ENV_VARS.DEFAULT_KEY_VALUE_STORE_ID];

// This should never happen, but handle it gracefully to prevent crashes.
Expand All @@ -40,11 +39,9 @@ export class ActorGetPublicUrlCommand extends ApifyCommand<typeof ActorGetPublic
return;
}

const apifyClient = await getApifyStorageClient();
const apifyClient = (await getApifyStorageClient()) as ApifyClient;
const store = await apifyClient.keyValueStore(storeId).get();

const publicUrl = new URL(`${apiBase}/v2/key-value-stores/${storeId}/records/${key}`);

if (!store) {
error({
message: `Key-Value store with ID '${storeId}' was not found. Ensure the store exists and that the correct ID is set in ${ACTOR_ENV_VARS.DEFAULT_KEY_VALUE_STORE_ID}.`,
Expand All @@ -53,13 +50,8 @@ export class ActorGetPublicUrlCommand extends ApifyCommand<typeof ActorGetPublic
return;
}

// @ts-expect-error Add types to client
const { urlSigningSecretKey } = store;

if (urlSigningSecretKey) {
publicUrl.searchParams.append('signature', createHmacSignature(urlSigningSecretKey as string, key));
}
const publicTarballUrl = await apifyClient.keyValueStore(storeId).getRecordPublicUrl(key);

console.log(publicUrl.toString());
console.log(publicTarballUrl);
}
}
20 changes: 19 additions & 1 deletion src/commands/actors/push.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import open from 'open';

import { fetchManifest } from '@apify/actor-templates';
import { ACTOR_JOB_STATUSES, ACTOR_SOURCE_TYPES, MAX_MULTIFILE_BYTES } from '@apify/consts';
import { createHmacSignature } from '@apify/utilities';

import { ApifyCommand } from '../../lib/command-framework/apify-command.js';
import { Args } from '../../lib/command-framework/args.js';
Expand Down Expand Up @@ -255,7 +256,24 @@ Skipping push. Use --force to override.`,
contentType: 'application/zip',
});
unlinkSync(TEMP_ZIP_FILE_NAME);
tarballUrl = `${apifyClient.baseUrl}/key-value-stores/${store.id}/records/${key}?disableRedirect=true`;
const tempTarballUrl = new URL(
`${apifyClient.baseUrl}/key-value-stores/${store.id}/records/${key}?disableRedirect=true`,
);

/**
* Signs the tarball URL to grant temporary access for restricted resources.
* When a store is set to 'RESTRICTED', direct URLs are disabled. Instead of
* appending a security token, we add a signature to the URL parameters.
* https://github.com/apify/apify-core/issues/22197
*
* TODO: Use keyValueStore(:storeId).getRecordPublicUrl from apify-client instead once it is released.
*/
if (store?.urlSigningSecretKey) {
const signature = createHmacSignature(store.urlSigningSecretKey, key);
tempTarballUrl.searchParams.set('signature', signature);
}

tarballUrl = tempTarballUrl.toString();
Comment on lines +259 to +276
Copy link
Member

@vladfrangu vladfrangu Aug 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this use the same getRecordPublicUrl?

Copy link
Contributor Author

@danpoletaev danpoletaev Aug 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd leave it as it is! It helps to make sure, that the URL has correct signature.

If for some reason getRecordPublicUrl will return URL without signature, we'll catch it in the test

sourceType = ACTOR_SOURCE_TYPES.TARBALL;
}

Expand Down
20 changes: 17 additions & 3 deletions test/api/commands/push.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@ import { mkdir, writeFile } from 'node:fs/promises';

import type { ActorCollectionCreateOptions } from 'apify-client';

import { ACTOR_SOURCE_TYPES, SOURCE_FILE_FORMATS } from '@apify/consts';
import { ACTOR_SOURCE_TYPES, SOURCE_FILE_FORMATS, STORAGE_GENERAL_ACCESS } from '@apify/consts';
import { createHmacSignature } from '@apify/utilities';

import { testRunCommand } from '../../../src/lib/command-framework/apify-command.js';
import { LOCAL_CONFIG_PATH } from '../../../src/lib/consts.js';
Expand Down Expand Up @@ -236,7 +237,20 @@ describe('[api] apify push', () => {
testActor = (await testActorClient.get())!;
const testActorVersion = await testActorClient.version(actorJson.version).get();
const store = await testUserClient.keyValueStores().getOrCreate(`actor-${testActor.id}-source`);
await testUserClient.keyValueStore(store.id).delete(); // We just needed the store ID, we can clean up now

// Update the store's general access to RESTRICTED
await testUserClient.keyValueStore(store.id).update({ generalAccess: STORAGE_GENERAL_ACCESS.RESTRICTED });

expect(store.urlSigningSecretKey).toBeDefined();
const signature = createHmacSignature(store.urlSigningSecretKey!, `version-${actorJson.version}.zip`);
Copy link
Contributor

@tobice tobice Aug 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(opt) I'm never fond of when tests simply replicate the code being tested 😄 It's not much of a test...

Is this running against actual Apify?

If so, I'd suggest to:

  1. Send a request to restrict access on the key-value store.
  2. Test that the tarballUrl can be downloaded.

Copy link
Member

@vladfrangu vladfrangu Aug 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this running against actual Apify?

Yes, yes it is!

Copy link
Contributor Author

@danpoletaev danpoletaev Aug 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactored the test


// Check if the tarball URL is accessible
await expect(fetch((testActorVersion as { tarballUrl: string }).tarballUrl)).resolves.toHaveProperty(
'status',
200,
);

await testUserClient.keyValueStore(store.id).delete();

if (testActor) await testActorClient.delete();

Expand All @@ -245,7 +259,7 @@ describe('[api] apify push', () => {
buildTag: 'latest',
tarballUrl:
`${testActorClient.baseUrl}/key-value-stores/${store.id}` +
`/records/version-${actorJson.version}.zip?disableRedirect=true`,
`/records/version-${actorJson.version}.zip?disableRedirect=true&signature=${signature}`,
envVars: testActorWithEnvVars.versions[0].envVars,
sourceType: ACTOR_SOURCE_TYPES.TARBALL,
});
Expand Down
100 changes: 50 additions & 50 deletions yarn.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ __metadata:
languageName: node
linkType: hard

"@apify/consts@npm:^2.20.0, @apify/consts@npm:^2.23.0, @apify/consts@npm:^2.25.0, @apify/consts@npm:^2.36.0, @apify/consts@npm:^2.44.0":
version: 2.44.0
resolution: "@apify/consts@npm:2.44.0"
checksum: 10c0/3b6c1baf7c02b59b5ab04a73c66732c392ed5e96afcd79dd71c42b1c97a61857bc5656d9d312cced0ec4541c43a526148abafe1095b316a1c004bd5f95722999
"@apify/consts@npm:^2.20.0, @apify/consts@npm:^2.23.0, @apify/consts@npm:^2.25.0, @apify/consts@npm:^2.36.0, @apify/consts@npm:^2.44.1":
version: 2.44.1
resolution: "@apify/consts@npm:2.44.1"
checksum: 10c0/3d6c5df96d9338f51acd3bf0063f92f21030d4a79e40653d949c12f9c4077366d087505ffb5859199c492f31f93538566961471a96610b756fbbb0a5519b6ad7
languageName: node
linkType: hard

Expand Down Expand Up @@ -49,37 +49,37 @@ __metadata:
linkType: hard

"@apify/input_schema@npm:^3.17.0":
version: 3.18.3
resolution: "@apify/input_schema@npm:3.18.3"
version: 3.18.4
resolution: "@apify/input_schema@npm:3.18.4"
dependencies:
"@apify/consts": "npm:^2.44.0"
"@apify/input_secrets": "npm:^1.2.4"
"@apify/consts": "npm:^2.44.1"
"@apify/input_secrets": "npm:^1.2.5"
acorn-loose: "npm:^8.4.0"
countries-list: "npm:^3.0.0"
peerDependencies:
ajv: ^8.0.0
checksum: 10c0/6c8a7acd60007baeece5e7149585e193ecdab49c98436ad54e578be4e7adbb0db428d9dda28b492af570f4ed8e70690ccb8bca1fe460e24443cfe4a812940b1d
checksum: 10c0/6d2b0e3aa38a553a26db2fc389354e5ec130243eabcb21050a65183fd1ba3b003ec271d6fd5806287ff84adb79ce3559ec188fea245f7c3bbda50d80ae449b88
languageName: node
linkType: hard

"@apify/input_secrets@npm:^1.2.0, @apify/input_secrets@npm:^1.2.4":
version: 1.2.4
resolution: "@apify/input_secrets@npm:1.2.4"
"@apify/input_secrets@npm:^1.2.0, @apify/input_secrets@npm:^1.2.5":
version: 1.2.5
resolution: "@apify/input_secrets@npm:1.2.5"
dependencies:
"@apify/log": "npm:^2.5.21"
"@apify/utilities": "npm:^2.18.1"
"@apify/log": "npm:^2.5.22"
"@apify/utilities": "npm:^2.18.2"
ow: "npm:^0.28.2"
checksum: 10c0/6a71ea85c6b9cabf2b29f0ab2e3acd42bb9b9eac8371699dc7ed3d706ae4204de97100fd764800fa038ea2439a8ff422e3dff214edf9c8db2cf8d8cf2ec29ffe
checksum: 10c0/6ce482850140dc25cd6a5c6f0c76c33c168686b8f6470aa8efb7557f2c5b42ab8cb2ab9d0e503bbc9e9d9c3df6325cf66886c33cb2b776e5061e545c85df0b4e
languageName: node
linkType: hard

"@apify/log@npm:^2.2.6, @apify/log@npm:^2.4.0, @apify/log@npm:^2.4.3, @apify/log@npm:^2.5.21":
version: 2.5.21
resolution: "@apify/log@npm:2.5.21"
"@apify/log@npm:^2.2.6, @apify/log@npm:^2.4.0, @apify/log@npm:^2.4.3, @apify/log@npm:^2.5.22":
version: 2.5.22
resolution: "@apify/log@npm:2.5.22"
dependencies:
"@apify/consts": "npm:^2.44.0"
"@apify/consts": "npm:^2.44.1"
ansi-colors: "npm:^4.1.1"
checksum: 10c0/6115c9bfc36bcc2ba5812cec66fd8c8a5e62e7d4177460e41cd68d92da1704def067359c715f39de5c4238f220ba6ec7caeb402467106b7d116789d872fbdb71
checksum: 10c0/10ba3b00b1b57fb49adfd6800e5d2fc46c471eb442510aedf47edb303fedde9164e527022e0f3bd5f78d1778025084d13145e45d15aa1955afe086d6b2bbcf84
languageName: node
linkType: hard

Expand All @@ -95,11 +95,11 @@ __metadata:
linkType: hard

"@apify/pseudo_url@npm:^2.0.30":
version: 2.0.62
resolution: "@apify/pseudo_url@npm:2.0.62"
version: 2.0.63
resolution: "@apify/pseudo_url@npm:2.0.63"
dependencies:
"@apify/log": "npm:^2.5.21"
checksum: 10c0/d0919998055781f4304794db4973fd00d4a7f47fa8fc94ad39c087d7e0901a547722f0b7f9081dd5da0d57af7129514c09b9f917bab62dd55a9f46df2d7818c2
"@apify/log": "npm:^2.5.22"
checksum: 10c0/fe4170e099db2af5ce6e6ebc3fba31d1838c5bceb6bd239eb93b50d87a5e2fbf36959200518a030e97ff2627d7c4784f5ae9216fe56e6519fdf0f0c2c1df7edf
languageName: node
linkType: hard

Expand All @@ -117,13 +117,13 @@ __metadata:
languageName: node
linkType: hard

"@apify/utilities@npm:^2.13.0, @apify/utilities@npm:^2.15.1, @apify/utilities@npm:^2.18.0, @apify/utilities@npm:^2.18.1, @apify/utilities@npm:^2.7.10":
version: 2.18.1
resolution: "@apify/utilities@npm:2.18.1"
"@apify/utilities@npm:^2.13.0, @apify/utilities@npm:^2.18.0, @apify/utilities@npm:^2.18.2, @apify/utilities@npm:^2.7.10":
version: 2.18.2
resolution: "@apify/utilities@npm:2.18.2"
dependencies:
"@apify/consts": "npm:^2.44.0"
"@apify/log": "npm:^2.5.21"
checksum: 10c0/082f4ab78d38478e9a3c7c3bc876196b3bbfae863c25f0c643e933ddbc9637c9f58155d5208f328d36011d2e7507393e33f77c3784a760b691a9d8230877f246
"@apify/consts": "npm:^2.44.1"
"@apify/log": "npm:^2.5.22"
checksum: 10c0/1f3d24ca9470dbf3bfb3539daf7e58cbd36debf18b4d9e4ff640b4b692b7435492a84c735bc082bbc6d28a96750251e9846c49ea4553cc0f90c610605411ebeb
languageName: node
linkType: hard

Expand Down Expand Up @@ -500,14 +500,14 @@ __metadata:
linkType: hard

"@cucumber/query@npm:^13.0.2":
version: 13.5.0
resolution: "@cucumber/query@npm:13.5.0"
version: 13.6.0
resolution: "@cucumber/query@npm:13.6.0"
dependencies:
"@teppeis/multimaps": "npm:3.0.0"
lodash.sortby: "npm:^4.7.0"
peerDependencies:
"@cucumber/messages": "*"
checksum: 10c0/af8ba3491fd5de38aab964578e7b744c9c02f03c0f2a2101c80da4d628160b6c1e6fd8f1edcec30c3200bb9b1d1074a38e8b420b58fda873a7e66e9b032f3479
checksum: 10c0/a8df203c590bdd2da3c86b10abbe7db0da0391e1920eeb742fad944c8b42e5af1d2f93f99443b8ae7949698b29c340558e9e2709d88bfc71767bafe049fc3141
languageName: node
linkType: hard

Expand Down Expand Up @@ -1374,11 +1374,11 @@ __metadata:
linkType: hard

"@types/bun@npm:^1.2.5":
version: 1.2.19
resolution: "@types/bun@npm:1.2.19"
version: 1.2.20
resolution: "@types/bun@npm:1.2.20"
dependencies:
bun-types: "npm:1.2.19"
checksum: 10c0/c5bdc7c25fd7d33fe9b27f0065c9f0386e8ca61bb141e3d3f1b799c76a71ac2c3a0984b10999d092af5809ea67434f7b5fab97942d5630a07cbff0ae9c58b49a
bun-types: "npm:1.2.20"
checksum: 10c0/62972bde7496b804ae166cbe859579b4770a8e9d44bb87d5ae38e232c7b42a98e1acc93a8d6533449385a7c8daf437d71a7d0cd80b42e5035ff21563befbb2e7
languageName: node
linkType: hard

Expand Down Expand Up @@ -2237,7 +2237,7 @@ __metadata:
"@apify/eslint-config": "npm:^1.0.0"
"@apify/input_schema": "npm:^3.17.0"
"@apify/tsconfig": "npm:^0.1.1"
"@apify/utilities": "npm:^2.15.1"
"@apify/utilities": "npm:^2.18.0"
"@biomejs/biome": "npm:^2.0.0"
"@crawlee/memory-storage": "npm:^3.12.0"
"@crawlee/types": "npm:^3.11.1"
Expand Down Expand Up @@ -2271,7 +2271,7 @@ __metadata:
adm-zip: "npm:~0.5.15"
ajv: "npm:~8.17.1"
apify: "npm:^3.2.4"
apify-client: "npm:^2.12.6"
apify-client: "npm:^2.14.0"
archiver: "npm:~7.0.1"
axios: "npm:^1.11.0"
chai: "npm:^4.4.1"
Expand Down Expand Up @@ -2321,9 +2321,9 @@ __metadata:
languageName: unknown
linkType: soft

"apify-client@npm:^2.12.1, apify-client@npm:^2.12.6":
version: 2.13.0
resolution: "apify-client@npm:2.13.0"
"apify-client@npm:^2.12.1, apify-client@npm:^2.14.0":
version: 2.14.0
resolution: "apify-client@npm:2.14.0"
dependencies:
"@apify/consts": "npm:^2.25.0"
"@apify/log": "npm:^2.2.6"
Expand All @@ -2336,7 +2336,7 @@ __metadata:
ow: "npm:^0.28.2"
tslib: "npm:^2.5.0"
type-fest: "npm:^4.0.0"
checksum: 10c0/0596ba6aa4a534fc514e4d32b5ed232f1be7fd49cd3759380332edd136f5697add34a9e2ff3f8e98088f9945faf64ea5a7ebe53eb3d7b3886333ab3eddddfa32
checksum: 10c0/05eec1e28baa5ff8c8b4273e2671c8075aee1f53b40dad2d00d9d6ac96fdda2e1fe9a62708efa7f8bfc89c09719108822c71dd20ff0e44d5339c35a1ece37578
languageName: node
linkType: hard

Expand Down Expand Up @@ -2690,14 +2690,14 @@ __metadata:
languageName: node
linkType: hard

"bun-types@npm:1.2.19":
version: 1.2.19
resolution: "bun-types@npm:1.2.19"
"bun-types@npm:1.2.20":
version: 1.2.20
resolution: "bun-types@npm:1.2.20"
dependencies:
"@types/node": "npm:*"
peerDependencies:
"@types/react": ^19
checksum: 10c0/5475ad5c3f224244115c98da1329bc2ce700884d589eac487eaa111599361ed742c1041eefc0af1a8065dc9f722a6cbfaafe70aa049fcba1c3136f8ec61c4f52
checksum: 10c0/cea622d818d296df77a9a4c88b9b01e35953b0c82a816d3d64b139aa42b3948841b2cd696a08422ea44cc5b71b92385712736083d973159c938600cb369b8ac4
languageName: node
linkType: hard

Expand Down Expand Up @@ -2853,9 +2853,9 @@ __metadata:
linkType: hard

"caniuse-lite@npm:^1.0.30001733":
version: 1.0.30001733
resolution: "caniuse-lite@npm:1.0.30001733"
checksum: 10c0/2c03ad3362be7c93c09537f3853156ade5c70fb131888971dd538631971873ba27b83c5aad48dd06b6cde005fe57caae2db5d0f467d0c63a315391add70f01f5
version: 1.0.30001734
resolution: "caniuse-lite@npm:1.0.30001734"
checksum: 10c0/5869cb6a01e7a012a8c5d7b0482e2c910be3a2a469d4ef516a54db3f846fbaedb2600eeaa270dae9e2ad9328e33f39782e6f459405fcca620021f5f06694542d
languageName: node
linkType: hard

Expand Down
Loading