Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

events: Add full api_path; rename Send #22487

Merged
merged 5 commits into from
Aug 23, 2023
Merged

Conversation

swenson
Copy link
Contributor

@swenson swenson commented Aug 21, 2023

Biggest change: we rename Send to SendEvent in logical.EventSender.. Initially we picked Send to match the underlying go-eventlogger broker's Send method, and to avoid the stuttering of events.SendEvent.

However, I think it is more useful for the logical.EventSender interface to use the method SendEvent so that, for example, framework.Backend can implement it.

This is a relatively change now that should not affect anything except the KV plugin, which is being fixed in another PR.

Another change: if the api_path metadata is present, then the plugin-aware EventBus will prepend it with the plugin mount. This allows the api_path to be the full path to any referenced secret.

This change is also backwards compatible, since this field was not present in the KV plugin. (It did use the slightly different path field, which we can keep for now.)

Tested with a KV plugin (hashicorp/vault-plugin-secrets-kv#124) modified to output the secret_path metadata:

2023-08-21T15:39:47.384-0700 [INFO]  events: Sending event: event="event
:{id:\"5989b10e-3356-bcf9-5f41-88d2ef699de2\" metadata:{fields:{key:\"cu
rrent_version\" value:{string_value:\"1\"}} fields:{key:\"oldest_version
\" value:{string_value:\"0\"}} fields:{key:\"operation\" value:{string_v
alue:\"data-write\"}} fields:{key:\"path\" value:{string_value:\"data/fo
o2\"}} fields:{key:\"secret_path\" value:{string_value:\"secret/foo2\"}}
}} event_type:\"kv-v2/data-write\" plugin_info:{mount_class:\"secret\" m
ount_accessor:\"kv_f52ab3c4\" mount_path:\"secret/\" plugin:\"kv\"}"

@swenson swenson added this to the 1.15 milestone Aug 21, 2023
@swenson swenson requested review from briankassouf, tomhjp, ncabatoff and a team August 21, 2023 22:43
@github-actions github-actions bot added the hashicorp-contributed-pr If the PR is HashiCorp (i.e. not-community) contributed label Aug 21, 2023
swenson pushed a commit to hashicorp/vault-plugin-secrets-kv that referenced this pull request Aug 21, 2023
This preserves the existing `path` entry in the metadata for
backwards compatibility.

The `secret_path` is the path to the secret in Vault.

The `operation` is what operation is being performed, which may
be plugin-dependent.

This depends on hashicorp/vault#22487
being merged and bumped in `go.mod` before this will compile
(since it references new constants in the SDK, though this could
be removed).
@github-actions
Copy link

Build Results:
All builds succeeded! ✅

@github-actions
Copy link

github-actions bot commented Aug 21, 2023

CI Results:
All Go tests succeeded! ✅

Copy link
Contributor

@tomhjp tomhjp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Left a few comments but nothing urgent

changelog/22487.txt Show resolved Hide resolved
sdk/logical/events.go Outdated Show resolved Hide resolved
sdk/logical/events.go Outdated Show resolved Hide resolved
sdk/logical/events.go Outdated Show resolved Hide resolved
data.Metadata.Fields[logical.EventMetadataSecretPath] == nil {
return data
}
data.Metadata.Fields[logical.EventMetadataSecretPath] = structpb.NewStringValue(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't feel like it matters a huge amount either way, but is it maybe a safer/better pattern to append to the event rather than mutate it? e.g. if the plugin reports an event with relative_path, we add secret_path with the mount path in front.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. I am okay either way?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you're not against it I am slightly inclined towards plugins setting relative_api_path and then Vault using that to populate api_path as it feels less confusing for plugin devs - that way the data is always accurate all the way through the pipeline. Whereas if we only use api_path the plugin is kind of filling in a half-truth until it gets corrected inside Vault.

vault/eventbus/bus.go Outdated Show resolved Hide resolved

// TestSecretPathIsPrependedWithMount tests that "secret_path", if present in the
// metadata, is prepended with the plugin's mount.
func TestSecretPathIsPrependedWithMount(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice test!

@tomhjp
Copy link
Contributor

tomhjp commented Aug 22, 2023

I just realised, in the PR description example, the secret_path doesn't look right: key:\"secret_path\" value:{string_value:\"secret/foo2\"}. As the event type is kv-v2, the secret_path should be secret/data/foo2. I think this must just be a copy-paste error from an earlier version or something though, because the code looks like it should produce the correct result?

// common event metadata keys
const (
EventMetadataSecretPath = "secret_path"
EventMetadataOperation = "operation"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to enforce some rules on this field because it should be tightly scoped to one of a few valid values. Could we define a type EventOperation string and enumerate all the valid options for this? And then validate them as we receive events in the core's event bus?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is better to leave metadata to be more free-form, so that plugins can define and use whatever fields they want. (Hence why we're using a generic structpb and not a fixed protobuf message.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My motivating use-case for this comment was the RFC discussion where our consumers need to know whether the secret data has changed and they therefore need to re-fetch it. I don't think this field satisfies that requirement unless we have a schema that they can rely on for interpreting events. I'm open to other solutions too though; off the top of my head:

  • Do we envisage emitting events for non-write events? If not, perhaps consumers should always assume underlying data change? Although I would worry a little bit about boxing ourselves in by committing to that implicit assumption.
  • A different, more tightly scoped field name like a boolean "data_changed", or "updated", or "refetch" etc.?

sdk/logical/events.go Outdated Show resolved Hide resolved
@swenson
Copy link
Contributor Author

swenson commented Aug 22, 2023

@tomhjp re: the secret/data/foo vs secret/foo, currently, it will product secret/foo for the secret_path.

KV expects you to prepend data/ and metadata/ to its API endpoints to say whether you are operating on data or metadata, but secret itself doesn't really include data/ or metadata/ in the path, right?

This is one of the reasons I left data/ and metadata/ in the path, but not the secret_path.

I guess a better question would be, is the point of secret_path (or api_path) to match the underlying storage, to match the API endpoint that might be called, or something else?

@swenson
Copy link
Contributor Author

swenson commented Aug 22, 2023

(Discussed with @tomhjp elsewhere, and it seems better to use api_path and include data/ and metadata/ in the KV events path.)

@swenson swenson changed the title events: Add full secret_path; rename Send events: Add full api_path; rename Send Aug 22, 2023
@swenson swenson force-pushed the vault-19140/full-secrets-path branch from 8a3e88a to 2a28102 Compare August 22, 2023 21:35
Copy link
Contributor

@tomhjp tomhjp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of minor ongoing discussion points, but still no blockers as I'm happy to follow up on those in a separate PR if preferred.

Comment on lines 16 to 20
// EventMetadataApiPath is used in event metadata to show the API path that can be used to fetch any underlying
// data. For example, the KV plugin would set this to `data/mysecret`. The event system will automatically prepend
// the plugin mount to this path, if present, so it would be come `secret/data/mysecret`, for example.
// If this is an auth plugin event, this will additionally be prepended with `auth/`.
EventMetadataApiPath = "api_path"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Go naming conventions

Suggested change
// EventMetadataApiPath is used in event metadata to show the API path that can be used to fetch any underlying
// data. For example, the KV plugin would set this to `data/mysecret`. The event system will automatically prepend
// the plugin mount to this path, if present, so it would be come `secret/data/mysecret`, for example.
// If this is an auth plugin event, this will additionally be prepended with `auth/`.
EventMetadataApiPath = "api_path"
// EventMetadataAPIPath is used in event metadata to show the API path that can be used to fetch any underlying
// data. For example, the KV plugin would set this to `data/mysecret`. The event system will automatically prepend
// the plugin mount to this path, if present, so it would be come `secret/data/mysecret`, for example.
// If this is an auth plugin event, this will additionally be prepended with `auth/`.
EventMetadataAPIPath = "api_path"

sdk/logical/events.go Show resolved Hide resolved
if err != nil {
return err
}
metadata := map[string]string{}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this change is in progress anyway as I saw the allocations comment was resolved: I think the same algorithm here will work with one less map allocated if we delete this map and use ev.Metadata.Fields directly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, oops, let me copy that over.

data.Metadata.Fields[logical.EventMetadataSecretPath] == nil {
return data
}
data.Metadata.Fields[logical.EventMetadataSecretPath] = structpb.NewStringValue(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you're not against it I am slightly inclined towards plugins setting relative_api_path and then Vault using that to populate api_path as it feels less confusing for plugin devs - that way the data is always accurate all the way through the pipeline. Whereas if we only use api_path the plugin is kind of filling in a half-truth until it gets corrected inside Vault.

// common event metadata keys
const (
EventMetadataSecretPath = "secret_path"
EventMetadataOperation = "operation"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My motivating use-case for this comment was the RFC discussion where our consumers need to know whether the secret data has changed and they therefore need to re-fetch it. I don't think this field satisfies that requirement unless we have a schema that they can rely on for interpreting events. I'm open to other solutions too though; off the top of my head:

  • Do we envisage emitting events for non-write events? If not, perhaps consumers should always assume underlying data change? Although I would worry a little bit about boxing ourselves in by committing to that implicit assumption.
  • A different, more tightly scoped field name like a boolean "data_changed", or "updated", or "refetch" etc.?

// data. For example, the KV plugin would set this to `data/mysecret`. The event system will automatically prepend
// the plugin mount to this path, if present, so it would be come `secret/data/mysecret`, for example.
// If this is an auth plugin event, this will additionally be prepended with `auth/`.
EventMetadataApiPath = "api_path"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the extra comment, I've just done the same again and spent some time on the KV PR after reviewing this.

For an example like deleting a KV secret (POST delete/foo), the ergonomics feel a little weird with this name. path and api_path are pretty synonymous from a plugin dev point of view, and whichever we use, it feels like delete/foo is the least surprising thing to put there. How about we keep path as-is for KV, and add data_path for the path at which consumers can fetch the associated sensitive data?

Suggested change
EventMetadataApiPath = "api_path"
EventMetadataApiPath = "data_path"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense to me.

@swenson
Copy link
Contributor Author

swenson commented Aug 23, 2023

@tomhjp there are definitely non-write events we might generate in the future, like lease expirations.

I think the operation might be good to keep, but we could add another field like modified = true.

@swenson swenson force-pushed the vault-19140/full-secrets-path branch from a006c16 to ebaf13a Compare August 23, 2023 17:40
Copy link
Contributor

@tomhjp tomhjp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 👍

Christopher Swenson added 5 commits August 23, 2023 14:16
Biggest change: we rename `Send` to `SendEvent` in `logical.EventSender`..
Initially we picked `Send` to match the underlying go-eventlogger
broker's `Send` method, and to avoid the stuttering of `events.SendEvent`.

However, I think it is more useful for the `logical.EventSender`
interface to use the method `SendEvent` so that, for example,
`framework.Backend` can implement it.

This is a relatively change now that should not affect anything
except the KV plugin, which is being fixed in another PR.

Another change: if the `secret_path` metadata is present, then
the plugin-aware `EventBus` will prepend it with the plugin mount.
This allows the `secret_path` to be the full path to any referenced
secret.

This change is also backwards compatible, since this field was not
present in the KV plugin. (It did use the slightly different `path`
field, which we can keep for now.)

Tested with a KV plugin modified to output the `secret_path` metadata:

```
2023-08-21T15:39:47.384-0700 [INFO]  events: Sending event: event="event
:{id:\"5989b10e-3356-bcf9-5f41-88d2ef699de2\" metadata:{fields:{key:\"cu
rrent_version\" value:{string_value:\"1\"}} fields:{key:\"oldest_version
\" value:{string_value:\"0\"}} fields:{key:\"operation\" value:{string_v
alue:\"data-write\"}} fields:{key:\"path\" value:{string_value:\"data/fo
o2\"}} fields:{key:\"secret_path\" value:{string_value:\"secret/foo2\"}}
}} event_type:\"kv-v2/data-write\" plugin_info:{mount_class:\"secret\" m
ount_accessor:\"kv_f52ab3c4\" mount_path:\"secret/\" plugin:\"kv\"}"
```
@swenson swenson force-pushed the vault-19140/full-secrets-path branch from ebaf13a to 90560d3 Compare August 23, 2023 21:16
@swenson
Copy link
Contributor Author

swenson commented Aug 23, 2023

Thanks!

@swenson swenson merged commit 925702d into main Aug 23, 2023
@swenson swenson deleted the vault-19140/full-secrets-path branch August 23, 2023 22:11
swenson pushed a commit to hashicorp/vault-plugin-secrets-kv that referenced this pull request Aug 23, 2023
This preserves the existing `path` entry in the metadata for
backwards compatibility.

The `data_path` is the path to the secret in Vault.

The `operation` is what operation is being performed, which may
be plugin-dependent.

This depends on hashicorp/vault#22487
being merged and bumped in `go.mod` before this will compile
(since it references new constants in the SDK, though this could
be removed).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hashicorp-contributed-pr If the PR is HashiCorp (i.e. not-community) contributed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants