aws_bedrock: retain contextualGroundingPolicy check details by efd6 · Pull Request #11890 · elastic/integrations

efd6 · 2024-11-27T08:22:43Z

Proposed commit message

See title.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes [Amazon Bedrock]: Amazon Bedrock Guardrail Contextual grounding check is not mapped. #11809

Screenshots

elasticmachine · 2024-11-27T08:55:44Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

shashank-elastic

Filed mapping correction looks good

chrisberkhout

It looks like we should be looking at body.trace.guardrail.outputAssessments in general, since we are looking at body.amazon_bedrock_trace.guardrail.outputs. I probably should have done that in my last change, which started looking at outputs.

If that's added, then the other change is to expand which details objects get added to gen_ai.policy.match_detail.

What do you think about this version? The test output is slightly different.

--- a/packages/aws_bedrock/data_stream/invocation/elasticsearch/ingest_pipeline/default.yml
+++ b/packages/aws_bedrock/data_stream/invocation/elasticsearch/ingest_pipeline/default.yml
@@ -170,6 +170,7 @@ processors:
         "topicPolicy": "topic_policy"
         "customWords": "custom_words"
         "sensitiveInformationPolicy": "sensitive_information_policy"
+        "contextualGroundingPolicy": "contextual_grounding_policy"
         "invocationMetrics": "invocation_metrics"
         "guardrailCoverage": "guardrail_coverage"
         "textCharacters": "text_characters"
@@ -356,6 +357,7 @@ processors:
           )
           .flatMap(body -> [
             [body.trace?.guardrail?.inputAssessment],
+            body.trace?.guardrail?.outputAssessments,
             [body.amazon_bedrock_trace?.guardrail?.input],
             body.amazon_bedrock_trace?.guardrail?.outputs
           ].stream())
@@ -404,7 +406,7 @@ processors:
           .collect(Collectors.toList());
 
         ctx.gen_ai.policy.match_detail = nameAndDetailsOfPolicies.stream()
-          .filter(policy -> policy.details.match != null)
+          .filter(policy -> policy.details.match != null || policy.name == 'contextual_grounding_policy')
           .map(policy -> policy.details)
           .distinct()  // no sort because HashMap isn't Comparable
           .collect(Collectors.toList());

efd6 · 2024-11-27T21:35:14Z

Yeah, I think that's good. The changes that I see in the test expectation actually fix concerns that I had with the current behaviour, so that is nice. Will wait for @shashank-elastic's input though.

elastic-vault-github-plugin-prod · 2024-11-27T21:56:16Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

shashank-elastic · 2024-11-28T02:17:32Z

Yeah, I think that's good. The changes that I see in the test expectation actually fix concerns that I had with the current behaviour, so that is nice. Will wait for @shashank-elastic's input though.

@efd6 The filed mapping for outputassesments for contextualGroundingPolicy look good!

chrisberkhout · 2024-11-28T09:08:05Z

My diff was on main version of this file, so this part would just be:

Suggested change

ctx.gen_ai.policy.match_detail = Stream.concat(

nameAndDetailsOfPolicies.stream()

.filter(policy -> policy.details.match != null || policy.name == 'contextual_grounding_policy')

.map(policy -> policy.details),

allBodies.stream()

.filter(body -> body.trace?.guardrail?.outputAssessments instanceof List)

.flatMap(body -> body.trace.guardrail.outputAssessments.stream())

.filter(output -> output instanceof HashMap)

.flatMap(output -> output.entrySet().stream())

.map(assessment -> assessment.getValue())

.filter(details -> details.contextualGroundingPolicy?.filters instanceof List)

.flatMap(details -> details.contextualGroundingPolicy.filters.stream())

)

.distinct() // no sort because HashMap isn't Comparable

.collect(Collectors.toList());

ctx.gen_ai.policy.match_detail = nameAndDetailsOfPolicies.stream()

.filter(policy -> policy.details.match != null || policy.name == 'contextual_grounding_policy')

.map(policy -> policy.details)

.distinct() // no sort because HashMap isn't Comparable

.collect(Collectors.toList());

Currently the 2nd part of the concat generates nothing because it uses the camel case name contextualGroundingPolicy.

Thanks for clarifying, I missed that. Much nicer.

elasticmachine · 2024-11-28T10:40:18Z

💚 Build Succeeded

Buildkite Build
Commit: 188311dbd563de93940cbfb88e69e3478a5c6134

History

💚 Build #18811 succeeded 687d504f666b086695ea82273f4c12baea22db59
💚 Build #18755 succeeded be801ece6d962ed832afe3264a95f4713cbc912c

cc @efd6

elastic-sonarqube · 2024-11-28T10:40:25Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elastic-vault-github-plugin-prod · 2024-11-28T10:58:53Z

Package aws_bedrock - 0.15.0 containing this change is available at https://epr.elastic.co/package/aws_bedrock/0.15.0/

…11890)

efd6 added enhancement New feature or request Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Integration:aws_bedrock Amazon Bedrock labels Nov 27, 2024

efd6 self-assigned this Nov 27, 2024

efd6 force-pushed the 11809-aws_bedrock branch from 42c9b10 to be801ec Compare November 27, 2024 08:23

efd6 marked this pull request as ready for review November 27, 2024 08:55

efd6 requested a review from a team as a code owner November 27, 2024 08:55

efd6 requested a review from shashank-elastic November 27, 2024 08:56

shashank-elastic approved these changes Nov 27, 2024

View reviewed changes

chrisberkhout reviewed Nov 27, 2024

View reviewed changes

efd6 force-pushed the 11809-aws_bedrock branch from be801ec to 5525432 Compare November 27, 2024 21:30

efd6 requested a review from shashank-elastic November 27, 2024 21:36

chrisberkhout requested changes Nov 28, 2024

View reviewed changes

efd6 requested a review from chrisberkhout November 28, 2024 10:15

chrisberkhout approved these changes Nov 28, 2024

View reviewed changes

efd6 merged commit 072343f into elastic:main Nov 28, 2024

harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025

aws_bedrock: retain contextualGroundingPolicy check details (elastic#…

130d8a6

…11890)

harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025

aws_bedrock: retain contextualGroundingPolicy check details (elastic#…

95740be

…11890)

efd6 deleted the 11809-aws_bedrock branch February 5, 2025 21:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws_bedrock: retain contextualGroundingPolicy check details#11890

aws_bedrock: retain contextualGroundingPolicy check details#11890
efd6 merged 0 commit intoelastic:mainfrom
efd6:11809-aws_bedrock

efd6 commented Nov 27, 2024

Uh oh!

elasticmachine commented Nov 27, 2024

Uh oh!

shashank-elastic left a comment

Uh oh!

chrisberkhout left a comment

Uh oh!

efd6 commented Nov 27, 2024

Uh oh!

elastic-vault-github-plugin-prod Bot commented Nov 27, 2024

Uh oh!

shashank-elastic commented Nov 28, 2024

Uh oh!

chrisberkhout Nov 28, 2024

Uh oh!

efd6 Nov 28, 2024

Uh oh!

elasticmachine commented Nov 28, 2024

Uh oh!

elastic-sonarqube Bot commented Nov 28, 2024

Uh oh!

elastic-vault-github-plugin-prod Bot commented Nov 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

-        ctx.gen_ai.policy.match_detail = Stream.concat(
-          nameAndDetailsOfPolicies.stream()
-            .filter(policy -> policy.details.match != null || policy.name == 'contextual_grounding_policy')
-            .map(policy -> policy.details),
-          allBodies.stream()
-            .filter(body -> body.trace?.guardrail?.outputAssessments instanceof List)
-            .flatMap(body -> body.trace.guardrail.outputAssessments.stream())
-            .filter(output -> output instanceof HashMap)
-            .flatMap(output -> output.entrySet().stream())
-            .map(assessment -> assessment.getValue())
-            .filter(details -> details.contextualGroundingPolicy?.filters instanceof List)
-            .flatMap(details -> details.contextualGroundingPolicy.filters.stream())
-        )
-        .distinct()  // no sort because HashMap isn't Comparable
-        .collect(Collectors.toList());
+        ctx.gen_ai.policy.match_detail = nameAndDetailsOfPolicies.stream()
+          .filter(policy -> policy.details.match != null || policy.name == 'contextual_grounding_policy')
+          .map(policy -> policy.details)
+          .distinct()  // no sort because HashMap isn't Comparable
+          .collect(Collectors.toList());

Conversation

efd6 commented Nov 27, 2024

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Nov 27, 2024

Uh oh!

shashank-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

chrisberkhout left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 commented Nov 27, 2024

Uh oh!

elastic-vault-github-plugin-prod Bot commented Nov 27, 2024

🚀 Benchmarks report

Uh oh!

shashank-elastic commented Nov 28, 2024

Uh oh!

chrisberkhout Nov 28, 2024

Choose a reason for hiding this comment

Uh oh!

efd6 Nov 28, 2024

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Nov 28, 2024

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube Bot commented Nov 28, 2024

Quality Gate passed

Uh oh!

elastic-vault-github-plugin-prod Bot commented Nov 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants