[O365_metrics]Split and refactor other data streams.

[ritalwar]

Proposed commit message

This PR includes the following updates:

1. Splits the mailbox_usage datastream into two separate datastreams: mailbox_usage_detail and mailbox_usage_quota_status.
2. Renames the outlook_app_usage datastream to outlook_app_usage_version_counts.
3. Renames the yammer_device_usage datastream to viva_engage_device_usage_user_counts.
4. Refactors datastreams to use report_date as the timestamp when available; otherwise, defaults timestamp to report_refresh_date.
5. Adds fingerprinting over dimension fields to handle duplicates.
6. Refactors datastreams to explicitly convert fields to long in the pipeline and makes related adjustments.
7. Updates the Codeowner file.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

Screenshots

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package o365_metrics 👍(8) 💚(4) 💔(2)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
viva_engage_groups_activity_group_detail	5952.38	4016.06	-1936.32 (-32.53%)	💔
outlook_activity	13888.89	6993.01	-6895.88 (-49.65%)	💔

To see the full report comment with /test benchmark fullreport

[ishleenk17]

IIRC, didn't we decide this interval as 24 hrs ?

[ritalwar]

Yes, but we noticed that reports occasionally get refreshed daily, so to prevent any data loss, we decided to call the API twice a day.

[ishleenk17]

There is a lot of convert processor usage in the pipelines. Which makes it hard to read.
Is it possible to do this ia script in painless?

[ritalwar]

I understand, but since not all fields need conversion and the selection is conditional, applying the convert processor selectively was the most feasible approach for now. Using Painless would require more complex logic, but we can definitely explore it for future optimizations.

[muthu-mps]

@ritalwar - When changing the name of the data stream, Is there a reason to keep outlook_app_usage_version_counts instead of outlook_app_usage_versions_user_counts as per the documentation here? As this is versions user count why we are not including the user in the data stream name?

[ritalwar]

@ritalwar - When changing the name of the data stream, Is there a reason to keep outlook_app_usage_version_counts instead of outlook_app_usage_versions_user_counts as per the documentation here? As this is versions user count why we are not including the user in the data stream name?

I understand your concern. While we split the datastreams and provided detailed names based on the reports, we also aimed to keep them concise to avoid overly long names. Where possible, we shortened them without losing essential information. The description clearly specifies what each datastream fetches. Similarly, we named data stream mailbox_usage_quota_status instead of mailbox_usage_quota_status_mailbox_counts to keep it clear yet manageable.

[muthu-mps]

• For the users who has already installed 0.3.0 might have a different data stream(Viva engage) for yammer device usage data stream while upgrading to newer version. I hope this is okay as the integration is still in Beta.
• Change looks good to me otherwise!

[elastic-sonarqube]

Quality Gate failed

Failed conditions
68.2% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: dda05c32b46a31e571897bdb46ca990d79eda169

History

• 💚 Build #21164 succeeded 6a73fb8abe8126603f86fd213c0f29601dc812c2
• 💚 Build #21098 succeeded 3cfacf50b61b0f0e749e18ce632642b8e1a0f424
• 💔 Build #21095 failed 52df317fb8ac56d189bcab6316f6c10ab9ce29b6
• 💔 Build #21092 failed eee6b879726d8e718c77c19a836276f862de4317
• 💔 Build #21064 failed 17efe2a9b898ea63e7ec260fe763a08b7d53cd48
• 💔 Build #21056 failed b9974d80ade7889fdd8ef44d7b9e1e0e993cfe17

cc @ritalwar

[ritalwar]

• For the users who has already installed 0.3.0 might have a different data stream(Viva engage) for yammer device usage data stream while upgrading to newer version. I hope this is okay as the integration is still in Beta.

Yes, since it's in beta, it's always recommended to use the latest version. Given that version 0.4.0 includes some fixes also as mentioned in the changelog, they should update to it.

[ishleenk17]

Looks good!

[elastic-vault-github-plugin-prod]

Package o365_metrics - 0.4.0 containing this change is available at https://epr.elastic.co/package/o365_metrics/0.4.0/