Improve handling/formatting of percent values in Matomo

[sgiehl]

Matomo stores values that are displayed as percent values as rates.
So a value like 12.345% would be stored as 0.12345 in the database.

To receive the percent value out of the rate all percent metric classes are currently using the format method. e.g.

matomo/plugins/Actions/Columns/Metrics/ExitRate.php

Lines 43 to 46 in eeb6e3b

	public function format($value, Formatter $formatter)
	{
	return $formatter->getPrettyPercentFromQuotient($value);
	}

matomo/core/Metrics/Formatter.php

Lines 159 to 162 in 3b00013

	public function getPrettyPercentFromQuotient($value)
	{
	return NumberFormatter::getInstance()->formatPercent($value * 100, 4, 0);
	}

This works perfectly when formatting results for the API. But in our UI there are several places where we actually can't directly work with formatted numbers, as we need to apply mathematical operations. This currently produces problems as when fetching unformatted values from the API the rate value would be returned. But as the UI doesn't know that the value would need to be calculated before formatting, we can't use it unformatted. In the past we have implemented various workarounds.

The biggest example would be the API parameter format_metrics=bc, which actually does not format anything except of percent values and is used as default for formatting. That way the percent values are correctly calculated but also formatted. If those numbers then need to be used for calculation this can result in a problem for certain languages. While the English presentation like 14.55% can be easily converted to float, this doesn't apply for e.g. the German one 14,55% or the Turkish %14,55. Both might end up in invalid/incorrect numbers.

There might be various ways to fix this. We might need to discuss the possible solutions to find a proper way, that doesn't end up in another workaround that we might need to refactor at some point.

My ideas for a possible solution would be:

• Change all rate/percent metric classes so they do the calculation (multiply with 100) within the compute method (or similar), so the formatting method only applies the formatting.

  This might actually be some sort of breaking change, as the API would no longer return rate values when data is fetched unformatted. But we would be able to get rid of the bc formatting, and UI would be able to calculate with all values.

• Let the UI know which values are returned as rates, so the UI can calculate the percent value when unformatted values are fetched

  This would be a quite big change, as currently the UI does not know which types the metrics are of. But if it's on purpose to return rate values and we can't change that, this might be an alternative solution.

refs #19940

[sgiehl]

Had some more thoughts on the way of solving the formatting issues.

I might no longer go with the solution of changing how percent values are formatted in backend/API.

Instead we should aim to adjust our API, so unformatted responses contains additional metadata about the metric types, so it's possible to format them based on that.
We already have added some metadata for looker studio. Maybe those data could be re-used for that purpose as well.

After implementing this and migrating all places our UI should fully work with unformatted values.

So the proposed steps to ensure a correct handling of percent values would be:

• Check that our UI (javascript) has all required methods to format any type of metric, e.g. number, money, byte, percent, time,... (I think most should already be there, but not 100% sure)
• Introduce a new API parameter that allows to include the metric type metadata in normal API responses or do that automatically when metrics are requested unformatted (currently this is only done for processed reports)
• Deprecate the BC formatting that is currently default and aim to remove it in next major. Afterwards fully unformatted should be the default.
• Migrate one of the UI reports that currently "struggles" with formatted / unformatted values to fetch data fully unformatted and format them using JS. (e.g. the SingleMetricView widget currently fetches all data twice https://github.com/matomo-org/matomo/blob/5.x-dev/plugins/CoreVisualizations/vue/src/SingleMetricView/SingleMetricView.vue#L276-L308)
• Migrate one of the places where PHP consumes data from API and struggles with formatted values (e.g. Fatal error: ceil(): Argument #1 ($num) must be of type int|float, string given in BaseDraw.php #19940)
• If everything works well we can either go on with migrating the remaining places or create follow-up issues for them to do that step by step.

[ThorstenMichel]

There is even a bug regarding Turkish percentages, I'm not sure if I shall open a new bug ticket for that:

Steps to reproduce

• Change UI language to Türkçe
• Open Behavior

Results

• If bounce rate = 0%, it is correct displayed as: %0 (as usual in Turkiye)
• If bounce rate >= 0%, it is displayed as all kind of letters; it seems %n is interpreted as entities
  

[sgiehl]

@ThorstenMichel Thanks for reporting this one. There hadn't been any report yet that percent values in tables aren't displayed at all for Turkish. I found the problem, but not yet sure how to solve it best.

[diosmosis]

@sgiehl fyi, @mattab asked me to look into this for the moment. I'll put up an incomplete PR soon-ish if you'd like to look at the implementation I'm going w/ early.

[sgiehl]

@diosmosis the core team is currently building up a vision how metrics and their formatting should look like in long term. We are not yet at a point where we could share any details, but if your changes aren't in line with the approach we establish we might not be able to merge your contribution or need to rewrite it afterwards again, unnecessarily increasing the work for everyone.

[diosmosis]

@sgiehl here is the solution I am working towards:

• use ReportMetadataStore.ts to access column metadata in client (no new code added to get column types or modifications to API params/API output)
• create a MetricFormatter JS class that uses https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/NumberFormat and similar functionality to format values based on column type
• use these two classes in SingleMetricView/DataTable

does this approach have a problem in it? In the meantime, I will try to speak to @mattab about this as he chose the issue as one of the ones he'd like me to work on.

[paul-buttimore]

We also have a customer who has noticed the video play rate in the evolution report is displayed as 0.27% instead of 27% as shown in the media titles report