Add a new goal to distribute topic leaders fairly

[hmit]

Summary

1. Why: In current CC goals, there is no goal to balance a topic leaders only which ensure a fair distribution of load for not log compacted topics. The rebalance looks at historic loads for replica placement and it's possible that load changes temporarily causing load-skew for brokers. A potential solution to this problem is to distribute topic leader partitions fairly across all the brokers.
2. What: Add a new Topic Replica Leader distribution goal.

This PR adds a new goal to distribute Leaders Replica for every topic in a fair manner across all the brokers. If the partition count is not a multiple of brokers, some brokers do end up with extra partitions.

The thresholds can be configured using 3 new knobs added to AnalyzerConfig.

The new goal is modeled after existing TopicReplicaDistributionGoal with following differences:

1. This is a hard goal
2. It looks at leader replicas in a broker only
3. For candidate brokers with equal number of leader replicas for a topic, it uses total leader replicas as a tie-break

Categorization

• documentation
• bugfix
• new feature
• refactor
• security/CVE
• other

[CCisGG]

Today Cruise Control already have LeaderReplicaDistributionGoal, which I think would achieve the similar thing with the new goal that you are proposed. That said I have a few questions:

1. How is the new goal different from the LeaderReplicaDistributionGoal
2. Looks like the new goal tries to make the leadership balanceness for each individual topic. Is that necessary?
3. Why does it need to be a hard goal?

[hmit]

@CCisGG, thanks for taking a look at this! Please find my responses inline:

Today Cruise Control already have LeaderReplicaDistributionGoal, which I think would achieve the similar thing with the new goal that you are proposed. That said I have a few questions:

How is the new goal different from the LeaderReplicaDistributionGoal
LeaderReplicaDistributionGoal balances the leader partitions across the brokers to be balanced. In practice, we noticed that leader replicas for a single topic are still skewed onto fewer brokers, so when that topic experiences a burst-y produce, corresponding brokers are adversely affected. Similarly, there is a TopicReplicaDistributionGoal that balances across topic replicas but not quite the same as this goal. That's because the leader replicas take bulk of the resources on a kafka cluster.
Looks like the new goal tries to make the leadership balanceness for each individual topic. Is that necessary?
Yes, it's necessary to avoid "false" skew when the leader replicas of a bursty topic are not evenly distributed across brokers.
Why does it need to be a hard goal?
Shipping this as a hard goal ensures running this goal before other hard goals optimizing cpu, network, etc. This is critical to achieve the most balanced cluster possible.

To mitigate the adverse effect of this goal for keyed or compacted topics, we added this goal as a hard goal but not recommend adding it to the default list of hard goals. This allows operations on such kafka clusters to proceed as normal.