Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature][Core] Add slot allocation strategy #8233

Open
wants to merge 39 commits into
base: dev
Choose a base branch
from
Open

[Feature][Core] Add slot allocation strategy #8233

wants to merge 39 commits into from

Conversation

zhangshenghang
Copy link
Member

@zhangshenghang zhangshenghang commented Dec 7, 2024 •
edited
Loading

Purpose of this pull request

⚠️ Tips ： After the code is OK, I will add documents

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

@github-actions github-actions bot added dependencies Pull requests that update a dependency file Zeta e2e labels Dec 7, 2024
@zhangshenghang zhangshenghang marked this pull request as ready for review December 7, 2024 13:10
@zhangshenghang
Copy link
Member Author

@Hisoka-X @hailin0 PTAL. After the code is OK, I will add documents

Hisoka-X
Hisoka-X reviewed Dec 9, 2024
View reviewed changes
...test/java/org/apache/seatunnel/engine/e2e/allocatestrategy/SystemLoadAllocateStrategyIT.java Show resolved Hide resolved
...rc/main/java/org/apache/seatunnel/engine/server/resourcemanager/AbstractResourceManager.java Outdated Show resolved Hide resolved
...rc/main/java/org/apache/seatunnel/engine/server/resourcemanager/AbstractResourceManager.java Outdated Show resolved Hide resolved
...src/main/java/org/apache/seatunnel/engine/server/resourcemanager/ResourceRequestHandler.java Outdated Show resolved Hide resolved
...src/main/java/org/apache/seatunnel/engine/server/resourcemanager/ResourceRequestHandler.java Outdated Show resolved Hide resolved
...src/main/java/org/apache/seatunnel/engine/server/resourcemanager/ResourceRequestHandler.java Outdated Show resolved Hide resolved
@Hisoka-X Hisoka-X mentioned this pull request Dec 9, 2024
Open
4 tasks
@xiaochen-zhou
Copy link
Contributor

Apart from the SYSTEM_LOAD and RANDOM modes, I feel that maybe another mode could be added. In Dynamic Slot Mode, selecting the smallest worker that meets the resource requirements could increase the success rate of tasks that require a large amount of resources. @Hisoka-X @hailin0 @zhangshenghang #8242

@Hisoka-X
Copy link
Member

Hisoka-X commented Dec 9, 2024

Apart from the SYSTEM_LOAD and RANDOM modes, I feel that maybe another mode could be added. In Dynamic Slot Mode, selecting the smallest worker that meets the resource requirements could increase the success rate of tasks that require a large amount of resources. @Hisoka-X @hailin0 @zhangshenghang #8242

Reply in #8242 (comment)

Hisoka-X
Hisoka-X reviewed Dec 10, 2024
View reviewed changes
...e-server/src/test/java/org/apache/seatunnel/engine/server/utils/SystemLoadCalculateTest.java Outdated Show resolved Hide resolved
...ache/seatunnel/engine/server/resourcemanager/allocation/strategy/SlotAllocationStrategy.java Outdated Show resolved Hide resolved
...src/main/java/org/apache/seatunnel/engine/server/resourcemanager/ResourceRequestHandler.java Outdated Show resolved Hide resolved
...src/main/java/org/apache/seatunnel/engine/server/resourcemanager/ResourceRequestHandler.java Outdated Show resolved Hide resolved
...rc/main/java/org/apache/seatunnel/engine/server/resourcemanager/AbstractResourceManager.java Outdated Show resolved Hide resolved
...rg/apache/seatunnel/engine/server/resourcemanager/allocation/strategy/SlotRatioStrategy.java
workerAssignedSlots.put(worker.getAddress(), new ImmutableTriple<>(0.0, assignedSlots, 0));

int totalSlots = worker.getUnassignedSlots().length + worker.getAssignedSlots().length;
if (totalSlots == 0) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the worker use dynamic slot, the totalSlots is 0. But we should give a chance to dynamic worker to execute task.

...rg/apache/seatunnel/engine/server/resourcemanager/allocation/strategy/SlotRatioStrategy.java Outdated Show resolved Hide resolved
...g/apache/seatunnel/engine/server/resourcemanager/allocation/strategy/SystemLoadStrategy.java Outdated Show resolved Hide resolved
@zhangshenghang
Copy link
Member Author

@Hisoka-X PTAL

@hailin0 hailin0 mentioned this pull request Dec 16, 2024
Closed
3 tasks
@Hisoka-X
Copy link
Member

Overall LGTM. Thanks @zhangshenghang . But I think we should merge this after 2.3.9 released.

@zhangshenghang
Copy link
Member Author

Overall LGTM. Thanks @zhangshenghang . But I think we should merge this after 2.3.9 released.

Ok, thank you for your review

@Hisoka-X
Copy link
Member

Hisoka-X commented Jan 9, 2025

Hi @zhangshenghang , almost forgot, we need add new doc to introduce our task schedue logic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Hisoka-X Hisoka-X Hisoka-X left review comments

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
dependencies Pull requests that update a dependency file document e2e Zeta
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature][Core] Add slot allocation strategy
3 participants
@zhangshenghang @xiaochen-zhou @Hisoka-X