Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor chunked matrix for accommodating memory-mapped utility module #192

Merged
merged 2 commits into from
Dec 10, 2022

Conversation

weiliw-amz
Copy link
Contributor

Issue #, if available:
N/A

Description of changes:
Refactor chunked matrix in inference.hpp for PECOS XLinear model inference

  • Concatenated original chunked matrix's fragmented memory allocation
    • For accommodating subsequent memory-mapped utility module.
    • This change increases time cost of making chunked matrix by 5%~10% for large models (>50G), but is necessary and cannot be avoided.
  • Reduced memory footprint of making chunked matrix

Fixed type inconsistency in matrix.hpp.

Tested on large (~50-100G) XLinear models for result validation and time efficiency.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@weiliw-amz weiliw-amz marked this pull request as ready for review December 9, 2022 06:32
@weiliw-amz weiliw-amz self-assigned this Dec 9, 2022
@rofuyu rofuyu merged commit aad94c6 into amzn:mainline Dec 10, 2022
@weiliw-amz weiliw-amz deleted the refactor-inference-for-mmap branch December 10, 2022 01:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants