Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support transformer models #16

Merged
merged 13 commits into from
Oct 17, 2023
Merged

Support transformer models #16

merged 13 commits into from
Oct 17, 2023

Conversation

dacorvo
Copy link
Collaborator

@dacorvo dacorvo commented Oct 17, 2023

This adds support for Transformer models:

  • quantize LayerNorm,
  • add QTensor dispatches for typical transformer operations (split/merge per-head),
  • add text-classification example (SST2).

dacorvo added 13 commits October 12, 2023 18:36
@dacorvo
feat(nn): add QLayerNorm
e3ac77c
@dacorvo
feat(quantize): also replace LayerNorm
f4bada5
@dacorvo
fix(qtensor): add missing transpose args
6e08d35
@dacorvo
feat(qtensor): support float division
113faf7
@dacorvo
feat(qtensor): add numpy() method
3c09cd5
@dacorvo
feat(qtensor): dequantize when adding QTensor of different scales
45ef689 
Instead of raising an error, we dequantize before adding QTensor of
different scales.
This is required to support attention blocks that add activation tensors
from two different branches.
In attention blocks, the float output of the addition will in general be
immediately requantized by the next QLinear.
@dacorvo
feat(qtensor): add default unary dispatches
4bdb3d2 
This adds two default unary dispatches:

- one for type-agnostic operations, where we can simply apply the
operation on the data withoput changing the scale,
- one for unuspported operations, where we dequantize to apply the
operation on float values.
@dacorvo
feat(examples): add BERT model example
8afd149
@dacorvo
feat(calibrate): also calibrate QLayerNorm
c0006c8
@dacorvo
refactor(examples): create subfolders by tasks
8de2d45
@dacorvo
chore: update setup.sh
f957ae0
@dacorvo
doc: update README
5ae72a1
@dacorvo
fix: avoid using evaluate
bb553fc
@dacorvo dacorvo merged commit d76820c into main Oct 17, 2023
@dacorvo dacorvo deleted the support_transformer_models branch October 17, 2023 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@dacorvo