Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TextGeneration] Add Streaming Functionality #1246

Merged
merged 13 commits into from
Sep 21, 2023
Merged

[TextGeneration] Add Streaming Functionality #1246

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
46361d2
add streaming functionality
dsikka Sep 14, 2023
c0c7e37
remove print
dsikka Sep 17, 2023
ab838e7
set back default value
dsikka Sep 17, 2023
7c2ec5d
rebase
dsikka Sep 18, 2023
444f0a2
update to yield
dsikka Sep 18, 2023
5bd29aa
update pipeline.py
dsikka Sep 18, 2023
20a46a7
update tests
dsikka Sep 19, 2023
fb38e2f
refactor out streaming functions and remove yield in process_engine_o…
dsikka Sep 20, 2023
fd40bf9
fix tests
dsikka Sep 20, 2023
592c81c
update pipeline to use kwargs
dsikka Sep 21, 2023
3a18e5f
rebase
dsikka Sep 21, 2023
b833a22
Update src/deepsparse/transformers/pipelines/text_generation.py
dsikka Sep 21, 2023
7db3f39
Update src/deepsparse/pipeline.py
dsikka Sep 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
remove print
  • Loading branch information
@dsikka
dsikka committed Sep 21, 2023
commit c0c7e37338a33742ec22a8eb1f68c34efd0a1002
2 changes: 0 additions & 2 deletions src/deepsparse/transformers/pipelines/text_generation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -528,7 +528,6 @@ def _create_generated_text_output(
finished_reason[0],
logits,
)
print(generation)
generations.append(generation)

return TextGenerationOutput(
Expand Down Expand Up @@ -676,7 +675,6 @@ def engine_forward(
finished_reason.append(FinishReason.STOP)
break

# TODO: Add any generic callback reason?
if callback is not None and callback(token) is False:
_LOGGER.debug(
"callback %s returned False, stopping generation."
Expand Down