-
Notifications
You must be signed in to change notification settings - Fork 175
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[server] Add model
argument to server cli
#1584
Conversation
is this a breaking change if we were using |
@mgoin yes. If we make We could add a separate model path entry point but click can't support both AFAIK |
@dsikka per @markurtz lets add a model_path kwarg back in (can rename the positional arg to something else) and allow it to override the positional if given. (Would need to make the positional arg optional in this case I guess) |
model
argument to server cli
* update model path to be an argument; remove unused openai command pathway * add model path arg and option
* update model path to be an argument; remove unused openai command pathway * add model path arg and option
Summary:
click
set-up, to get the use case described in this ticket (shown below): https://app.asana.com/0/1206109050183159/1206524727025314/fdeepsparse.server \ "zoo:llama2-7b-ultrachat200k_llama2_pretrain-pruned50_quantized" \ --integration openai
PR Update
This PR allows us to get the following cli command:
deepsparse.server --integration openai "hf:mgoin/TinyStories-1M-ds"
We can run the following commands with the current set-up:
Caveats (@bfineran):
Shoutout to @rahul-tuli for his click knowledge and help