NB-Public/chatbot
Watch
1
0
Fork
You've already forked chatbot
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Use vLLM API #6

Merged
sven.seeberg merged 1 commit from feature/vllm-support into main 2024-12-26 14:56:02 +01:00
Conversation 1 Commits 1 Files changed 1 +18 -138
sven.seeberg commented 2024-12-25 16:36:30 +01:00
Owner
Copy link

Switch to Llama3.3 on vLLM. Remove streaming response as I could not get it to work. However, it should be possible to implement it.

Switch to Llama3.3 on vLLM. Remove streaming response as I could not get it to work. However, it should be possible to implement it.
sven.seeberg commented 2024-12-26 11:58:07 +01:00
Author
Owner
Copy link

Response lines:

data: {"id":"chatcmpl-f0e2cd75ed01464da5be03d1bc0feeb4","created":1735211127,"model":"meta-llama/Llama-3.3-70B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" ich"}}]}

Currently it breaks at the last line, which is:

data: {"id":"chatcmpl-2ce0dd19279244759003bf5d641d3dc2","created":1735209859,"model":"hosted_vllm/meta-llama/Llama-3.3-70B-Instruct","object":"chat.completion.chunk","choices":[{"finish_reason":"stop","index":0,"delta":{}}]}
Response lines: ``` data: {"id":"chatcmpl-f0e2cd75ed01464da5be03d1bc0feeb4","created":1735211127,"model":"meta-llama/Llama-3.3-70B-Instruct","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" ich"}}]} ``` Currently it breaks at the last line, which is: ``` data: {"id":"chatcmpl-2ce0dd19279244759003bf5d641d3dc2","created":1735209859,"model":"hosted_vllm/meta-llama/Llama-3.3-70B-Instruct","object":"chat.completion.chunk","choices":[{"finish_reason":"stop","index":0,"delta":{}}]} ```
sven.seeberg force-pushed feature/vllm-support from 0b17c0f1c2 to bff68f3b13 2024-12-26 14:55:19 +01:00 Compare
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
  
Effort
High
 estimated 24 hours of work

  
Effort
Low
 estimated 4 hours of work

  
Effort
Medium
 estimated 12 hours of work

  
Prio
High
 Mission critical

  
Prio
Low
 Nice to have

  
Prio
Medium
 Improves function or usability substantially

  
Type
Bug
 Prevents existing functionality from being used

  
Type
Feature
 Adds new features

No labels
Effort
High
Effort
Low
Effort
Medium
Prio
High
Prio
Low
Prio
Medium
Type
Bug
Type
Feature
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
NB-Public/chatbot!6
No description provided.