Welcome!

By registering with us, you'll be able to discuss, share and private message with other members of our community.

SignUp Now!
  • Guest, before posting your code please take these rules into consideration:
    • It is required to use our BBCode feature to display your code. While within the editor click < / > or >_ and place your code within the BB Code prompt. This helps others with finding a solution by making it easier to read and easier to copy.
    • You can also use markdown to share your code. When using markdown your code will be automatically converted to BBCode. For help with markdown check out the markdown guide.
    • Don't share a wall of code. All we want is the problem area, the code related to your issue.


    To learn more about how to use our BBCode feature, please click here.

    Thank you, Code Forum.

Chat locally with Llama2-7b via console

EkBass

EkBass

Silver Coder
Hi folks.

I am surprised of the development of the smaller models in past two years. Current 13b and even 7b models can outrun DaVinci 3.0 thought OpenAI has also moved on with its products.
However, im using I7-7700 with 32gb ram (no GPU) and this client with 7b model runs smoothly. No need to pay about the tokens for OpenAI if you can tolerate few seconds of waiting time.

Somekind of GUI would be handy, i thought to use TKinter. However, i am not familiar with python and even making this to work took two evenings. If you like to add simple GUI for this, i would be more than happy about it.

Check out: GitHub - EkBass/console-chat-for-llama-2-7b-chat: Simple console program to chat locally with llama-2-7b-chat
 
Ok. Tokeniser had issues after 512 tokens. It was a problem of old things i accidently left there. Should be fixed now. Max context is 8192 tokens and history is preserved in context up to 2048 characters.

Logging system sucks, i guess XML or json is something i want to add some day in random future.
 

New Threads

Latest posts

Share this page

Buy us a coffee!

Code Academy Advertisement
DataCamp Advertisement

About Us

Code Forum is a community platform where coding enthusiasts can connect with other developers, engage in discussions, ask for help, and share their knowledge with a supportive community. It's a perfect place to improve your coding skills and to find a community of like-minded individuals who share your passion for coding.

Site links

Stay Connected

Back
Top Bottom