Can anyone help me how to use this in colab along with local llms

[YaswanthDasamandam]

Hi ,
I normally use colab for using smaller Local models . But Here I am not able to create llm lite server and the autogen at the same time .
because of inline . Can anyone help

I read the Issue #46 here it was a discussion about llmlite server ..

But can anyone create a colab for working with local llms . Then the project would be useful for everyone.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.set_default_device("cuda")

model = AutoModelForCausalLM.from_pretrained("microsoft/phi-1_5", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-1_5", trust_remote_code=True)
inputs = tokenizer('''```python
def print_prime(n):
   """
   Print all primes between 1 and n
   """''', return_tensors="pt", return_attention_mask=False)

outputs = model.generate(**inputs, max_length=200)
text = tokenizer.batch_decode(outputs)[0]

This would be very helpful for everyone

[YaswanthDasamandam]

If I am using the previous method to create what should I send as arguments here instead of the below code

        response = oai.ChatCompletion.create(
            context=messages[-1].pop("context", None), messages=self._oai_system_message + messages, **llm_config
        )
        return True, oai.ChatCompletion.extract_text_or_function_call(response)[0]

[LeoLjl]

Hi @YaswanthDasamandam, could you please check this out? https://microsoft.github.io/autogen/blog/2023/07/14/Local-LLMs

This method requires colab pro, as running commands in terminal is a colab pro feature.

[YaswanthDasamandam]

Is it possible to modify code so user defined methods can also be used instead of llm lite server ??

[YaswanthDasamandam]

First of all thank you for the response

[sonichi]

You can either override the generate_oai_reply method or the create function, depending on which is more convenient for you.

[YaswanthDasamandam]

Thank you for the response.

1. In future are there any plans of creating a general class for these kind of implementations ?
2. Currently everything is revolved around OpenAI API , surely open models are impoving , especially from microsoft . The solution given in the blog post is just a temperory fix . If this project to be used for big companies or projects or even small developers then it is not an efficient way . I have been giving a thought about implementing a class . If I find any simple way to change I will definetely post it here .
   3.Main issue with creating class is all the code that is written for API need to be removed which makes it harder to make any changes without breaking the code .
   4.It would be great if there were plans for creating scheme or general class and be able to modify it :)
   Thank you making autogen public .

[sonichi]

Thanks for sharing your feedback. Please take a look at #95

[YaswanthDasamandam]

how can we use huggingface models with hugging face token