Communicating with GPT4All with C# through HTTP POST JSON

[stradiotto]

Hi

1. I'm trying to make a communication from Unity C# to GPT4All, through HTTP POST JSON

2. I started GPT4All, downloaded and choose the LLM (Llama 3)

3. In GPT4All I enable the API server

4. I start a first dialogue in the GPT4All app, and the bot answer my questions

5. In Unity 2023, I wrote the following code for a component (Note that I'm using TotalJSON, which transforms instances ):

---

5.1) In LLMJSONAux.cs

public class Message
{
public string role;
public string content;
}

public class HttpJSONQuery
{
public string model = "";
public Message[] messages;
public float temperature = .7f;
public int max_tokens = -1;
public bool stream = false;
}

public class Choice
{
public int index;
public Message message;
public string finish_reason;
}

public class Usage
{
public int prompt_tokens;
public int completion_tokens;
public int total_tokens;
}

public class HttpJSONResponse
{
public string id;
//public string _object;
public int created;
public string model;
public Choice[] choices;
public Usage usage;
}

---

5.2) In GPT4AllHTTPPOST.cs

using Leguar.TotalJSON;
using System.IO;
using System.Net;
using UnityEngine;

// https://www.xoborg.com/blog/configuracion-de-gpt4all-y-localai
public class GPT4AllHTTPPOST : MonoBehaviour
{
void testGPT4All()
{
HttpWebRequest httpWebRequest =
(HttpWebRequest)WebRequest.
Create("http://localhost:4891/v1/chat/completions");

    httpWebRequest.ContentType = "application/json";
    httpWebRequest.Method = "POST";

    HttpJSONQuery query = new HttpJSONQuery();
    //query.model = "Phi-3-mini-4k-instruct.Q4_0.gguf";
    query.model = "Meta-Llama-3-8B-Instruct.Q4_0.gguf";
    query.messages = new Message[2] {
        new Message{role = "system", content = "You are a devilish person." },
        new Message{role = "user",   content = "Tell me another joke" }
    };
    query.temperature = .7f;
    query.max_tokens = -1;
    query.stream = true;

    using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
    {
        /*string data = "" +
            "{ " +
                "\"messages\": [    " +
                    "{ \"role\": \"system\", \"content\": \"You are a helpful coding assistant.\" }, " +
                    "{ \"role\": \"user\", \"content\": \"Give me the name of a brazilian actor.\" }  " +
                    "],  " +
                "\"temperature\": 0.7,   " +
                "\"max_tokens\": -1,   " +
                "\"stream\": null" +
            "} " +
            "";
        
            streamWriter.Write(data);*/
        

        JSON json = JSON.Serialize(query);
        string data = json.CreatePrettyString();
        print(data);
        streamWriter.Write(data);
    }

    var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
    using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
    {
        string result = streamReader.ReadToEnd();
        JSON json = JSON.ParseString(result);
        HttpJSONResponse response =
        json.Deserialize<HttpJSONResponse>();
        //print(response.choices[0].message.content);
        print(result);
    }
}

void Start()
{
    testGPT4All();
}

}

6. I put the GPT4AllHTTPPOST component into a game object and run the Unity application I just made

7. The Console logs are these:

7.1) The JSON instruction that was generated and sent:

{
"model": "Meta-Llama-3-8B-Instruct.Q4_0.gguf",
"messages": [
{
"role": "system",
"content": "You are a devilish person."
},
{
"role": "user",
"content": "Tell me another joke"
}
],
"temperature": 0.7,
"max_tokens": -1,
"stream": true
}
UnityEngine.MonoBehaviour:print (object)
GPT4AllHTTPPOST:testGPT4All () (at Assets/_GAME/__Prototypes/LLM Studio/GPT4AllHTTPPOST.cs:49)
GPT4AllHTTPPOST:Start () (at Assets/_GAME/__Prototypes/LLM Studio/GPT4AllHTTPPOST.cs:67)

7.2) The GPT4All JSON answer:

{"choices":[{"finish_reason":"stop","index":0,"message":{"content":"","role":"assistant"},"references":[]}],"created":1717310181,"id":"foobarbaz","model":"Llama 3 Instruct","object":"text_completion","usage":{"completion_tokens":0,"prompt_tokens":25,"total_tokens":25}}
UnityEngine.MonoBehaviour:print (object)
GPT4AllHTTPPOST:testGPT4All () (at Assets/_GAME/__Prototypes/LLM Studio/GPT4AllHTTPPOST.cs:61)
GPT4AllHTTPPOST:Start () (at Assets/_GAME/__Prototypes/LLM Studio/GPT4AllHTTPPOST.cs:67)

8. I do the same instruction for LMStudio, and LMStudio answers correctly to this Unity prototype, and responds my query ("content": "Tell me another joke").

9. So, what part in my JSON instruction I'm missing here, for GPT4All?

PS.: I tried to find Nomic AI e-mail, with no success, but from their web page I could find their Discord account, and this Github discussion list. Also, I have made a search for similar discussions and found nothing about it on GPT4All bindings. If the kind of discussion already exists, it means that my search was not complete, and I apologize for repeating the same request here.

Thanks in advance.