Summarising group conversations

In large group chats, we often have trouble keeping up with the many conversations and threads going on between the participants. Failing to check your messages frequently may often leave hundreds of messages unread. Catching up can be time consuming.

Because most of my services run on GCP, I decided to look for GCP products that served my needs as I believed integration would be simple. I came across Google’s model for summarising google chat conversations, Pegasus. However, using this would require me to train the model and deploy the model to enable the service. I realised this would take too much time (I am slightly impatient). I came across the Vertex AI PaLM API as well, but this seemed limited compared to the service I ultimately chose.

Yes, I ended up using OpenAI’s service. ChatGPT is extremely powerful and it was clear that I should be taking advantage of the technology behind it.

So my application was going to be a telegram bot. My bot would be able to join group chats and record messages that can be extracted and summarised by OpenAI’s service. My bot was built on NodeJS and you’ll be hard pressed to find a service that lacks a pre-built NodeJS module.

OpenAI’s service is pretty easy to use. The following code shows how easy it is. It’s akin to using ChatGPT!

import { Configuration, OpenAIApi } from 'openai';

const configuration = new Configuration({
  apiKey: OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);

openai.createCompletion({
      model: "text-davinci-003",
      prompt: "Can you summarise the following conversation and return the summary in point form, highlighting the top 3 points: " + conversation,
      max_tokens: 100,
      temperature: 0.5,
      top_p: 0.8
    })

Thankfully, OpenAI provides a summarisation model, text-davinci-003. While all models could probably give you decent results, the key is in optimising the parameters you use so that you maximise the use of the API. While exploring the API, I noticed that token use was correlated to the number of characters in your requests. While I can limit the maximum tokens used for the response (100), my main token use derives from the conversation input - which can be pretty lengthy if you have a long conversation to summarise.

Temperature and Top P are other parameters which are harder to tune. This article summarises it well. Essentially, I chose a temperature of 0.5 as I felt summarisation was not a creative task and requires accuracy. I chose a top_p of 0.8 in order to return a decently lengthed summary. These parameters have had reasonable success so far. However, I reckon more investigation is necessary to fully understand the parameters.

Another useful product of this exploration was news summaries. The bot seamlessly summarises news articles posted to my group conversation.

For more information, you may want to check out the OpenAI quickstart and introduction. I do find this API phenomenal. It has enabled many of the AI features I’ve been wanting to build. However, for cost reasons, I dont’t believe this is a scalable solution. It would probably work for personal projects, but a large scale application taking advantage of this API would probably need to negotiate for better costs or deploy their own servers for running the service.