This content originally appeared on Level Up Coding - Medium and was authored by Itsuki
Of course, there are lots of libraries we can use for evaluating sentiment scores of a given text, NLTK is probably the most famous one!
However, most of the libraries out there only evaluate positive, negative, and neutral. I want a little more than that!
To be specific, I want the 7 basic emotions.
fear, anger, joy, sad, contempt, disgust, and surprise
There is only ONE library I found that says it will provide the result for this.
This library the python version of ML-Ask (eMotive eLement and Expression Analysis system), an emotion analyzer for Japanese text that is based on pattern matching.
Two problems!
- Japanese only (actually not really a problem from my perspective)
- Pattern matching. That is we have a dictionary, the input is matched against the dictionary to evaluate the emotion. Therefore, if the word we pass in is not part of the dictionary, no evaluation can be made!
Conclusion? Not what I am looking for!
Let’s see if Generative AI can solve our (or my) problem! We will be testing using Bedrock Anthropic Claude 3 here, but OpenAI should perform similarly!
Task Definition
Let’s first define the task we want the AI to do!
Input
A text. I am thinking of a text from a text message or a chat room.
Output
Score for each emotion in JSON format like following.
{'fear': 0.1, 'anger': 0.0 ... }To force the output to be in the format we want, we will be using the JSON mode.
JSON Mode is not an actual mode that we can set for while invoking the model but a way of utilizing Function calling (tool use) to force LLM to response in JSON format following a specific schema, even if we don’t have any intention of running that output through a tool or function.
That is the output JSON we want is actually passed in as the input to the tool!
Couple points to keep in mind when using Tool use this way.
- Single tool
- Set tool_choice to force the model to request a tool
- The name of the tool and description should be from the model’s perspective because the output we want is actually the input to the tool
We will get into more details on how to implement this is this coming section!
Implementation
I will be using Python + Bedrock Converse API here but feel free to choose a Language and a Generative AI you like! (I don’t like Python that much though…But Jupyter Notebook is one of the best for testing out ideas!)
Tool Definition
Let’s first define the tool that we will be using.
tool_name = "print_emotion_scores"
description = "Print emotion score of a given text."
tool_definition = {
"toolSpec": {
"name": tool_name,
"description": description,
"inputSchema": {
"json": {
"type": "object",
"properties": {
"fear": {
"type": "number",
"description": "Score for fear, ranging from 0.0 to 1.0.",
},
"anger": {
"type": "number",
"description": "Score for anger, ranging from 0.0 to 1.0.",
},
"joy": {
"type": "number",
"description": "Score for joy, ranging from 0.0 to 1.0.",
},
"sad": {
"type": "number",
"description": "Score for sad, ranging from 0.0 to 1.0.",
},
"contempt": {
"type": "number",
"description": "Score for contempt, ranging from 0.0 to 1.0.",
},
"disgust": {
"type": "number",
"description": "Score for disgust, ranging from 0.0 to 1.0.",
},
"surprise": {
"type": "number",
"description": "Score for surprise, ranging from 0.0 to 1.0.",
},
},
"required": ["fear", "anger", "joy", "sad", "contempt", "disgust", "surprise"],
}
},
}
}
Yes, the name is print.
You might want to call it evaluate, but remember, the scores that we want are actually passed in as the input to the tool, and therefore, the name and description should be from the model’s perspective!
Also, Note that I am not forcing the scores to be added to 1.0 so that we can allow compound emotions!
You can find out more about Call a tool with the Converse API here, and JSON Mode Function Calling here.
Invoke Model
If you are looking at this article, I bet you are already really familiar with invoking a generative AI model so I will skip my introduction on that!
I will be using BedrockRuntime.Client.converse.
from pprint import pprint
import boto3
client = boto3.client("bedrock-runtime", region_name="us-east-1")
model_id = "anthropic.claude-3-haiku-20240307-v1:0"
target_text = "Some Text..."
prompt = f"""
You will be acting as an AI Empath.
Your are an expert at reading emotions within text messages and chats.
The target text will be surrounded by <text></text>.
You have to use using {tool_name} to print out the score for each emotion.
<text>
{target_text}
</text>
"""
messages = [
{
"role": "user",
"content": [{"text": prompt}],
}
]
# Send the message to the model
response = client.converse(
modelId=model_id,
messages=messages,
toolConfig={
"tools": [tool_definition],
"toolChoice": {
"tool": {
"name": tool_name,
},
},
},
)
pprint(response['output']['message']['content'][0]['toolUse']['input'])
Note that in addition to prompting the AI that they have to use the tool, we also set toolChoice to force the model to request our print_emotion_scores tool.
Let’s first check out something pretty straightforward, target_text = “I feel super awesome today!”.
{'anger': 0.1,
'contempt': 0.05,
'disgust': 0.05,
'fear': 0.1,
'joy': 0.8,
'sad': 0.0,
'surprise': 0.2}I am surprised that AI gives a non-zero score for anger, disgust and contempt, but compare to the score for joy, we can probably decide that we can ignore those while processing!
What about something more vague, For example, target_text = “You don’t even know how to do that?”?
Obviously (to human being), this is a compound of multiple emotions, and lies within some kind of context.
How good (or bad) will our Generative AI do? Let’s give it a run.
{'anger': 0.4,
'contempt': 0.2,
'disgust': 0.3,
'fear': 0.1,
'joy': 0.1,
'sad': 0.3,
'surprise': 0.2}Actually, not bad at all!
Conclusion
Time for some conclusions based on my super personal opinion!
Why To Use
- Sentiment is not enough! You want emotions like I do!
- Need to analyze something non-Japanese with a higher accuracy.
- Want a tool that is able to capture vague emotions and context beyond the text itself.
Why Not To Use
- You have to PAY! A Fair amount!
Probably less if you are using OpenAI, but thinking of using Bedrock, you probably cannot invoke it directly from your front-end and need some kind of endpoints built in Lambda. That means it is probably a good idea to add an API Gateway and some Custom Domain and blah!
Thank you for reading!
That’s all I have for today!
Wondering if changing the prompt to something else can provide an even better result? If you have found a better one, please leave me a comment! Would be happy to know!
Happy emotion analyzing!
Sentiment/Emotion Analysis with Generative AI (JSON Mode) was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Itsuki
Itsuki | Sciencx (2024-10-19T13:14:59+00:00) Sentiment/Emotion Analysis with Generative AI (JSON Mode). Retrieved from https://www.scien.cx/2024/10/19/sentiment-emotion-analysis-with-generative-ai-json-mode/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.