Visual Results
Ground Truth: Fear
GPT-3.5 Prompting
Full Caption: Chloe is a female adult. Chloe is or has frowning, open mouth. Chloe’s physical environment is right in front of an alien hand in the dark. Chloe is likely feeling...?
GPT Prediction: Fear
Minus Interactions: Chloe is a female adult. Chloe is or has frowning, open mouth. Chloe's physical environment is right in front of an alien hand in the dark. Chloe is likely feeling...?
GPT Prediction: Fear
Minus Environment: Chloe is a female adult. Chloe is or has frowning, open mouth. Chloe is likely feeling....?
GPT Prediction: Disapproval
Notable Observation: Fear is an emotion that may benefit from environmental context. The F1 score for this image dropped from 0.52 and 0.48 with
environments to 0.34 without environments, i.e, when "right in front of an alien hand in
the dark" was removed from the full caption of this image, the predicted emotion changed
from Fear to Disapproval.
Ground Truth: Embarrassment
GPT-3.5 Prompting
Full Caption: Lucas is a male adult. Lucas is a(n) groom. Lucas is or has lips that flatten, palms open. Mia is Lucas’ bride and she is smiling. Lucas’ physical environment is cake falling down
at wedding. Lucas is likely feeling....?
GPT Prediction: Embarrassment
Minus Interactions: Lucas is a male adult. Lucas is a(n) groom. Lucas is or has lips that flatten, palms open. Lucas's physical environment is cake falling down at wedding. Lucas is likely feeling....?
GPT Prediction: Embarrassment
Minus Environment: Lucas is a male adult. Lucas is a(n) groom. Lucas is or has lips that flatten, palms open. Mia is Lucas' bride and she is smiling. Lucas is likely feeling....?
GPT Prediction: Happiness
Notable Observation: When the environment was removed from the following image's caption, Happiness, a positive emotion, was predicted which was not on the list of
13 negative emotions that we provided to GPT-3.5 to choose from.
Ground Truth: Embarrassment
GPT-3.5 Prompting
Full Caption: Jack is a male adult. Jack is or has smiling. Beth is a customer and she is side-eyeing Jack. Zoe is a customer and she is staring at Jack. Jack's physical environment is eating in a movie theatre.
Jack is likely feeling....?
GPT Prediction: Embarrassment
Minus Interactions: Jack is a male adult. Jack is or has smiling. Jack's physical environment is eating in a movie theatre. Jack is likely feeling....?
GPT Prediction: Happiness
Minus Environment: Jack is a male adult. Jack is or has smiling. Beth is a customer and she is side-eyeing Jack. Zoe is a customer and she is staring at Jack. Jack is likely feeling....?
GPT Prediction: Embarrassment
Notable Observation: When the interactions were removed from the following image's caption, Happiness, a positive emotion, was predicted which was not on the list of
13 negative emotions that we provided to GPT-3.5 to choose from.
Ground Truth: Sadness
GPT-3.5 Prompting
Full Caption: Jane is a female adult. Jane is or has taking off eyeglasses. Mia is Jane's daughter and Jane is putting her hand on Mia's shoulder while Mia has her back turned to Jane. Jane's physical environment is on a couch.
Jane is likely feeling....?
GPT Prediction: Sadness
Minus Interactions: Jane is a female adult. Jane is or has taking off eyeglasses. Jane's physical environment is on a couch. Jane is likely feeling....?
GPT Prediction: Embarrassment
Minus Environment: Jane is a female adult. Jane is or has taking off eyeglasses. Mia is Jane's daughter and Jane is putting her hand on Mia's shoulder while Mia has her back turned to Jane. Jane is likely feeling....?
GPT Prediction: Love
Notable Observation: During the ablation studies, when interaction or envrionmental features were removed from the caption of this image, the GPT 3.5 Predictions interestingly changed to
Embarrassment and Love (a positive emotion) respectively.
Ground Truth: Pain/Suffering - Physical
GPT-3.5 Prompting
Full Caption: Jack is a male adult. Jack is or has frowning, rubbing the back. Jack’s physical environment is on a bed with medication on the side.
Jack is likely feeling....?
GPT Prediction: Pain/Suffering - Physical
Minus Interactions: Jack is a male adult. Jack is or has frowning, rubbing the back. Jack's physical environment is on a bed with medication on the side.
Jack is likely feeling....?
GPT Prediction: Pain/Suffering - Physical
Minus Environment: Jack is a male adult. Jack is or has frowning, rubbing the back.
Jack is likely feeling....?
GPT Prediction: Disapproval
Notable Observation: This image shows that when "medication on the side" was removed from
the full caption, the predicted emotion changed from Physical Pain/Suffering
to Disapproval. This is because the lack of description about environment in the image cap-
tions seemed to impact Physical Pain/Suffering the most. The
F1 score dropped from 0.73 with full captions to 0.46 and it is
even lower than the F1 score for captions without interactions.
As a result, we observed that Physical Pain/Suffering needed
both the interaction and environment descriptions to be well
recognized by GPT-3.5
Ground Truth: Embarrassment
GPT-3.5 Prompting
Full Caption: Nancy is a female adult. Nancy is a(n) contestent. Nancy is or has squeezing eyes shut, body posture that loosens or collapses, falling onto ground. Alec is a contestent and reaching hand out to Nancy. Nora is a contestent and reaching hand out to Nancy. Nancy's physical environment is at a modelling competition.
Nancy is likely feeling....?
GPT Prediction: Embarrassment
Minus Interactions: Nancy is a female adult. Nancy is a(n) contestent. Nancy is or has squeezing eyes shut, body posture that loosens or collapses, falling onto ground. Nancy's physical environment is at a modelling competition.
Nancy is likely feeling....?
GPT Prediction: Embarrassment
Minus Environment: Nancy is a female adult. Nancy is a(n) contestent. Nancy is or has squeezing eyes shut, body posture that loosens or collapses, falling onto ground. Alec is a contestent and reaching hand out to Nancy. Nora is a contestent and reaching hand out to Nancy.
Nancy is likely feeling....?
GPT Prediction: Embarrassment
Ground Truth: Embarrassment
GPT-3.5 Prompting
Full Caption: Lucas is a Male Child. Lucas is or has Using something to hide face, Knees pulling together, Bringing the feet together. Beth is Lucas' classmate and looking at Lucas. Marie is Lucas' classmate and looking at Lucas. Lucas's physical environment is on stage in a classroom.
Lucas is likely feeling....?
GPT Prediction: Embarrassment
Minus Interactions: Lucas is a Male Child. Lucas is or has Using something to hide face, Knees pulling together, Bringing the feet together. Lucas's physical environment is on stage in a classroom.
Lucas is likely feeling....?
GPT Prediction: Embarrassment
Minus Environment: Lucas is a Male Child. Lucas is or has Using something to hide face, Knees pulling together, Bringing the feet together. Beth is Lucas' classmate and looking at Lucas. Marie is Lucas' classmate and looking at Lucas.
Lucas is likely feeling....?
GPT Prediction: Embarrassment
Ground Truth: Embarrassment
GPT-3.5 Prompting
Full Caption: Terry is a male adult. Karl is a security guard and he is grabbing onto Terry and carrying him out from the stadium. Terry’s physical environment is at a sports game.
Terry is likely feeling....?
GPT Prediction: Fear
Minus Interactions: Terry is a male adult. Terry's physical environment is at a sports game.
Terry is likely feeling....?
GPT Prediction: Excitement
Minus Environment: Terry is a male adult. Karl is a security guard and he is grabbing onto Terry and carrying him out from the stadium.
Terry is likely feeling....?
GPT Prediction: Fear