Visual Results


deep_tsf.png

Ground Truth: Fear

GPT-3.5 Prompting

Full Caption:

Chloe is a female adult. Chloe is or has frowning, open mouth. Chloe’s physical environment is right in front of an alien hand in the dark. Chloe is likely feeling...?

GPT Prediction: Fear

Minus Interactions:

Chloe is a female adult. Chloe is or has frowning, open mouth. Chloe's physical environment is right in front of an alien hand in the dark. Chloe is likely feeling...?

GPT Prediction: Fear

Minus Environment:

Chloe is a female adult. Chloe is or has frowning, open mouth. Chloe is likely feeling....?

GPT Prediction: Disapproval

Notable Observation:

Fear is an emotion that may benefit from environmental context. The F1 score for this image dropped from 0.52 and 0.48 with environments to 0.34 without environments, i.e, when "right in front of an alien hand in the dark" was removed from the full caption of this image, the predicted emotion changed from Fear to Disapproval.

deep_tsf.png

Ground Truth: Embarrassment

GPT-3.5 Prompting

Full Caption:

Lucas is a male adult. Lucas is a(n) groom. Lucas is or has lips that flatten, palms open. Mia is Lucas’ bride and she is smiling. Lucas’ physical environment is cake falling down at wedding. Lucas is likely feeling....?

GPT Prediction: Embarrassment

Minus Interactions:

Lucas is a male adult. Lucas is a(n) groom. Lucas is or has lips that flatten, palms open. Lucas's physical environment is cake falling down at wedding. Lucas is likely feeling....?

GPT Prediction: Embarrassment

Minus Environment:

Lucas is a male adult. Lucas is a(n) groom. Lucas is or has lips that flatten, palms open. Mia is Lucas' bride and she is smiling. Lucas is likely feeling....?

GPT Prediction: Happiness

Notable Observation:

When the environment was removed from the following image's caption, Happiness, a positive emotion, was predicted which was not on the list of 13 negative emotions that we provided to GPT-3.5 to choose from.

deep_tsf.png

Ground Truth: Embarrassment

GPT-3.5 Prompting

Full Caption:

Jack is a male adult. Jack is or has smiling. Beth is a customer and she is side-eyeing Jack. Zoe is a customer and she is staring at Jack. Jack's physical environment is eating in a movie theatre. Jack is likely feeling....?

GPT Prediction: Embarrassment

Minus Interactions:

Jack is a male adult. Jack is or has smiling. Jack's physical environment is eating in a movie theatre. Jack is likely feeling....?

GPT Prediction: Happiness

Minus Environment:

Jack is a male adult. Jack is or has smiling. Beth is a customer and she is side-eyeing Jack. Zoe is a customer and she is staring at Jack. Jack is likely feeling....?

GPT Prediction: Embarrassment

Notable Observation:

When the interactions were removed from the following image's caption, Happiness, a positive emotion, was predicted which was not on the list of 13 negative emotions that we provided to GPT-3.5 to choose from.

deep_tsf.png

Ground Truth: Sadness

GPT-3.5 Prompting

Full Caption:

Jane is a female adult. Jane is or has taking off eyeglasses. Mia is Jane's daughter and Jane is putting her hand on Mia's shoulder while Mia has her back turned to Jane. Jane's physical environment is on a couch. Jane is likely feeling....?

GPT Prediction: Sadness

Minus Interactions:

Jane is a female adult. Jane is or has taking off eyeglasses. Jane's physical environment is on a couch. Jane is likely feeling....?

GPT Prediction: Embarrassment

Minus Environment:

Jane is a female adult. Jane is or has taking off eyeglasses. Mia is Jane's daughter and Jane is putting her hand on Mia's shoulder while Mia has her back turned to Jane. Jane is likely feeling....?

GPT Prediction: Love

Notable Observation:

During the ablation studies, when interaction or envrionmental features were removed from the caption of this image, the GPT 3.5 Predictions interestingly changed to Embarrassment and Love (a positive emotion) respectively.

deep_tsf.png

Ground Truth: Pain/Suffering - Physical

GPT-3.5 Prompting

Full Caption:

Jack is a male adult. Jack is or has frowning, rubbing the back. Jack’s physical environment is on a bed with medication on the side. Jack is likely feeling....?

GPT Prediction: Pain/Suffering - Physical

Minus Interactions:

Jack is a male adult. Jack is or has frowning, rubbing the back. Jack's physical environment is on a bed with medication on the side. Jack is likely feeling....?

GPT Prediction: Pain/Suffering - Physical

Minus Environment:

Jack is a male adult. Jack is or has frowning, rubbing the back. Jack is likely feeling....?

GPT Prediction: Disapproval

Notable Observation:

This image shows that when "medication on the side" was removed from the full caption, the predicted emotion changed from Physical Pain/Suffering to Disapproval. This is because the lack of description about environment in the image cap- tions seemed to impact Physical Pain/Suffering the most. The F1 score dropped from 0.73 with full captions to 0.46 and it is even lower than the F1 score for captions without interactions. As a result, we observed that Physical Pain/Suffering needed both the interaction and environment descriptions to be well recognized by GPT-3.5

deep_tsf.png

Ground Truth: Embarrassment

GPT-3.5 Prompting

Full Caption:

Nancy is a female adult. Nancy is a(n) contestent. Nancy is or has squeezing eyes shut, body posture that loosens or collapses, falling onto ground. Alec is a contestent and reaching hand out to Nancy. Nora is a contestent and reaching hand out to Nancy. Nancy's physical environment is at a modelling competition. Nancy is likely feeling....?

GPT Prediction: Embarrassment

Minus Interactions:

Nancy is a female adult. Nancy is a(n) contestent. Nancy is or has squeezing eyes shut, body posture that loosens or collapses, falling onto ground. Nancy's physical environment is at a modelling competition. Nancy is likely feeling....?

GPT Prediction: Embarrassment

Minus Environment:

Nancy is a female adult. Nancy is a(n) contestent. Nancy is or has squeezing eyes shut, body posture that loosens or collapses, falling onto ground. Alec is a contestent and reaching hand out to Nancy. Nora is a contestent and reaching hand out to Nancy. Nancy is likely feeling....?

GPT Prediction: Embarrassment


deep_tsf.png

Ground Truth: Embarrassment

GPT-3.5 Prompting

Full Caption:

Lucas is a Male Child. Lucas is or has Using something to hide face, Knees pulling together, Bringing the feet together. Beth is Lucas' classmate and looking at Lucas. Marie is Lucas' classmate and looking at Lucas. Lucas's physical environment is on stage in a classroom. Lucas is likely feeling....?

GPT Prediction: Embarrassment

Minus Interactions:

Lucas is a Male Child. Lucas is or has Using something to hide face, Knees pulling together, Bringing the feet together. Lucas's physical environment is on stage in a classroom. Lucas is likely feeling....?

GPT Prediction: Embarrassment

Minus Environment:

Lucas is a Male Child. Lucas is or has Using something to hide face, Knees pulling together, Bringing the feet together. Beth is Lucas' classmate and looking at Lucas. Marie is Lucas' classmate and looking at Lucas. Lucas is likely feeling....?

GPT Prediction: Embarrassment


deep_tsf.png

Ground Truth: Embarrassment

GPT-3.5 Prompting

Full Caption:

Terry is a male adult. Karl is a security guard and he is grabbing onto Terry and carrying him out from the stadium. Terry’s physical environment is at a sports game. Terry is likely feeling....?

GPT Prediction: Fear

Minus Interactions:

Terry is a male adult. Terry's physical environment is at a sports game. Terry is likely feeling....?

GPT Prediction: Excitement

Minus Environment:

Terry is a male adult. Karl is a security guard and he is grabbing onto Terry and carrying him out from the stadium. Terry is likely feeling....?

GPT Prediction: Fear


© This webpage was in part inspired from this template.