DeepKE VS Video-LLaVA

Scaling language models have revolutionized widespread NLP tasks, yet little comprehensively explored few-shot relation extraction with large language models. In this paper, we investigate principal methodologies, in-context learning and data generation, for few-shot relation extraction via GPT-3.5 through exhaustive experiments. To enhance few-shot performance, we further propose task-related instructions and schema-constrained data generation. We observe that in-context learning can achieve performance on par with previous prompt learning approaches, and data generation with the large language model can boost previous solutions to obtain new state-of-the-art few-shot results on four widely-studied relation extraction datasets. We hope our work can inspire future research for the capabilities of large language models in few-shot relation extraction. Code is available in \url{https://github.com/zjunlp/DeepKE/tree/main/example/llm.

Bard can read images and, being the same company as YouTube, probably has access to high quality video embeddings they use for YouTube search, probably the most sophisticated video search engine on the planet. It could definitely be using the video content directly without a text representation.

For an open source project that actually “sees” videos you can check out https://github.com/PKU-YuanGroup/Video-LLaVA

The related paper is here: https://arxiv.org/pdf/2311.10122.pdf

I think the TL;DR is "it can tell what's in the video and 'reason' about it"