LLaVA stands for Large Language and Vision Assistant, a powerful multimodal model that combines the strengths of language and vision. Based on OpenAI’s CLIP and a fine-tuned version of Meta’s Llama 2 7B model, LLaVA uses visual instruction tuning to support image-based natural instruction following and visual reasoning capabilities. This allows LLaVA to perform a range of tasks, including:

Visual question answering: answering questions based on image content

Caption generation: generating text descriptions of images

Optical Character Recognition: identifying text in image

Multimodal dialogue: engaging in conversations that involve both text and images