QVQ-72B-Preview is an experimental research model developed by the Qwen team, focusing on enhancing visual reasoning capabilities.
Key Insights
• MMMU Benchmark: QVQ-72B-Preview scored an impressive 70.3%, reflecting its strong multidisciplinary reasoning and comprehension capabilities.
• MathVision: The model demonstrated significant progress in mathematical reasoning, outperforming earlier benchmarks.
• OlympiadBench: It showed an enhanced ability to handle complex problem-solving tasks.
Recognizing Model Limitations
While the QVQ-72B-Preview shows remarkable capabilities, several limitations must be addressed:
1. Language and Code-Switching: The model may mix or switch between languages, potentially reducing clarity.
2. Recursive Reasoning: It risks falling into recursive reasoning loops, producing unnecessarily long responses without conclusive answers.
3. Safety Concerns: Comprehensive safety measures are essential to ensure the model operates reliably. Caution is advised during deployment.
4. Performance Gaps:
• It does not fully match the abilities of Qwen2-VL-72B in some tasks, such as basic recognition of people, animals, or plants.
• During multi-step visual reasoning, the model might lose focus on image content, resulting in hallucinations.
Current Limitations
• The model supports single-round dialogues and image outputs but does not handle video inputs.
Quickstart Guide
To streamline usage, a toolkit is available for managing various visual inputs like Base64, URLs, and interleaved images/videos. Installation command:
pip install qwen-vl-utils
Example: Using the Chat Model with transformers and qwen_vl_utils
from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info
# Load the model on available devices
model = Qwen2VLForConditionalGeneration.from_pretrained(
“Qwen/QVQ-72B-Preview”, torch_dtype=”auto”, device_map=”auto”
)
# Load default processor
processor = AutoProcessor.from_pretrained(“Qwen/QVQ-72B-Preview”)
# Optional: Adjust visual token range
# min_pixels = 256*28*28
# max_pixels = 1280*28*28
# processor = AutoProcessor.from_pretrained(“Qwen/QVQ-72B-Preview”, min_pixels=min_pixels, max_pixels=max_pixels)
# Prepare input messages
messages = [
{
“role”: “system”,
“content”: [
{“type”: “text”, “text”: “You are a helpful and harmless assistant. You are Qwen developed by Alibaba. You should think step-by-step.”}
],
},
{
“role”: “user”,
“content”: [
{
“type”: “image”,
“image”: “https://qianwen-res.oss-cn-beijing.aliyuncs.com/QVQ/demo.png”,
},
{“type”: “text”, “text”: “What value should be filled in the blank space?”},
],
}
]
# Process messages for inference
text = processor.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
text=[text],
images=image_inputs,
videos=video_inputs,
padding=True,
return_tensors=”pt”,
)
inputs = inputs.to(“cuda”)
# Generate output
generated_ids = model.generate(**inputs, max_new_tokens=8192)
generated_ids_trimmed = [
out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)