Summarization of Art Pieces Using Vision-Language Models

Ankara University Computer Engineering Graduation Project

Mustafa Taner Turan¹, Melike Vurucu¹, Bülent Tuğrul²

¹Ankara University, Computer Engineering Department (Students)

²Ankara University, Computer Engineering Department (Advisor)

The Crucifixion with the Virgin, Saint John, Saint Jerome, and Saint Mary Magdalene [left panel] by Pietro Perugino (c. 1482/1485)

Example art piece with generated caption — The Crucifixion with the Virgin, Saint John, Saint Jerome, and Saint Mary Magdalene [left panel] by Pietro Perugino (c. 1482/1485)

Our Caption

A balding man with pale skin stands in front of a rocky cave opening under a clear blue sky in this tall, narrow painting. The man’s body faces us but he turns his head to look off to our right in profile. His gray hair is cut short and he has high cheekbones, a prominent nose, sunken eyes, hollow cheeks, and his lips are parted. A black cloth drapes over one shoulder and across his hips. He holds his left hand, to our left, up by that breast and leans heavily on a wooden staff planted in the ground with his other hand. Plants grow along the bottom edge of the panel and beyond the man. A lion lies on the dirt path behind him, looking at him. More trees grow against the tall, craggy rocks above the cave.

Ground Truth

A man stands, leaning on a staff used like a crutch, in a deep, rocky landscape in this round-topped vertical painting. The cleanshaven man has tanned skin and sparse gray hair. He is nude aside from a slate-gray cloth that wraps around his hips. The hand not bracing the crutch is held in a loose fist at his chest. A few plants grow and bloom along the bottom of the panel. The dirt path on which the man stands winds through low, grass-covered mounds to a cave opening at the base of a tall, steep cliff face. A lion stands on the path between the man and the cave. An owl perches in the bare, spike-like branches of a spindly tree growing to our right of the man.

“A comparative study evaluating the ability of small (<10B Parameters) Vision-Language Models (VLMs) to generate objective art descriptions when fine-tuned under data scarcity.”

Introduction

Detailed visual descriptions of artworks are essential for visually impaired museum visitors. However, generating them is a manual, labor-intensive process that requires specialized art history knowledge, fundamentally limiting accessibility at scale.

This challenge can be framed as a specialized, long-form image captioning problem. Relying on general-purpose Large Language Models (LLMs) for this task is suboptimal and leads to high operational costs. In contrast, lightweight Vision-Language Models (VLMs) are highly suited for this; when fine-tuned effectively, they enable cost-efficient local deployment and significantly reduce deployment costs.

In this study, we fine-tune various small VLMs to generate purely visual, objective art descriptions. We present a comparative analysis evaluating their task-specific performance, hallucination resistance, and overall capabilities.

Methodology

Used Datasets

NGA Open Data: Our primary dataset, consisting of 2,553 images paired with descriptions that adhere to NGA's accessibility guidelines.
(~10% reserved for testing, the remainder for training)
Fig 1. Word count distribution of NGA Open Data dataset's train and test splits.
PixelProse: Synthetic dense-captioning dataset.
(Subsets of 5K–20K samples were used for intermediate task fine-tuning (IFT) to evaluate its impact on performance.)

Training Framework

Quantization: 4-bit quantized base models are used.
QLoRA (Low-Rank Adaptation): Injected trainable rank decomposition matrices exclusively into the attention, multilayer perceptron (MLP) layers of the language backbone, and the cross-modal connector.
Vision encoders were kept completely frozen.

Selected Models

The following models were used for ablation studies:

SmolVLM2 Video Instruct (256M and 500M): Selected for their extremely low parameter counts and efficiency.
Qwen 3.5 (4B and 9B): Selected for leading performance on MMStar, HallucinationBench, and RealWorldQA.
Initial hyperparameter tuning and dataset ablation studies were conducted on this model.
Qwen 3 (2B and 8B): Included to evaluate generational architectural differences.

Ablation Studies and Findings

IFT on PixelProse yielded no performance gains on the NGA evaluation set.
Using multiple similar prompts during training degraded the results.

Consequently, all final models were trained using a standardized configuration exclusively on the NGA dataset using a single prompt.

Evaluation Strategy

We scored model outputs on a 1–5 scale across four axes using an LLM-as-a-Judge framework (Claude Sonnet):

Coverage: Proportion of ground-truth entities accurately captured.
Correctness: Factual accuracy of the described entity attributes.
Hallucination: Avoidance of fabricated or ungrounded visual elements.
Tone: Strict adherence to NGA’s image description guidelines.

Results

**Table 1.** Trained model architectures and parameter distribution.
Model Name	Total Params (M)	LoRA Params (K)	Trainable (%)
Qwen3-VL-2B-Instruct	1256M	34.865K	2.77%
Qwen3-VL-8B-Instruct	5095M	87.294K	1.71%
Qwen3.5-4B	2632M	42.467K	1.61%
Qwen3.5-9B	5783M	58.196K	1.01%
SmolVLM2-256M-Video-Instruct	167M	9.769K	5.84%
SmolVLM2-500M-Video-Instruct	319M	17.367K	5.44%

**Table 2.** LLM-as-a-Judge evaluation scores (1–5) and fine-tuning deltas (Δ = fine-tuned − base) per axis. ↑/↓ denote the column-best and column-worst values per group, respectively.
Model Name	Base Performance					Fine-Tuned Performance					Δ
Model Name	Cov	Cor	Hal	Tone	Ovr	Cov	Cor	Hal	Tone	Ovr	Cov	Cor	Hal	Tone	Ovr
Qwen3-VL-2B-Instruct	2.47	1.99	1.83	1.80	2.02	2.55	2.07	1.99	3.04↑	2.41	+0.08	+0.08	+0.16	+1.24	+0.39
Qwen3-VL-8B-Instruct	2.89↑	2.22↑	1.86↑	1.96↑	2.23↑	2.47	1.95	1.83	2.85	2.28	−0.42↓	−0.27↓	−0.03	+0.90↓	+0.05↓
Qwen3.5-4B	2.75	2.04	1.70	1.75	2.06	2.54	2.06	2.0	2.90	2.37	−0.21	+0.02	+0.30	+1.15	+0.31
Qwen3.5-9B	2.75	2.05	1.53	1.63	1.99	2.62↑	2.16↑	2.09↑	3.01	2.47↑	−0.13	+0.11	+0.56↑	+1.38	+0.48
SmolVLM2-256M-Video-Instruct	1.27↓	1.32↓	1.44	1.32	1.34↓	1.89↓	1.36↓	1.34↓	2.52↓	1.78↓	+0.62↑	+0.04	−0.10↓	+1.20	+0.44
SmolVLM2-500M-Video-Instruct	1.46	1.39	1.41↓	1.16↓	1.36	2.06	1.57	1.48	2.76	1.97	+0.60	+0.18↑	+0.06	+1.60↑	+0.61↑
Cov: Coverage, Cor: Correctness, Hal: Hallucination, Ovr: Overall.

Δ heatmap (green = improvement, red = regression)

Δ heatmap showing per-model, per-axis fine-tuning deltas; green cells indicate improvement, red cells indicate regression — Δ heatmap (green = improvement, red = regression)

Fine-tuning gains are inversely proportional to base capacity: SmolVLM2 improved by up to +0.61 overall, while Qwen3-VL-8B gained only +0.05, regressing in Coverage and Correctness. Crucially, post-fine-tuning rankings are preserved — a rescaling, not a reordering. Tone improved universally across all models (∆ +0.90–1.60), while Hallucination proved the most resistant axis.

Conclusion

Small datasets align tone but cannot teach factual grounding to small VLMs.
LoRA cannot bridge a capacity gap that pre-training left unfilled.
Synthetic intermediate fine-tuning and multi-prompt training degraded performance in low-data regimes rather than reducing hallucinations.
LoRA layer count showed no correlation with performance; targeting specific layers actively destabilized models.
LLM-as-a-Judge is scalable but noisy — automated scores require cautious interpretation.