ViT 및 VLM 메커니즘 이해 및 코드 스크래치하기

이 글은 VLM 스크래치하는 방법을 나눔한다.

현재 시점(2026.3)에서 최상위 오픈소스 소형 VLM 중 가성비 제일 좋은 것은 gemma-4, qwen vlm 모델이다. 단 gemma-4 는 아무리 작은 모델도 파인튜닝을 위해선는 32GB 이상 VRAM이 필요하다. 

개인 로컬 GPU 에서는 SmolVLM, nanoVLM, 팔마 등이 그나마 적절한 VRAM(16GB) 에서 동작한다.

VLM 파인튜닝 레퍼런스

• Fine-Tuning Gemma 3 VLM using QLoRA for LaTeX-OCR Dataset
• Vision-language-models-VLM: vision language models finetuning notebooks (Medgemma - paligemma - florence .....)
• How to Fine-Tune Qwen3-VL on Your Own Dataset | Datature Blog (over 32g vram)
• VLM-LORA finetuning using OpenCLIP Workload — AMD Enterprise AI for robotics
• The Definitive Guide to Fine-Tuning a Vision-Language Model on a Single GPU (with code) with DORA | by Pavan Kunchala | Medium
• LoRA in Vision Language Models: Efficient Fine-tuning with LLaVA | by Phrugsa Limbunlom (Gift) | Artificial Intelligence in Plain English
• nanoVLM: The simplest repository to train your VLM in pure PyTorch
• SmolVLM - small yet mighty Vision Language Model

VLM 스크래치 레퍼런스

• Training a Vision Language Model from scratch (VLM multi-modal) | by Saptarshi MT | Medium
• Implementation of Vision language models (VLM) from scratch: A Technical Deep Dive. | by Achraf Abbaoui | Medium
• Wiring the Multimodal Mind: Building a Vision Language Model (VLM) from Scratch - Part 1 | by Priyanthan Govindaraj | Medium
• Vidit-Ostwal/VLM-from-scratch: This is majorly for my own learning purpose.
• Building a Nano Vision-Language Model from Scratch
• nipunbatra/vlm-from-scratch
• Building PaliGemma VLM From Scratch using Pytorch | by Shanmuka Sadhu | Jan, 2026 | Medium

오픈소스 라이브러리

• seemore: From scratch implementation of a vision language model in pure PyTorch

ViT 개념 설명 레퍼런스

• Vision Transformer (ViT) from Scratch
• Vision Language Model from scratch in Pytorch #vlm - Qiita
• ViT Scratch Implementation - PyTorch
• Building Vision Transformers (ViT) from Scratch | by Maninder Singh | Medium
• 今井美樹 彼女と TIP ON DUO 歌詞 - 歌ネット
• Building a Vision Transformer from Scratch in PyTorch - GeeksforGeeks
• Training a Vision Transformer from Scratch on CIFAR-10:No Pre-training, No Problem | by Akshay Gokhale | Medium
• Vision Transformer For CIFAR-10