While the use of artificial intelligence in ophthalmology has increased over the past few years, inflammatory diseases like choroiditis remain widely unexplored. A delayed diagnosis in such cases can cause severe complications that can lead to visual disability, and hence, automated diagnosis systems can be of great importance. In this study, we present a systematic, evidence-driven methodology for designing a lightweight choroiditis classifier using a real-world small dataset with a constraint of low resource availability. This hybrid framework consists of two components. The first feature extraction component extracts features using the MobileNet-V3-Small model. The second component, namely the classification component, utilises the Support Vector Machine as the binary classifier. This optimal combination was identified through systematic comparative experiments. Statistical testing confirms the robustness of the classifier selection. The model gives a cross-validation accuracy of 94% and a held-out test accuracy of 97.06% with a training time of approximately 3 minutes for the end-to-end pipeline on a carefully collected and previously introduced choroiditis dataset. Being lightweight and computationally efficient, this model is a suitable candidate for future development into a reliable computer-aided diagnosis tool that could assist experts in reviewing images, providing telemedicine care, and prioritising patient appointments based on the initial results of the automated systems.