Soil image classification plays a crucial role in agricultural and environmental practices. Traditional methods of soil classification often involve manual labor, which can be time-consuming and prone to human error. Recent advances in computer vision and machine learning have opened new horizons for automating this classification process. This research paper presents a comprehensive study and evaluates the performance of four convolutional neural network (CNN) architectures a custom CNN, ResNet50, InceptionV3, and MobileNetV2 on a custom soil image dataset comprising 1800 labelled images across four soil classes such as Black, Laterite, Red and White. The dataset created using smartphone camera to captured images under varying natural conditions. The objective of this work is to explore the effectiveness and accuracy of different machine learning algorithms used in categorizing soil types based on visual data. Each model’s performance is evaluated in terms of classification accuracy, precision, recall, and F1-score. Results indicate that ResNet50 achieves the highest accuracy 97.3%, followed closely by MobileNetV2 94.7%. The custom CNN, while computationally efficient, achieved 88.2%. We conclude that transfer learning with deep CNNs is highly effective for soil classification, and MobileNetV2 is a strong recommended for mobile applications. The comparative analysis demonstrates their effectiveness in distinguishing between different soil types, textures, and compositions. It also highlights how important it is to select the appropriate CNN architectures for certain tasks related to soil classification.  This work belongs to the increasing collection of information at the interface between soil science and computer vision. It offers a strategy to apply sophisticated deep learning-based algorithms to assess soil type more reliably and effectively, serving as a springboard for future research in the field of soil image analysis and classification.