Early polyp detection is vital in avoiding colorectal cancer, a top contributor to cancer-related deaths globally. Automated polyp detection has been greatly improved by deep learning, but extensive deployment tends to be hampered by patient data privacy fears. For this purpose, we suggest a federated learning (FL) paradigm that enables decentralized model training without sharing raw patient data while having high diagnostic accuracy. We first performed an extensive evaluation using DL models on a polyp dataset that we gathered. MobileNetV2 was the best performing model as per important metrics like accuracy, precision, recall, and F1 score. In order to further expand its representational power, we incorporated a transformer module into MobileNetV2 so that the model can better capture long-range dependencies and context information. Our new Transformer-Enhanced MobileNetV2 model was then implemented on several simulated clients in a federated learning scenario. This configuration enabled training over decentralized clinical data without violating patient privacy. We utilized standard FL algorithms for model averaging and evaluated the system in accuracy, precision, recall, F1 score, and convergence time. The devised approach performed extremely well with an accuracy of 98.37% and an F1 score of 0.971 while converging effectively in 27-33 rounds. These findings imply that a promising path for safe and effective medical image analysis is to combine transformer designs with lightweight models in a federated condition.