Federated Learning (FL) enables collaborative model training on decentralized data, offering privacy advantages but struggling with data quality variations and adversarial attacks. This paper introduces FEDMAD (Federated Learning for Medical Data with Enhanced Defense), a novel framework designed to enhance robustness in such environments. FEDMAD integrates Homomorphic Encryption (HE) for model update privacy with a quality-aware aggregation mechanism based on a client’s local training loss (1/loss). Our key contribution is the robust aggregation of these quality scores using Median Absolute Deviation (MAD)-based clipping to defend against dishonest score reporting by adversaries. We evaluated FEDMAD on a real-world smoker prediction task using the TenSEAL HE library. Results demonstrate that FEDMAD’s quality-aware mechanism effectively mitigates the impact of noisy clients. More importantly, MAD-based score aggregation is essential for neutralizing dishonest score reporting attacks and preventing model collapse, a scenario where simpler percentile-based clipping fails. While FEDMAD shows significant resilience, our study highlights remaining challenges with sophisticated model poisoning attacks, suggesting directions for future research.