In Artificial Intelligence, voice categorization is important for various applications. Tamil, being one of the oldest languages in the world, comprises rich regional slang differing in tone, pronunciation, and emotive expression. These slang words are difficult to categorize because they are informal and there is limited annotated audio data. This study proposes an enhanced deep learning framework for Tamil slang classification using a balanced audio corpus. The framework integrates data-specific pre-processing techniques, including Mel spectrograms, Chroma features and spectral contrast, to capture the nuanced characteristics of Tamil speech. A DenseNet backbone, combined with LSTM and GRU layers, models both temporal and spectral information. The suggested FRAE-PSA module is an innovative application of the Pyramid Split Attention (PSA) mechanism adapted to support regional and affective variations of speech. Different from current PSA or Transformer-based approaches, FRAE-PSA splits the audio frequency spectrum and adapts attention weights dynamically based on auxiliary tasks. A multi-branch architecture is employed to fuse temporal and spectral features effectively and multi-task learning is used to enhance regional accent and emotion detection. Custom loss functions and lightweight networks optimize model efficiency. Experimental results show up to a 15% improvement in classification accuracy over baseline models, demonstrating the framework's effectiveness for real-world Tamil slang classification tasks.