Humans have distinct experience with the primary emotions of happiness and sadness, with significant gender difference in emotional processing. We conducted EEG source-level analysis of male and female brains in the alpha band while they listened to happy and sad music. Results show that, regardless of stimuli, females have significantly enhanced mean brain activation than males, while males have relatively enhanced mean brain functional connectivity. Additionally, regions such as the cingulate gyrus, medial frontal gyrus, and superior frontal gyrus are more prominently activated in females. Females also have enhanced brain connectivity during sad music listening than happy music, while males have enhanced brain activity during happy music listening compared to sad music in regions underlying emotional processing. Neural response to happy and sad musical stimuli by female and male participants reveal distinct brain activations, thus supporting distinct experiences underpinning them.