Being fit in the shower is annoying! What do you think?!?

Ok so i was debating whether to ask this question or not because I didn't want to sound like an , but here it goes anyway. I workout a lot and try to stay in shape so I try to spend time at the gym about 4 times a week. (hole part) so I do realize that I have good genetics and a well proportion muscular physic, I usually go to the gym in baggy clothing so it won't seem like I'm showing off in tight clothing like other dudes who go to the gym, I just stay fit for myself and try to not show off so I wont come off as being y in front of other people. When I'm done with a work out I mind my own business and head for the showers after-wards, but what annoys me is that I noticed that other men stare at me in the shower and I try my best to ignore it, but it gets to me after a while. I don't know why men stare so long when I'm in the shower? My gf thinks I'm weird because I hide my body in public also by wearing old clothes or baggy clothing, I don't know, but I hate the attention, but I like to be really fit. Maybe something is wrong with me, but I just don't want to come off as a coincided especially in front of the fellas because that's just not who I am. I'm soft spoken and try to be down to earth as much as possible and my true friends know that about me already, but the stares in the shower? I find it awkward. Let me know your thoughts or any info, thanks in advance!