Has eating a lot of soya bean increased female hormones in my body ?

I love body building, But I m a vegetarian. So to get my post workout proteins I used to eat 100 to 150 gms of soyabean daily, I did this for 3 to 4 years. As I joined a better gym I came to know that soyabean has estrogen which is a female hormone which is not good for men. It even developes in males. After consulting my dietician I have stopped eating soyabean since last 5 months. I have started taking other protein rich products like whey protein, Skim milk & yogurt. But my body is not hardening, I mean my body feels soft & its not that muscular either so my question is has consuming soyabeans for such a lomg time made me feminine & impotent ? How do I get rid of this estrogen & imcrease levels of testesterone in my body?