Ali, I think there are basically two reasons that people avoid dairy:
(1) Some people are concerned that cows milk contains oestrogen which might stimulate ER+ BC to grow in humans. Some are also concerned that cows milk may contain antibiotics which might compromise the immune system of human beings.
The truth is that cows are not fed hormones in the UK (it's against the law), but there is still concern about oestrogen in milk, because cows can be milked while pregnant. Consequently, if cows are milked during the final trimester of pregancy, oestrogen levels can be quite high (although nodbody knows if bovine oestrogen crosses the species barrier). There is also a debate about whether antibiotics can pass into milk/have any effect on humans, so the simplest way of avoiding this, without cutting dairy is to opt for organic dairy products (as organic farmers do not milk cows in the final trimester).
(2) Other people avoid dairy because they are following something called the China diet (people believe in this diet because it appears China has low levels of BC). But some think the China diet could actually be harmful because it contains high levels of soya,which contain phytooestrogens that could stimulate ER+ BC. Also the China diet is low in protein, which could lead to dietary deficiencies, and weaken people with already compromised immune sytems. Other critics have said that if the Chinese diet does offer protection against BC, that advantage is most likely gained during childhood. The theory is that feeding children phytoestrogens, de-sensitises their cells to the effects of hormonal stimulas later on. The other point to mention is that while China have low levels of BC, they also have the highest levels of stomach cancer in the world.
Personally I think faddy diets are unhealthy. I try to eat lots of fruit and veg, and my protein comes from eggs, fish, skinless chicken, pulses, and low fat milk and cheese (to limit fat). I also have organic, because I dont want to risk additives.