Yes, I ended up with severe depression and feeling absolutely appalling after 9 1/2 weeks of Anastrozole. I was advised to come off it. It took about 5 weeks or so to return to where I had been before. My Breast Care Nurse and a nurse I spoke to from Macmillan both said that most women who have a history of depression find they have to come off (and stay off) endocrine therapies (and that includes Tamoxifen, although its pharmacology is different).
Obviously it is your own decision as to what you can cope with and what you can't. Individuals vary, but the general aim of all Endocrine therapies (including Tamoxifen) is to starve the body of oestrogen by one process or another. A woman who is very sensitive to hormonal fluctuations is particularly likely to have problems. Oestrogen plays an important part in the brain in the management of neuroreceptors.
My own decision is to refuse further Endocrine therapy of any sort, having tried Letrozole (allergic rash) and Anastrozole (adverse reactions associated with oestrogen depletion). I have several reasons for not trying Tamoxifen, one being that I have to take a drug (as a form of antidepressant) that is a dopamine agonist, which increases the level of dopamine in the brain. Tamoxifen is a dopamine antagonist and interferes with dopamine transporters in the brain. My own GP and my BCN have both said it would be very undesirable to take the two at the same time as they would be in opposition to each other.
I do hope you find a solution suitable for you. You might need to keep off Anastrozole for longer than two weeks. I would have another word with your BCN.