As Dawnhc says it's never too late to ask. For more information and support about this please do give the helpline here a ring. 0808 800 6000
You have really been through such a traumatic time. I think it is so awful to have very real concerns with this disease just set aside the way your oncologist has done. I don't think it is ever too late to ask to change your oncologist but does the hospital you attend have other breast cancer teams. Is this onc the consultant? If you aren't sure how to proceed maybe your hospital has a PALS Office (Patient Advice and Liaison Service) and they would certainly be able to help you.
I hope the follow up scan shows you have responded well to the treatment. Do come and join us on the Secondary part of the site as I think quite a few with secondary diagnosis don't always read the other areas of the forum.