Trying for a baby after cancer

Hi, this is my first post here and I’m looking for some advice from other women who may have been in my situation. I was diagnosed with Breast cancer 7 years ago. I had chemo immediately, a full masectomy, full lymph node clearance, radiotherapy plus bone transfusions and herceptin. I’ve been on Tamoxifen since and was also started on Zolodex before my first Chemo. I know I am very lucky in some respects as I’ve hit my 6 years clear and I’m doing as well as can be expected. However I really want to try for a baby with my husband. I was diagnosed with cancer when I was just 29 and due to the aggressive nature of the cancer I didn’t have time to save any eggs, hence starting the zolodex immediately. I do have a child from a previous relationship however they were only 2 when I was diagnosed so i feel in part that my longing for a child is also because I hate how cancer took part of those baby years with my child from me . My husband and I put any idea of a baby  to the back of our mind as recovery was our first priority however now I feel like maybe it might be worth looking into seeing if it could be a possibility. I’ve spoken with my oncologist and he has agreed to take a break from the zolodex and tamoxifen to see if my periods would even return as a first step but he has said it does come with a risk. However I’m so scared, I just don’t know if I’m being really selfish and risking it all for essentially a pipe dream. Has anyone been in a similar situation? I don’t really know what I’m looking for but would just like to see if women who have been in my position have either taken the plunge and risked it or have made the decision to put recovery first. I know I should just be thankful I’m even here and that I do have a child and part of me wonders if by even chancing it I could end up being “punished” with bad luck and end up with a recurrence. What would you do? Should I just accept that it’s too risky or is it something worth taking a risk for. I’m so confused and tbh so bloody angry that this is annother thing cancer has taken from me. Any idea how I can make this decision and then live with the consequences. Thank to anyone who has read this 

I work with a girl who was diagnosed at 41 , took a break from tamoxifen after 3 years and had 2 children viA ivf at 45 and 46. Then resumed tamoxifen. She has now been discharged from oncology . It is possible talk to your doctors