The surgeon would not have made a decision like this on his/her own. They have 'multi-disciplinary meetings' where each case is discussed and a treatment plan decided on.
Having said that you need to understand the reasoning behind this decision to be reasurred that the right treatment is being done.
In my experience, and what I have read, scans can tell you if there is definitely something there, it cannot tell you that there is definitely nothing there. I also think that the practise is still surgery, even if the cancer has had a complete response to chemo.
Ask for another appointment so that you can talk it through and understand what it all means so that you can make an informed decision, and of course it is YOUR decision.
What surgery did you have and before or after chemo? In what terms have they said all cancer gone, did you get clear margins?