Swollen, tender Right Breast - Advice Please

Hi All -
I went to see my GP 3 months ago as my right breast was very tender and slightly swollen. As I’m only 23 (and finished breast feeding one year ago) was told it was hormonal and sent away. I returned 4 weeks later as pain had increased and given a course of anti-biotics for possible infection. Swelling went down, but pain still present. After 7 days my GP rang asking to see me again, examined breasts again and said she could feel a slight lump around sore spot and referred me to specialist. I also have fatigue, dizzy spells and have lost 1 stone in weight over 2 months (unintentionally) Waited 2 weeks for appointment. Swelling returned with a vengeance and very tender again.
Had appointment yesterday - ultrasound was performed and given the all clear. Spent 2 minutes in a room with doctor who barely felt my breasts, said he couldn’t feel a lump and I was so young I shouldn’t worry. Told me to take painkillers for the pain which was probably an infection, then just left the room. Didn’t listen to my other symptoms. A nurse called in 2 seconds later to say get dressed and go home.
I am still concerned - breast is very very tender and feels really swollen. I myself can feel definite thickness and a lump, plus I feel quite ill. So tired and weak.
Am I just being over paranoid - as the ultrasound ruled out any lumps/cysts? Should I go back to my GP and demand more tests or just take advice to dose up on painkillers and ignore the swelling?
I am so worried and confused. Would appreciate any help/advice.
Thank you.

Hi Gib2009 and welcome here

In addition to the support on the forums our helpliners are on hand with practical and emotional support for you so please feel free to call 0808 800 6000, lines are open weekdays 9-5 and Saturdays 10-2

Take care
Lucy BCC