It's very easy to answer this post with a 'you should have'
We are all different and our ability to deal with aggressive drivers is varied. The reaction of drivers will vary greatly too.
I am no shrinking violet. I have challenged drivers on many occasions. I advise a polite and quite approach. Anything else and you run the risk of a violent/aggressive reaction either verbally or physically. You probably did the right thing. Every incident can only be dealt with at the time as you feel fits the circumstances.
I recently had a car pull out of a car park the driver on his mobile phone was paying no attention and in avoiding him I almost went over the handle bars.
I shouted at him to get off the phone and rode on. I was pulling up outside the café just 200 yards away. Then I saw the driver pull up and park. I walked slowly over to the car where upon the driver jumped out and began to scream and shout and began pushing me and grabbed hold of my jersey. I hadn't spoken at this point. I had to push this man away to get him to let go. I managed to calm him down and quietly explained my grievance. After a short while he came round to my view point and actually apologised and shook my hand.
My approach was not hostile and I did not use bad language or raise my voice. Had I done so I can only imagine the incident would have developed into a full blown punch up. My intention was to politely ask him to be more careful in the future,(similar to your situation?) fortunately something which he later promised to do.
On reflection I thought that had things gone pear shaped and the Police been called. He would no doubt have said I approached him in a threatening manner and he was fearful and merely defending himself. I would have been left looking like the one at fault.
It's all too easy to get into a confrontation that you didn't intend and end up in a worse situation. Judge each situation for yourself and be careful out there.