Pot-holes if not seen prior to impact, may result higher actual forces because the reaction force could apply within about 0.04 seconds compared with a quick reaction time of about 0.1 seconds, i.e. at 8m/s crossing a pot-hole of 300mm would take about 0.0375 seconds.
Page 4 shows data from hitting deep potholes and the estimated force from wearing a helmet, based on old data for heavier helmet weights than would generally be used today.
https://www.researchgate.net/publicatio ... jury_rates
There is reasonable evidence showing a higher fall off rate for helmet users. Many reasons can be suggested for the higher fall off rate.
The first step if riding over potholed surfaces regularly would be to use large wheels. Wearing a helmet will likely increase the out of balance forces and add to your risk of falling off.
I was once asked why I did not wear a helmet by a police officer and I told her, they increase the accident risk and make cycling cycling less convenient.
edit
you could add the evidence for their use is far from sound.