I find that too. I use the Ortlieb small ones, the gravel version because they come with an extra securing prong, and between those and a drybag on a small pizza rack at the front for bedding, Im good to go camping. None of that setup gets overly snagged on things, and it’s all very secure on bumpy bits. I carry another small dry-bag which I can strapuse Ortliebs, and they've never given me problems. Lightly loaded small ones
on top of the rack “just in case”, and that tends to get used for my jacket when I don’t need to wear it, or for shopping if I need to pick up anything along the way.
The only issue I have now is that my favourite two bikes aren’t quite low-enough geared (24”) when carrying camping gear in hilly places, so I’m trying to wear the chainrings out as fast as I can so that I’ve got an excuse to buy a chainset with small ones! (Not really; Im assembling a bike with mostly secondhand bits but will have the Spa 40/24 crankset, which will take me down to 19”)