I would like to clear up a couple of misunderstandings.
There is nowhere that Bikeability insists a cyclist must *always* ride well out. For one thing, there are *recommended* road positions that vary a little depending upon the road type and width. But in general, there are two basic road positions recommended by Bikeability and Cyclecraft.
The first is 'secondary', which is the normal riding position. The recommended distance from the kerb / edge is 1 metre, but this is not strict, and 0.75 is also ok in many circumstances. I used to tell the Bikeability kids to keep at least an arm's length from the kerb because kids have a hard time judging distance without a reference, and that's an easy way to check. I usually ride approximately in the 'groove' made by the inside wheels of motor vehicles.
The other position is 'primary', which is also sometimes referred to as 'taking the lane'. This position is used to contol the lane and/or make the cyclist more visible and/or prevent silly overtakes. It is effective for this purpose, as long as the lane is roughly 3.0 metres or less. Bikeability & Cyclecraft recommend primary position for the approach to junctions and other areas where conflict is likely. There are some exceptions & notes relating to this.
I generally use primary position on the approach to junctions, pinch points, on narrow single tracks road, going fast down hills (especially if the road is winding), and other points of possible conflict, but I certainly do not use it blindly in all of these circumstances. I use my judgement and experience.
I learned my first lesson about road position when I was a teenager, and was forced off the road by a driver who did not realise I need to cross some rail tracks at a square angle (the tracks crossed the road at an angle). The police officer who attended advised me to take the lane there. While I learned increasingly over the years to use 'taking the lane' for some circumstances, it was relatively recently as such things go, around 2009(?), that I read Cyclecraft and started applying what I learned there. I also taught Bikeability between 2010 and 2012. With a little more than 10 years practicing what I learned from Cyclecraft and it's American equivalent by John Forester, I think I have pretty good judgement for myself when it is appropriate to use it and not; where it is helpful and not.
Driver don't need to understand why a cyclist may use a particular road position. They just need not to run cyclists over, drive negligently, or overtake in a dangerous manner.
As for the OP having an excessive number of incidents, my experience is that different places & times of day have very different outcomes. There were a few streets in northern Essex towns where if I rode them at peak times, I was just about guaranteed an incident. Similarly, I used to ride to the schools I taught at, and one school in Braintree was guaranteed to provide me with a couple of incidents during my commute to and from a week's worth of Bikeability. Some areas of Essex, I can't recall ever having any incidents. One road going to & from my village tended to produce incidents about once every couple of months; mostly teenagers in illegally modified cars who seemed to think it would be a bit of fun to harrass a cyclist. I learned to avoid that road as much as possible, but it was the most direct route to the next town for shopping, so I couldn't always easily do so.
I am someone who always looks at myself, for what I could have done better to prevent or reduce the severity of an incident. sometimes, I can see chances to improve; upon analysis, I realise I could have made myself more visible, or I misjudged something about the circumstances. There are few incidents where there is nothing for me to to learn. The most serious one I've had in the last couple of years, a taxi overtook me on the approach to a junction. The light was changing, so the driver (of course) pulled in as they were stopping. I realised as soon as I understood that the taxi driver was overtaking, what was going to happen, so I started braking. The driver stopped with the inside tyres about 5 cm from the kerb. I managed to stop and pull in to the kerb, but the taxi brushed my elbow and came very, very close to making me hit the kerb. It wasn't the only silly overtake I'd had on the approach to that junction. It was just the worst. My lesson from that? I now take the lane (primary position) at the top of the hill before the junction. It's only about 20 metres before I used to, and I haven't had any silly overtakes since then.
In Norway, I experience an incident about once every year or two, instead of once every week or two (as I experienced living in Essex). It took me some months of commuting every day by bike to stop bracing for a game of chicken at every pinch point, and I didn't even realise I had been doing it until my brain let me relax.
“In some ways, it is easier to be a dissident, for then one is without responsibility.”
― Nelson Mandela, Long Walk to Freedom