FWIW I like frames with laterally stiff chainstays, but it is easy to make the main triangle too stiff, or at least so stiff that it is of no great benefit most of the time, at the expense of other things.
Test rides of very (torsionally) stiff frames often feel encouraging, but after a full day in the saddle I usually have a slightly different view of them.
Its worth mentioning that there are at least four different loading conditions;
1) pedalling normally
2) pedalling forcefully whilst seated
3) climbing out of the saddle
4) sprinting out of the saddle
The parts of the frame that are stressed and the loads involved are completely different in each case. The power output can vary by about a factor of x10 between these conditions. The only common feature is that the BB shell sees a bending load because of the force on the pedals; however the magnitude of these forces and the parts of the frame that react and resist them are very different.
When riding normally, the bending load in the BB is effectively resisted by the seat tube alone; there is little or no load on the handlebars (you can usually pedal 'the same' with just fingertips or no hands) , and the remaining parts of the frame are not able to transfer load efficiently to the saddle, which is where the force is reacted laterally. [If you have ever ridden a bike with a fractured seat tube, it is immediately clear that the frame now has a hinge in it. If the DT is stiff enough, some semblance of 'normal pedalling' can be maintained, but only by reacting the pedalling loads at the handlebar.] The key thing about this pedalling mode is that it should be
efficient because this is how most of your energy will be expended in the course of a day's ride. The perceived recipe for best efficiency is different for different folk.
Pedalling forcefully whilst seated usually involves working the handlebars to some extent. The power output is such that this level of exertion cannot be maintained for long; nearly everyone uses this level of effort (often without realising) when starting off, and during any brief acceleration. Because the handlebars are being worked, you can feel the torsional stiffness of the DT. In this pedalling mode the stiffness of the seat tube is no longer as important.
Climbing out of the saddle can be done in two different ways; in one way the handlebars see a high torque, but in the other way the bike is leant more from side to side and (especially in sustained efforts of this type) the leaning provides the bulk of the reaction torque into the frame to resist the pedalling loads. In these cases there are lateral loads on the wheels, the forks, and the chainstays that are not present in some other pedalling modes. Note also that the steering geometry starts to be important in an unusual way; in a nutshell, a low trail machine may require less handlebar wrestling than a high trail machine, but in practice you can get used to anything.
Sprinting out of the saddle is like the previous mode except that the power is several times greater; lateral loads are significant and sprinting on a very flexible frameset can be somewhat alarming.
In the latter two modes it isn't clear how important stiffness really is. Arguably if the frame is torsionally stiffer, all it does is allow you to put a higher lateral load into the wheels and tyres more easily, no matter how badly you pedal or steer. This may not make you go any faster, and indeed it may just allow you to 'beat up' a lightweight tyre or wheel more easily than if they were installed in a more flexible frame.
The mode where you 'feel the stiffness working for you' most often and most easily is the one where you pedal forcefully in the saddle. However this comprises only a small proportion of most rider's efforts over a day's ride. It is not clear if there is a major improvement in efficiency if the frame is made stiffer in the DT or not; it feels like there is, but this could be an illusion. Even if there is a real benefit, lets say +10% in that mode, if that mode only comprises 1% of your riding, you are looking at 0.1% overall improvement or something like that.
So the 'seated, pedalling normally' mode is the one where you might spend 90-95% of your time riding and here the thing that dominates the perceived stiffness (the DT) isn't really that important. In this mode quite small changes in efficiency will affect the bulk of a day's ride. JH's planing theory may or may not be a real thing; I think that I 'pedal better' on some frames rather than others, and that if they are too stiff, it works less well for me.
If we look at the evidence for frame stiffness improving efficiency there isn't that much really; it has been an assumption that 'more stiffness is better' for many people (myself included for years) but there is some evidence that frames that are too stiff are no good too; in the past increased torsional stiffness also meant increased vertical stiffness and such frames are definitely needlessly uncomfortable as well as
actually slower (in all likelihood) over any road that isn't perfectly smooth. Sean Kelly sprinted efficiently on a frame that even I could tie in knots quite easily...
If the frame acts in any way like a spring, that spring needs to be matched to the loads and masses involved in order to 'work right'; it is all rather chicken and eggish, but it is quite possible that some riders are more efficient on some frames because they happen to pedal (through nature or habituation) in such a way as they exploit the spring-like properties of the frame to best effect.
Years ago as a teenager I remember seeing the traditional 'toe test' for frame stiffness for the first time; standing to one side of the machine, one hand on the handlebar stem, one on the saddle, and then push the BB sideways with your toe, and let it recover. This way you could soon see if the frame was really stiff or not. However it was clear to me even then that stiffer frames were not always
better frames and it was also clear that not all frames of equal stiffness were as perfectly springy as one another, either. Subjectively the 'toe test' would be a good indicator of how the frame would
feel when accelerating, but (presuming that you were unlikely to pick up anything other than the grossest differences in how
efficiently springy the frame was) it was less easy to tell if you would be
quick (i.e. efficient) on any given frame.
[Only the worst frames were so soft (usually in the chainstays) that they would 'take a set' in the toe test, and these would also be the frames that would be most likely to break in hard use too. FWIW I've never heard anyone claim that chainstays that were less laterally stiff would be of any real benefit to anything, and I know that soft chainstays can exceed yield in hard use, too, so 'more is better' is well accepted with regard to chainstays.]
If one is planning to carry a heavy load, then more frame stiffness is often a good thing, because it is more likely to keep the whole plot moving forwards in the intended direction. But outside of that I don't think that vastly increased stiffness in the main triangle necessarily does that much good; it may
feel better on brief accelerations, but even if that is of genuine benefit, it is difficult to see how it can improve matters that much over the course of a day's ride.
By contrast if the 'pedalling normally' mode is affected (for good or ill) by one percent, then that will stay with you all day, every day. It is much less obvious what affects this; if it were, there would be less variation in frame stiffnesses in supposedly 'good frames' and this wouldn't still be such a contentious topic....
cheers