Uber car - software problem

Commuting, Day rides, Audax, Incidents, etc.
atlas_shrugged
Posts: 263
Joined: 8 Nov 2016, 7:50pm

Uber car - software problem

Postby atlas_shrugged » 6 Nov 2019, 1:28pm

The Uber car that killed a cyclist/pedestrian as she crossed the road had a 'software problem'.

The 'software problem' was that it failed to recognise the woman crossing the road because it failed to recognise her as a pedestrian. The software was only programmed to recognise pedestrians at pedestrian crossings.

Ah OK so then it is her fault - so that's OK then.

brooksby
Posts: 376
Joined: 21 Aug 2014, 9:02am
Location: Bristol

Re: Uber car - software problem

Postby brooksby » 6 Nov 2019, 1:41pm

I read that article - the investigation has basically admitted that they didn't teach their AI to recognise "jaywalking pedestrians". Makes you wonder what else they didn't think to teach their AIs about...

User avatar
RickH
Posts: 4538
Joined: 5 Mar 2012, 6:39pm
Location: Horwich, Lancs.

Re: Uber car - software problem

Postby RickH » 7 Nov 2019, 3:39am

There are bound to be other things the AI (of whatever flavour) doesn't recognise. That is the reason there is supposed to be an attentive driver monitoring the driving & intervening where necessary at this stage of development. That seems to have been one of the biggest failings in this incident.

The advantage of AI is that, whenever how to deal with some new situation is learned, it is learned by all the vehicles.

kwackers
Posts: 13732
Joined: 4 Jun 2008, 9:29pm
Location: Warrington

Re: Uber car - software problem

Postby kwackers » 7 Nov 2019, 8:33am

RickH wrote:There are bound to be other things the AI (of whatever flavour) doesn't recognise. That is the reason there is supposed to be an attentive driver monitoring the driving & intervening where necessary at this stage of development. That seems to have been one of the biggest failings in this incident.

The advantage of AI is that, whenever how to deal with some new situation is learned, it is learned by all the vehicles.

Indeed.

I've said on other threads that this is the critical part of development.
The AI can make mistakes and unfortunately there can be accidents and unlike manually driven cars where we're socially conditioned to accept deaths and injuries there's no such conditioning for AI driven vehicles where every incident almost reaches "end of days" levels of hysteria.

There are failures here by both Uber and the driver although you could argue that the AI was good enough for her to feel comfortable watching a video when she should have been watching the road.
So far KSI's by AI's are way down per mile compared to the ape driven equivalent and that's only going to carry on getting better.

I look forward to removing the apes from the road, they were never designed for driving fast vehicles. Slow to respond and sensors that are too narrow in range and spectrum to be effective.

Sort of related I was test driving a couple of cars recently and the amount of tech that's appearing and related is astounding.
Radar systems that tell you if anyone is approaching when you back out of your drive, alerts that tell you if someone is overtaking when you go to move out, lane keeping alerts auto braking, sign reading, speed warnings and even limited self driving as in keeping the car in lane on motorways and A roads - and these aren't even expensive cars.
This I feel is how we'll get self driving cars. Increasing amounts of tech taking over from the driver until one day the "self drive" button appears and not long after steering wheels start disappearing.

Pete Owens
Posts: 1561
Joined: 7 Jul 2008, 12:52am

Re: Uber car - software problem

Postby Pete Owens » 7 Nov 2019, 11:50am

kwackers wrote:I look forward to removing the apes from the road, they were never designed for driving fast vehicles. Slow to respond and sensors that are too narrow in range and spectrum to be effective.

^^^ THIS ^^^

What we are looking at here is cowboy software development by Uber - but we should not extrapolate from one incident that self driving cars will not massively improve safety. That would be like condemning all cycles as lethal to pedestrians based on Charlie Alliston - or grounding Airbus planes due to the dodgy software that Boeing installed on the 737 Max.

Another point is that it is unreasonable to blame the woman in the vehicle. As you point out above humans have vastly inferior sensors and much slower reaction times - this means that by the time a human has noticed that a computer has failed to apply the brakes it is already too late. If the developers were relying on real-time supervised emergency override then that is a fault of the development team for allocating that task to a human - not the fault of the human given an impossible task.

On the other hand it is possible and indeed common for computers with their quicker reaction times to provide a safety override for human operation. Indeed the car involved was equipped with such a system and the Uber developers disabled it!

rmurphy195
Posts: 1428
Joined: 20 May 2011, 11:23am
Location: South Birmingham

Re: Uber car - software problem

Postby rmurphy195 » 7 Nov 2019, 7:30pm

RickH wrote:There are bound to be other things the AI (of whatever flavour) doesn't recognise. That is the reason there is supposed to be an attentive driver monitoring the driving & intervening where necessary at this stage of development. That seems to have been one of the biggest failings in this incident.

The advantage of AI is that, whenever how to deal with some new situation is learned, it is learned by all the vehicles.


And it is missed out by all vehicles using the same software and sensor setup. So instead of an odd inattentive driver failing to spot a pedestrian on the road, but not on a crossing, all of the damn things will i.e. mass, not individual, failure. A weak point on all forms of computerisation (to use an outdated term)
Brompton, Condor Heritage, creaky joints and thinning white (formerly grey) hair
""You know you're getting old when it's easier to ride a bike than to get on and off it" - quote from observant jogger !

kwackers
Posts: 13732
Joined: 4 Jun 2008, 9:29pm
Location: Warrington

Re: Uber car - software problem

Postby kwackers » 7 Nov 2019, 9:29pm

rmurphy195 wrote:And it is missed out by all vehicles using the same software and sensor setup. So instead of an odd inattentive driver failing to spot a pedestrian on the road, but not on a crossing, all of the damn things will i.e. mass, not individual, failure. A weak point on all forms of computerisation (to use an outdated term)

And your point being what?

That every self driving car will suddenly hit a pedestrian in the dark? Doesn't seem very likely does it?
There were several similar vehicles on the road running the same software and hardware and none of the others had just the right combination of events occur to cause an accident.

The software and hardware used by us apes is hardly great. We kill a million people a year and injure tens of millions more.
Imagine if after I had a crash you could have your software upgraded to prevent you crashing in the same situation. Self driving cars can do that, us on the other hand make the same mistakes day in and day out.
See those apes on the road all the time, it's only luck rather than judgement that keeps the KSI numbers as low as they are.

One day we'll upgrade all the apes with better hardware, 360 degree vision, radar as well as visual senses and vastly faster processing that never gets sidetracked or falls asleep or starts daydreaming.
It'll be a great day for the millions of folk who were destined to die or be seriously injured in the next year and their friends and family.
Can't come soon enough imo.

User avatar
RickH
Posts: 4538
Joined: 5 Mar 2012, 6:39pm
Location: Horwich, Lancs.

Re: Uber car - software problem

Postby RickH » 7 Nov 2019, 9:34pm

rmurphy195 wrote:And it is missed out by all vehicles using the same software and sensor setup. So instead of an odd inattentive driver failing to spot a pedestrian on the road, but not on a crossing, all of the damn things will i.e. mass, not individual, failure. A weak point on all forms of computerisation (to use an outdated term)


Only missed out until the situation is encountered.

If the driver had been paying attention then the failure would have been harmlessly noted & a correction to the software added. Yes all vehicles may have a particular flaw until one encounters an uncommon situation but they will all get the "fix".

Google/Waymo seem to have gone down the road (sorry!) of bringing their cars to a safe stop when they encounter an unknown situation. I remember a report saying one of the situations one of their cars dealt with (by stopping) was a lady in a wheelchair going round in circlrs trying to chase some ducks off the road!

irc
Posts: 4565
Joined: 3 Dec 2008, 2:22pm
Location: glasgow

Re: Uber car - software problem

Postby irc » 7 Nov 2019, 10:00pm

kwackers wrote:SI look forward to removing the apes from the road, they were never designed for driving fast vehicles. Slow to respond and sensors that are too narrow in range and spectrum to be effective.


You may have a long wait.

Waymo Chief Executive Officer John Krafcik said self-driving cars will require driver assistance for many years to come, and that he doesn’t envision a day when the technology operates in all weather conditions and without some sort of "user interaction."


https://www.bloomberg.com/news/articles ... or-decades

And cars that are almost fully auto are not safe as any driver only required to intervene every hundred miles or more will be asleep or watching Netflix when that pedestrian walks out in front of them.

kwackers
Posts: 13732
Joined: 4 Jun 2008, 9:29pm
Location: Warrington

Re: Uber car - software problem

Postby kwackers » 8 Nov 2019, 10:36am

irc wrote:You may have a long wait.

I originally said I thought it would be 10 years before we started seeing true self driving cars and I'm still happy we're on course for that.

What has surprised me is how fast things are moving. AI is advancing much more rapidly both in terms of available hardware and the software that runs on it.
We're actually pretty close right now and improvements are exponential. Hard to imagine how good AI will be in 10 years...

Pete Owens
Posts: 1561
Joined: 7 Jul 2008, 12:52am

Re: Uber car - software problem

Postby Pete Owens » 8 Nov 2019, 3:05pm

irc wrote:https://www.bloomberg.com/news/articles ... or-decades

And cars that are almost fully auto are not safe as any driver only required to intervene every hundred miles or more will be asleep or watching Netflix when that pedestrian walks out in front of them.


Human supervision time to intervene in emergency situations in real-time is never going to be possible - however alert the human or however frequently that intervention would be required. The speed, signal handling capacity and processing power available is orders of magnitude slower than the computer - so by the time even the most alert human has not only noticed the hazard, but has noticed that the computer was failing to take action then it will be too late to intervene. the idea is a complete non-starter so would not be considered by any remotely competent software developer. If the Uber were indeed relying on the driver for a safety override as part of the development process then that alone demonstrates that they are a bunch of cowboys.

The sort of intervention the Waymo CEO is talking about are unusual situations where the car would require to be driven somewhere where it would would be very difficult to programme a computer to deal with automatically (or where the computer would consider it unwise to proceed). For example a village fete or sporting event with organised parking on a farmers field with marshals giving directions by waving big pointy fingers. In these sort of situations the computer would pull over safely and hand over control to the human in an orderly fashion - in just the same way a human might now hand over control to a computer to reverse park in a tight space.

User avatar
[XAP]Bob
Posts: 16940
Joined: 26 Sep 2008, 4:12pm

Re: Uber car - software problem

Postby [XAP]Bob » 11 Nov 2019, 10:31pm

The computer should have known, and been able to alert a driver, that it had just reudentified an object several times in the same space.

Of course it also should have had a dedicated safety driver - and didnt
A shortcut has to be a challenge, otherwise it would just be the way. No situation is so dire that panic cannot make it worse.
There are two kinds of people in this world: those can extrapolate from incomplete data.

Pete Owens
Posts: 1561
Joined: 7 Jul 2008, 12:52am

Re: Uber car - software problem

Postby Pete Owens » 12 Nov 2019, 3:19pm

[XAP]Bob wrote:
Of course it also should have had a dedicated safety driver

Why?

As I explained above - humans are simply too slow to perform as a safety override for a computer. They have slow moving unidirectional sensors, slow processing power to decide what do with that information and slow moving actuators with which to implement that decision. By the time a human has noticed that a computer his failing to respond to a hazard it is much too late for them to intervene. The task simply is not possible.

On the other hand having a computer providing safety oversight of human drivers is eminently practical and not uncommon.

User avatar
[XAP]Bob
Posts: 16940
Joined: 26 Sep 2008, 4:12pm

Re: Uber car - software problem

Postby [XAP]Bob » 13 Nov 2019, 7:39pm

Because this is a test vehicle - it is legally required to have a safety driver.
Said driver should have a display (probably a hud) or identified objects, properly coded... would have been immediately obvious that the person has not been recognised
A shortcut has to be a challenge, otherwise it would just be the way. No situation is so dire that panic cannot make it worse.
There are two kinds of people in this world: those can extrapolate from incomplete data.

User avatar
[XAP]Bob
Posts: 16940
Joined: 26 Sep 2008, 4:12pm

Re: Uber car - software problem

Postby [XAP]Bob » 13 Nov 2019, 7:42pm

The other advantage of course is sensor upgrades.. it is now possible to have lidar which also determines instant speed (radially, tangential speed still needs multiple samples and correlation)
A shortcut has to be a challenge, otherwise it would just be the way. No situation is so dire that panic cannot make it worse.
There are two kinds of people in this world: those can extrapolate from incomplete data.