Moon of Alabama Brecht quote
September 03, 2019

737 MAX - Boeing Insults International Safety Regulators As New Problems Cause Longer Grounding

United Airline and American Airlines further prolonged the grounding of their Boeing 737 MAX airplanes. They now schedule the plane's return to the flight line in December. But it is likely that the grounding will continue well into the next year.

After Boeing's shabby design and lack of safety analysis of its Maneuver Characteristics Augmentation System (MCAS) led to the death of 347 people, the grounding of the type and billions of losses, one would expect the company to show some decency and humility. Unfortunately Boeing behavior demonstrates none.

There is still little detailed information on how Boeing will fix MCAS. Nothing was said by Boeing about the manual trim system of the 737 MAX that does not work when it is needed. The unprotected rudder cables of the plane do not meet safety guidelines but were still certified. The planes flight control computers can be overwhelmed by bad data and a fix will be difficult to implement. Boeing continues to say nothing about these issues.

International flight safety regulators no longer trust the Federal Aviation Administration (FAA) which failed to uncover those problems when it originally certified the new type. The FAA was also the last regulator to ground the plane after two 737 MAX had crashed. The European Aviation Safety Agency (EASA) asked Boeing to explain and correct five major issues it identified. Other regulators asked additional questions.

Boeing needs to regain the trust of the airlines, pilots and passengers to be able to again sell those planes. Only full and detailed information can achieve that. But the company does not provide any.

As Boeing sells some 80% of its airplanes abroad it needs the good will of the international regulators to get the 737 MAX back into the air. This makes the arrogance it displayed in a meeting with those regulators inexplicable:

Friction between Boeing Co. and international air-safety authorities threatens a new delay in bringing the grounded 737 MAX fleet back into service, according to government and pilot union officials briefed on the matter.

The latest complication in the long-running saga, these officials said, stems from a Boeing briefing in August that was cut short by regulators from the U.S., Europe, Brazil and elsewhere, who complained that the plane maker had failed to provide technical details and answer specific questions about modifications in the operation of MAX flight-control computers.

The fate of Boeing's civil aircraft business hangs on the re-certification of the 737 MAX. The regulators convened an international meeting to get their questions answered and Boeing arrogantly showed up without having done its homework. The regulators saw that as an insult. Boeing was sent back to do what it was supposed to do in the first place: provide details and analysis that prove the safety of its planes.

What did the Boeing managers think those regulatory agencies are? Hapless lapdogs like the FAA managers`who signed off on Boeing 'features' even after their engineers told them that these were not safe?

Buried in the Wall Street Journal piece quoted above is another little shocker:

In recent weeks, Boeing and the FAA identified another potential flight-control computer risk requiring additional software changes and testing, according to two of the government and pilot officials.

The new issue must be going beyond the flight control computer (FCC) issues the FAA identified in June.

Boeing's original plan to fix the uncontrolled activation of MCAS was to have both FCCs active at the same time and to switch MCAS off when the two computers disagree. That was already a huge change in the general architecture which so far consisted of one active and one passive FCC system that could be switched over when a failure occurred.

Any additional software changes will make the issue even more complicated. The 80286 Intel processors the FCC software is running on is limited in its capacity. All the extras procedures Boeing now will add to them may well exceed the system's capabilities.

Changing software in a delicate environment like a flight control computer is extremely difficult. There will always be surprising side effects or regressions where already corrected errors unexpectedly reappear.

The old architecture was possible because the plane could still be flown without any computer. It was expected that the pilots would detect a computer error and would be able to intervene. The FAA did not require a high design assurance level (DAL) for the system. The MCAS accidents showed that a software or hardware problem can now indeed crash a 737 MAX plane. That changes the level of scrutiny the system will have to undergo.

All procedures and functions of the software will have to be tested in all thinkable combinations to ensure that they will not block or otherwise influence each other. This will take months and there is a high chance that new issues will appear during these tests. They will require more software changes and more testing.

Flight safety regulators know of these complexities. That is why they need to take a deep look into such systems. That Boeing's management was not prepared to answer their questions shows that the company has not learned from its failure. Its culture is still one of finance orientated arrogance.

Building safe airplanes requires engineers who know that they may make mistakes and who have the humility to allow others to check and correct their work. It requires open communication about such issues. Boeing's say-nothing strategy will prolong the grounding of its planes. It will increases the damage to Boeing's financial situation and reputation.

Previous Moon of Alabama posts on Boeing 737 MAX issues:

Posted by b on September 3, 2019 at 18:05 UTC | Permalink

« previous page

I am convinced that an 'old' deterministic processor like Intel Pentium, fabricated in a modern process, with a ballast free, user friendly operation system would outrun any new Intel i9 system with Windows 10

Posted by: b | Sep 5 2019 18:17 utc

To be nitpicking, Pentium already had two pipelines, U&V pipes.
If one want to "keep straight" he has to stop with intel 486 or AMD 5x86 (but not 5K). Also there later was short-lived "straight" mp6/VIA pentium-grade laptop-targeting CPU.

The idea that weak CPU can do nothing because it is weak is stupid, of course. One has just to look into any microcontroller forums, be it AtMel, Arduino, and up to Rapsberry Pie. 80286 will outrun Arduino without breaking sweat i believe. Still there are a lot of real-time clearly defined tasks that those "micro-controllers" do.

Posted by: Arioch | Sep 5 2019 20:37 utc | 101

It is impossible to do REAL real time programming on hardware that is not deterministic, regardless of how many GHz that processor runs at, while coding real time applications for old dinosaur processors like a 1.5MHz 6502 is very much doable.

Posted by: William Gruff | Sep 5 2019 19:42 utc

Oh, come on! It is the same reduction into absurd, just into the opposite direction.

Remember, "God is siding with bigger armies". Brute force means something too.

All the indeterminism of modern CPU would be shorter than one single frequency cycle of those "dinosaurs", and they needed many of them to perform every single command. AFAIR 8086/8 CPU needed half-hundred cycles for integer division op, and i would not even start with division on 6502. I would think that even calling and exiting dummy external interrupt handler would be faster on i9 than good old NOP on 80286.

Of course Windows 10 is not performance-oriented software and neither is deterministic. Using it, or Linux, for hard real-time task is impossible on anu hardware.

But run the same software on both 80286 and i9 and i9 would be better performance-wise, and Atom would be better both performance-wise and power-wise.

Posted by: Arioch | Sep 5 2019 20:48 utc | 102

re Walter | Sep 5 2019 16:14 utc

Similar ref to creating rad isotopes:
What is control?
If you can start s/g but you cannot stop it, can you control it?

Posted by: chu teh | Sep 5 2019 21:06 utc | 103

If you are brute forcing it, then it is not real time code, Arioch @103. What you end up with may be more than fast enough to do the job (typically), but that is not the same thing. Wikipedia sucks for anything that directly relates to the evils of modern empire and capitalism, but it is usually pretty good with some technical articles, with the Real-time computing one pointing out that "Real-time programs must guarantee response within specified time constraints..." The "guarantee" part is where thing can get tricky. The issue of provable correctness in a program when part of that correctness is hard real-time processing doesn't leave room for "Most of the time it will be good enough."

Sure, you can probably rewrite the control code for the 737 in Java and ruin it on a sufficiently fast Windows 10 workstation and get a couple sigmas of reliability out of it. That might very well be good enough for Boing and their customers, but it isn't provably correct real-time processing. It is a kludge rather than proper design... kinda like putting the wrong engines in the wrong place on an aircraft.

Posted by: William Gruff | Sep 5 2019 22:11 utc | 104

re vk Sep 5 2019 15:17 utc | 94
"Boeing didn't want to invest in a new design and calculated any lawsuits, deaths etc. etc. were worth the money saved in bending the FAA."

Of course "Boeing" the company is awkward and nebulous when looking for cause of failure. Companies typically exist as a sheet of paper in a Delaware file cabinet [incorporation].

Also typically, Boeing is run at highest levels by mere hired managers as, also typically, the founder is long dead. Thus since founders and hired managers are different species with different motivations, the motivations and intent are arguably never the same.

Succession has huge dangers, as with all rulers and especially dynastic families.

With regard to nuclear reactor design and safety, there has been an ongoing survival-race since before WW2. Leo Szilard's patent [in 1934!]on neutron chain-reaction fission sat in British Admiralty files until the United States could be proxied to develop it, the USA being the only sovereign state that had the $ and manpower and resources to pioneer it to completion.

USSR and other sovereign states knew their survival depended on having it, too; and still does. They must and will behave accordingly.

For the Homo Sapiens specie, it seems this is all part of some trap that so eludes recognition that it is together we continue toward the cliff. Who/what stops us from putting our attention on a different path?

Posted by: chu teh | Sep 5 2019 22:18 utc | 105

I am convinced that an 'old' deterministic processor like Intel Pentium, fabricated in a modern process, with a ballast free, user friendly operation system would outrun any new Intel i9 system with Windows 10
Posted by: b | Sep 5 2019 18:17 utc | 99

That sounds like comparing a man carrying a couple of books, with a man carrying a washing machine!

Posted by: BM | Sep 6 2019 8:07 utc | 106

@ : chu teh | Sep 5 2019 21:06 utc | 104

What is s/g ?

But generally the principle was stated by Shakespeare, in Hamlet's play, "The Mousetrap"...Our wills and fates do so contrary run
That our devices still are overthrown;
Our thoughts are ours, their ends none of our own.

Essentially once an action is taken the acting agent loses control over it. Better think things through...

Posted by: Walter | Sep 6 2019 10:12 utc | 107

Sure, you can probably rewrite the control code for the 737 in Java and ruin it on a sufficiently fast Windows 10 workstation

Posted by: William Gruff

Now again you switch the topic. You said it is impossible to have time warranties on modern processors, and then you switch to consumer-oriented software.

I am sure you can tell processor from OS, so this twist was intentional on your side.

It is possible to make realtime on modern processors - just run the same control software that you run on 80286.

Then you may claim there is no "mathematical proof" of "intel i9" correctness. But neither there is for 80286 from intel or any unknown manufacturer. Worse, there is not "correctbess proof" for physic laws both processors rely upon.

Posted by: Arioch | Sep 6 2019 14:37 utc | 108

@99 I once needed a real time operation system on an Intel 8086 DOS box. As none was readily available I wrote my own. Snapped the timer interrupt for time slicing/sharing, programmed a task handler for register swapping and a memory handler for mem swaps. Some simply queue handling for asynchronous signals and that was it. Most of it was assembler with some pascal for higher routines. Took me about six weeks and ran flawlessly in a real production environment.

Many people have done similar system that you have developed.  There were companies ( specially in Germany) developed and produced range of IO cards for industrial PC used with some real time OS ( RTDOS, QNX, Windows-NT with RT-extension, etc) because development cost ( hardware and software) on PC platform was a fraction of cost of VME platform.

Avionics and aircraft acceptance test and regulations are too strict. Intel chip ( 80286) architecture does not fit for a real time application for mission critical avionics, where a failure has life danger. For a robust RTOS some other criteria also are needed, e.g. free from memory leaks, free from process deadlock, etc.  Comparing 80286 with  m68k CPU we can see big difference, for example m68k  address bus and arbitration bus make more easier for designer to meet RTOS requirements.

Boeing outsource all its coding to low grade India coders!

That is a vulgar journalism. Avionics software acceptance test cost is multiple time more than software development and coding. It is good to have a look at DO-178B guideline and Avionics software procedures. Does not matter a software is codded in India or USA or elsewhere, it should go through a long long tests to be accepted and certified for flight.

Posted by: arata | Sep 7 2019 5:26 utc | 109

@99 These old processors were quite easy to program. They also ran deterministically. A huge advantage over today's processor with three level of caching, speculative executing and the mess that comes with it (think meltdown and spectre bugs).

Do we need  revisit  nostalgia again? Technology is dynamic and moves all the time with faster and faster pace, non stoppable. If we wish an old CPUs, no problem, we can get a FPGA, upload an IP Core of old CPU, that it is. Easier than before, we can build peripherals and IOs all of them of them on the same FPGA using VHDL. We can run our old codes there, with less power dissipation, less components, more clean and nit.

A deterministic response is not needed for desktop computers, it has a reverse impact on performance, that is reason technology evolved to address user requirements.

Posted by: arata | Sep 7 2019 18:10 utc | 110

The Intel 286 processor has a singular feature that needs to be addressed by alternative processors and that is that the feature size is so large that the processor is inherently radiation upset resistant. The space shuttle used Inter 386 processors for the same reason. An issue with today's flight control computers is that the chips are not very radiation tolerant because of the small feature size. Because of that the processors have to be custom designed to solve that problem. Like the BAE Systems RAD750 space processor and it's successors. They are orbiting inside Jupiter's magnetic field in the Juno program with few issues.

Posted by: Frank | Sep 7 2019 18:19 utc | 111

« previous page

The comments to this entry are closed.