Moon of Alabama Brecht quote
Back to Main

June 27, 2019
Boeing’s Software Fix For The 737 MAX Problem Overwhelms The Plane’s Computer

The Boeing 737 MAX continues to be a troublesome airplane.

Two crashes of the plane type, which cost the lives of 346 people, revealed a significant problem not only with the messed up Maneuvering Characteristics Augmentation System (MCAS).

It then turned out that the manual trim wheels which Boeing advised to use to counter MCAS are impossible to move when needed. Moon of Alabama detailed the problem back in May and last week the Wall Street Journal confirmed the issue. This also affects the older Boeing 737 NG.

While that problem has still not been solved a new one came up.

Boeing promised to release a software fix for the MCAS by April 2019. But that turned out to be more difficult than thought. Three month later there is still no final fix available. Meanwhile a new problem that will cause further delays was revealed only yesterday:

In a flight simulator last week, F.A.A. pilots tested erroneous activations of anti-stall software that pushes down the nose of the Max, two people with knowledge of the matter said. The software, known as MCAS, was involved in two crashes that killed 346 people.

In at least one instance, an F.A.A. pilot was unable to quickly and easily follow Boeing’s emergency procedures to regain control of the plane. The pilot rated that failure as catastrophic, meaning it could lead to the loss of an aircraft midflight, the people said.
…
The issue discovered last week is linked to the data-processing speed of a specific flight control computer chip, according to the two people with knowledge of the matter. In the test, the F.A.A. pilot encountered delays in executing a crucial step required to stabilize an aircraft.

It seems that the additional signal processing and calculations needed for the MCAS fix overload the Flight Control Computer's (FCC) processor and delay its reaction.

Boeing has been developing a software update for the Max for eight months, [a Boeing spokesman] said. It is unclear whether the new flaw can be resolved by reprogramming the software or requires a hardware fix, which would be costlier and could take much longer.

The 737 MAX has, like the previous 737 NG and Classic versions, two FCC's which each have two Central Processing Units (CPUs).

737 Flight Control System

bigger

As the former Boeing flight control engineer Peter Lemme wrote last year in a technical note of the issue:

Each FCC is comprised of two processors, each of which perform independently.

Each FCC has two 16-bit CPUs. The two processors have different part numbers to make sure that a design problem is not in both processors. The CPUs calculate different commands. …

In another note Lemme wrote:

The 737 FCC installation is a "dual-dual" configuration. Within each of the two autopilot computers there are two different processors, that each themselves are programmed by different people. The greatest threat is a common-mode software failure. Having two different groups program from a common set of requirements is a means to diminish a common mode failure.
…
The 737 dual-dual architecture is very unique. The decision to make speed trim single channel, single processor goes back to the 737 classic. The MCAS function is just another FCC software module that behaves, at a high level, like speed trim, whose architecture would have then been replicated.

The 737 uses only one FCC at a time and the Speed Trim System (STS), of which MCAS is a part, runs only on one of that Flight Computer's two internal processors.

The processors in question are said to be Intel 80286 type CPUs. The original Intel version of that CPU, sold between 1982 and 1991, had a maximum clockrate of 4, 6 or 8 MHz. It was later manufactured by a number of other firms, including by AMD and aeronautics company Harris, with a clockrate of 20 and 25 MHz. It is likely that the Boeing 737 FCC uses these or similar types.

These old processors are very reliable and error free. But they have less than 1/1000nds of the computing capacity of a modern cell phone. According to Lemme one CPU in the flight computer runs up to 11 different processes. All need to receive the input of external sensors, run through their algorithms, and signal a command to the relevant actuators that control the moveable flight surfaces of the plane. That the FAA pilot "encountered delays in executing a crucial step" caused by the computer points to a capacity overload.

Some decades ago your host programmed special input device drivers for Intel 80286 and alike systems. Their purpose was to record and process data from industrial process sensors, often hundreds at a time. Performance and timing issues required that the 80286 input drivers had be written in low level assembler language. But even with extremely optimized code the system would eventually come to its limits. The delayed procession of data from one sensor would eventually cascade into further delays and in the end the system would fail to record and process anything. The task was simply above its physical limits.

Flight control computer run special operation systems with minimal overhead. They are programmed in highly efficient programming languages. The software design and implementation follows a very strict process using specialized tools (see Green Hills' products for examples). All these are much better than what I used during my programming times.

Programs written for flight control purposes are already highly optimized. To further optimize them 'by hand' would break the regulated process that production of such software requires.

Boeing says that it can again fix the software to avoid the problem the FAA just found. It is doubtful that this will be possible. The software load is already right at the border, if not above the physical capabilities of the current flight control computers. The optimization potential of the software is likely minimal.

MCAS was a band aid. Due to the new engine position the 737 MAX version had changed its behavior compared to the older 737 types even though it still used the older types' certification. MCAS was supposed to correct that. The software fix for MCAS is another band aid on top of it. The fix for the software fix that Boeing now promises to solve the problem the FAA pilot found, is the third band aid over the same wound. It is doubtful that it will stop the bleeding.

The flight control computers the 737 MAX and NG use were developed in the early to mid 1990s. There are no off-the-shelf solutions for higher performance.

Boeing's latest announced time frame for bringing the grounded 737 MAX planes back into the air is "mid December". In view of this new problem one is inclined to ask "which year?"

—
Previous Moon of Alabama posts on Boeing 737 issues:

Posted by b on June 27, 2019 at 18:41 UTC | Permalink
Comments

To me, it sounds like the 737 reached the end of its expandability awhile back and Boeing has just gone too far trying to fiddle with it, all to avoid the costs of a brand new design which could fulfill the performance criteria they are trying to extract from the existing platform.

Posted by: adrian pols | Jun 27 2019 19:15 utc | 1

16 bit processors!
The last Intel chip with a 16 bit architecture was the 80286. It had a 1982 introduction date. That’s the ticket for passenger planes!
Maybe it’s a custom chip, from the MIC division of Boeing; which might mean it is 2017 vintage and cost the taxpayer $$$$$$ … but with that good old 1982 performance! Max clock speed on 16 bit processors isn’t very high either.

Posted by: BongBong | Jun 27 2019 19:24 utc | 2

It might be an I/O issue or RAM storage issue.
The 286 series has a 24 bit address bus which can handle up to 2 megabytes of RAM.
This isn’t necessarily a lot if there are multiple other programs running and lots of data coming in/going out.

Posted by: c1ue | Jun 27 2019 19:28 utc | 3

Great article, but we are getting rather technical. One point:
“MCAS was a band aid for an old plane type that had become aerodynamical unstable due to the new engine position on its 737 MAX version.”
The majority of pilots and engineers that I have heard say rather admittedly that the 737 Max is NOT aerodynamically unstable. But in order to avoid retraining pilots, Boeing added the MCAS system. So MCAS is not needed if you train and certify pilots to this type. Of course airlines would not like this; they want all pilots to be able to fly all variants of the 737.
You see too many media types are over their heads in terms of technology. But I do not mean B here, just in general.
This whole fiasco could be the downfall of Boeing commercial. They really need to get their act together.

Posted by: Meshpal | Jun 27 2019 19:36 utc | 4

The ‘Edsel’ 80286! That is amazing.

Posted by: bjd | Jun 27 2019 19:46 utc | 5

I would only disagree that code optimisation is not possible, everything else b wrote I agree with. But only people working on it would know if the code optimisation has any chance of delivering what is needed plus some safety margin and both in the worst case scenario of computing resource demand. The main question is would we, the passengers, trust that in such lose-lose situation such as this Boeing is not going to cut corners again? The only encouraging change is that FAA appears to finally be doing its job, although it is too early to be definite.
The comparison of FCC with the mobile phones is not adequate because the mobile phones do graphics which FCCs do not. This is why their 1/1000 of the mobile phone processing power is not so important. Even the best Western military jet F22 uses a similar, very low computing power processor. The advantage of old processors, such as 80286 is that their density is much, much lower and simplicity higher, hugely reducing a chance for glitches and instability.
It may be pointless to repeat that using a software solution to compensate for a major hardware inadequacy is like putting lipstick on a pig. It should have never been considered if the design was honest. Generally, software is good for enhancing hardware, adding extra features and most of all adding flexibility. But the concept of taking control away from the pilot to cover for an unstable hardware design is just simply out of this world, from the management land of greed and stupidity.
Interestingly, the most recent attacks of Boeing paid trolls online have focused on the issue of the unstable aerodynamic design. Obviously, few people will get into these planes again unless the perception management successfully spins this issue away, thus a lot of paid BS is being expanded on it.

Posted by: Kiza | Jun 27 2019 19:46 utc | 6

Incredible if they’re still using 80286s. My first PC in 1988 had one of those. They didn’t even have the full protected mode use of memory that the 80386 did. In fact I still have difficulty in believing it – we’re almost back to the 64K memory 8-bit computers that Apollo took to the Moon.

Posted by: Laguerre | Jun 27 2019 19:48 utc | 7

As the Boeing 737 Max controversy rolls on, the American planemaker has now been embroiled in a fresh row – after it was revealed it wants to shorten and replace some physical certification tests with software-powered processes.
Specifically, Boeing is “reducing the scope and duration of certain costly physical tests used to certify the planemaker’s new aircraft,” Reuters reported over the weekend.
We’re thus told that engineers working on Boeing’s new 777X airliner project are intending to use computer simulations instead of real-world flight tests to validate their engineering decisions.
The manufacturer wants to switch to software-based trials for things such as wing load testing, according to three sources who spoke to Reuters, instead of doing things like bending actual, and highly expensive, components until they snap.
https://www.theregister.co.uk/2019/06/17/boeing_certification_simulation_tests/

Posted by: John Smith | Jun 27 2019 20:01 utc | 8

This isn’t Boeing to end well: Plane maker to scrap some physical cert tests, use computer simulations instead
Actually probe expensive gear in real life? Pah. It’s 2019. We’re Boeing digital

Posted by: John Smith | Jun 27 2019 20:03 utc | 9

Boeing’s criminal track record seems to be a small sample of Corporate America, the focus on short term profits no matter what creates the background for justifying crimes, such as the most likely bribes to speed up the process inside FAA among many others. Boeing only reflects the leadership of the country for the past 4 decades, lies and fake justifications that killed millions in Asia and Africa.
I have been saying on twitter and GAB that Boeing’s 737MAX tab will be in the hundreds of billions of USD, pilots’ law suits, murdered families’ law suits (yes the 2 accidents were murder since the company knew about the defect and what it did to cover it up), should the 300+ planes stay grounded for over 1 year, this tab will be higher.
Boeing is toast and a goner, it could not have happen in a worst time…a long depression is coming and the US regime won’t be able to bail it out as done to GM and others in the past.
Expect the worst.

Posted by: Canthama | Jun 27 2019 20:06 utc | 10

Yes, they’re surely not programming it in anything resembling assembler or C etc. It must be something like PLC logic (but without the hieroglyphic form, one hopes).
I don’t work on aerospace systems, but wild guess is they maxed out the I/O capacity long ago and had to “get creative” to work around that (repeatedly complicating the overall system, multiplying the special cases, rather than ever paying the cost of a CPU/PLC upgrade – generally a prudent choice, until it wasn’t.)
This will make a great systems design case study.

Posted by: Ptb | Jun 27 2019 20:07 utc | 11

The limitation of the processors as described indicates to me a possible reason Boeing allowed only one input sensor for the MCAS design. It seems to me they probably knew they did not have the processing power to manage two or more inputs. This also leads me to wonder about the architecture of Airbus designs.

Posted by: Michael | Jun 27 2019 20:08 utc | 12

But the concept of taking control away from the pilot to cover for an unstable hardware design is just simply out of this world, from the management land of greed and stupidity.

Kiza @6
Best quote!

Posted by: Vasco da Gama | Jun 27 2019 20:11 utc | 13

@c1ue It might be an I/O issue or RAM storage issue.
I suspect I/O. The 80286 uses I/O interrupts. When a signal appears on the I/O interface (when someone presses a key) the processor interrupts the running process. The process’ current register values are pushed onto the stack. The handling routine for the I/O incident comes in. If there are too many I/O interrupts the original process has no chance to return. More I/O interrupts occur, more then current process data is pushed onto the stack, which thereby grows and in the end corrupts the rest of the memory. (I wrote my own memory handler and time slice OS to get some control over that phenomenon but it was iffy.)
The 737 has ADIRUs (Air data inertial reference unit) which sort out the sensor signals and do some preliminary calculations to avoid that problem. But even with those crutches there is some hard limit of what the CPUs can do.
@Meshpal say rather admittedly that the 737 Max is NOT aerodynamically unstable.
That is correct. I even pointed that out in my earlier pieces but tried to take a shortcut with this one. I should not have done so. I have now changed the wording.

Posted by: b | Jun 27 2019 20:12 utc | 14

I fell into a YouTube hole a couple of nights ago and came across this: https://m.youtube.com/watch?v=24Fe1wMy-9Y
A 737NG crashes nose first and it’s blamed on an optical illusion supposedly, although the 2 pilots both have extensive experience. There is mention how the pilots couldn’t right the plane due to aerodynamic forces with increased speed but I guess investigators didn’t link the reduced trim wheel size.
Learning more and more about Boeing’s criminal negligence makes me suspicious about this crash too. There were a lot of things happening “automatically” based on plane’s air speed and I wouldn’t be surprised if there were automatic systems that investigators were left unaware of.

Posted by: Murad | Jun 27 2019 20:14 utc | 15

Trapped in simulation: Can computer models replace real testing of passenger aircraft? — RT
Boeing seeking to reduce scope, duration of some physical tests for new aircraft – sources — Reuters

Posted by: John Smith | Jun 27 2019 20:15 utc | 16

Between 1997 and 2007, instead of investing a new 737-type aircraft and designing the 787 properly Boeing paid out $20 billion to repurchase shares and splashed out $8 billion in dividends (Lazonick). Boeing’s share price rose around 170% in the same period. Share buybacks artificially increase the share price by withdrawing shares from the market.
In answer to questions I had asked for a project I was doing, a member of the Boeing engineer’s union SPEEA wrote:
The 787 is not a Boeing aircraft. It is a fusion of disparate designs, some well-engineered, some very badly, all patched together under a pleasing-looking exterior. When you contrast this with the 777, which was a work of near-perfection, it makes you want to hold up your hands and scream. This aircraft could encounter serious problems in any one of a number of scenarios. It is a disaster waiting to happen.

Posted by: Lochearn | Jun 27 2019 20:16 utc | 17

Leeham is talking of another year
https://leehamnews.com/2019/06/27/lessors-airlines-seek-1-year-lease-extensions-in-max-groundings/#more-30559
The question of stability vs handling is hard to pin.
The stability margins at prestall were not so good on the max, but at the same time MCAS also rendered pilot feel. Someone more experienced would have to say if the original stability margins were unacceptable from a purely safety point of view, and if so if (working) automated adjustment is an acceptable/standard reply.

Posted by: gzon | Jun 27 2019 20:16 utc | 18

Re the post by Michael @10 “The limitation of the processors as described indicates to me a possible reason Boeing allowed only one input sensor for the MCAS design.”
this would also be the reason an AOA fault warning was an optional extra at large cost. The cost was most likely to do with the computer upgrade required. The AOA warning being the input straw that broke the CPU camels back.

Posted by: Peter AU 1 | Jun 27 2019 20:20 utc | 19

Taking the input from only one AOA sensor was the oddest thing about the MCAS system. Although only one computer was ever used for flight, the one being used always cross checked inputs from several sensors and I think the other computer to check that they matched, whereas the MCAS program took inputs from only one AOA sensor.

Posted by: Peter AU 1 | Jun 27 2019 20:25 utc | 20

I don’t see Boeing regaining any of the ground it’s lost on the mega-important Trust Factor when it comes to passenger airplanes. The lawsuits have begun to be filed and their publicity will further erode public perception of both Boeing and FAA via items found during discovery that haven’t been publicized.
William Gruff wrote about incompetence on the “mistranslation” thread where he referenced Boeing as an example. As with GM, Boeing will likely be bailed-out by US Taxpayers in yet another twist of the Socialism that permeates the US political-economy no one wants to talk about except for Sanders and Gabbard.

Posted by: karlof1 | Jun 27 2019 20:26 utc | 21

Boeing’s Latest 737 MAX Concern: Pilots’ Physical Strength — The Wall Street Journal
Turning manual crank during emergency procedure may be too difficult for some people

Posted by: John Smith | Jun 27 2019 20:27 utc | 22

I would think that by now a company like Boeing would have a little more modern chips. There are many just as dependable if not more. Gee , even a 486 would be an improvement. The two most prized words in the US today are ‘cheap’ and ‘cheaper’. We really need to get back to ‘quality’. At least a little bit…..

Posted by: ken | Jun 27 2019 20:31 utc | 23

Boeing Has So Many Grounded MAXs, They’re Parking Them in the Employee Lot
The headline really says it all: Boeing has so many MAX planes on hand at their Renton Factory in Washington State that they’ve had to move some of them to the employee parking lot. Driving into work everyday to see grounded planes in what used to be your parking space has to be a constant, uncomfortable reminder of the position that Boeing is in after two of their planes crashed within months of one another. And, there are a lot of grounded MAX planes taking up space in Boeing’s employee parking lot.
Seattle’s King 5 News has some great aerial footage of the factory, which shows just how many planes have taken over employee parking spaces. There are about 500 grounded MAX’s all around the world, and 100 of those are at the Renton factory. A few dozen of them have been making their rounds on social media as the MAX debacle drones on.
https://www.youtube.com/watch?v=46InmJexzYg

Posted by: John Smith | Jun 27 2019 20:33 utc | 24

Meshpal & b
From what I could make out earlier, the stall problem on the max was not so much the AOA and speed at which it stalled, rather what happened if it did stall. It seemed that if the max did stall it would be catastrophic as tail would drop first rather than the nose due to the lift generated by the engine nacelles at high AOA.

Posted by: Peter AU 1 | Jun 27 2019 20:42 utc | 25

Boing should cell the entire production facility to China for a high price including 5G technology, and retool a new facility, new everything and produce a proper low fuel jetliner to compete with Airbus.

Posted by: El Cid | Jun 27 2019 20:46 utc | 26

Not “cell” but sell.

Posted by: El Cid | Jun 27 2019 20:47 utc | 27

No doubt the intrepid bosses at the airplane company will twig to the next stage of band-aid, and simply increase the clock speed and dribble LN2 on the chips…
Boeing blew it…and if trumpie the clown is mouse-trapped into “doing” Iran…nobody will care about the flying pigs
Or if the clown act avoid the mouse-trap?
Then the dollar goes anyway…
Either way the airplane company is walking dead…
Sell it to the Ruskies for beer money…
Every wild party comes to an end…

Posted by: Walter | Jun 27 2019 20:54 utc | 28

It was later manufactured by a number of other firms, including by AMD and aeronautics company Harris, with a clockrate of 20 and 25 MHz.

So far as I can recall, I’ve had only one desktop computer with a CPU which ran that slowly. Asking an antique like that to run more than one efficiently written program is insane. To have such a device in a crucial location within a modern airplane is beyond crazy.
One other thing Boeing is doing strikes me as Dollar Wise, Penny Stupid.
Behind Boeing’s offer to settle with victims’ families in a 737 Max crash is a hardball legal strategy that could leave them with nothing
What the penny-pinching accountants don’t understand is that people are watching the Boeing situation closely. The bean-counters got them into the current mess, and stiffing the families of victims killed by the company’s incompetence and greed is a strategy which bets passengers everywhere aren’t paying attention. The stakes of the gamble is the future of the commercial division of Boeing.
Boeing screwed up – big time, and they’re going to have to put aside their dividend payments and stop the stock buy-backs for a while. If the fools decide to do otherwise, I won’t be grieving when they turn into a much smaller airplane company.

Posted by: Zachary Smith | Jun 27 2019 21:03 utc | 29

regarding post #27, make that Penny Wise, Dollar Stupid.
🙁

Posted by: Zachary Smith | Jun 27 2019 21:08 utc | 30

No worries. After WW3 irradiates Eurasia Operation Paperclip 2.0 can proceed cuz stealing evil geniuses is what Anglo American capitalism is all about.

Posted by: C I eh? | Jun 27 2019 21:11 utc | 31

It is plain to see that this plane will not fly again. I would not board this sucker, would you? Could this pile of scrap not have the wings cut off and the fuselage used as Greyhound buses or train cars? Maybe Elon Musk the grifter can do something with it on Space-X. This piece of crap is done, period.

Posted by: Taffyboy | Jun 27 2019 21:13 utc | 32

2014:
America’s Nuclear Arsenal Is Still Controlled By Antique Computers With Floppy Disks
2016:
America’s nukes are still controlled by 8-inch floppy disks
A hacker explains why US nukes controlled by ancient computers is actually a good thing

Posted by: John Smith | Jun 27 2019 21:17 utc | 33

Boeing and its 737 MAX is FUBAR.

Posted by: AriusArmenian | Jun 27 2019 21:21 utc | 34

It would be nice if this also completely erodes any trust in the FAA who have always been more than happy to blame pilots or tout government agendas and then help propagandize it in “reality light” TV shows about the accidents and not-so-accidents. They’ve got thousands of skeletons in their closet.

Posted by: Sunny Runny Burger | Jun 27 2019 21:22 utc | 35

For me, Boeing in its heyday represented the heights of productive as opposed to financial capitalism. The creation of the 747 was an enormous gamble that could have and nearly did sink Boeing. Rather than a project that made its way up through a chain of bureaucracy, decision-making on such a huge project was down to just two people.
Legend has it that Boeing CEO Bill Allen and PanAm CEO Juan Trippe, on a fishing trip in Puget Sound, started discussing the building of a very large aircraft that could transport people over long distances economically. Trippe said “ Would you build it if I bought it” to which Allen replied “ Would you buy it if I build it”. Over more informal fishing trip meetings Allen and Juan Trippe reached an agreement that changed the way the world traveled.
The brilliant chief engineer of the 747, Joe Sutter, stated: “People that study the hell out of a problem, like they do now, would have determined the 747 wasn’t worth the effort. It required Trippe’s and Allen’s vision. If they hadn’t been in place, the 747 wouldn’t have happened.”

Posted by: Lochearn | Jun 27 2019 21:44 utc | 36

There is quite a bit of knowledge here in the MOA comments. However, it does not appear to include embedded computer systems, which the MCAS ‘computer’ obviously is. Nope it’s not an 80286. It’s likely one of the thousands of choices from companies like Microchip, TI, NXP, Infineon or any of a dozen other embedded processor makers. Embedded processors come in many forms, including 8 bit (!), 16 bit, 32 bit and 64 bit. I do suspect Boeing is using an older (10 years or more since introduction) 16 bit embedded chip which is obviously under-configured. Changing out the chip is non-trivial. I suspect that if the 737 Max ever does fly again, it won’t be for several years. Oh and it was most likely programmed in C, the primary language for all embedded development.

Posted by: Don Wills | Jun 27 2019 21:47 utc | 37

@Michael – 10:
“The limitation of the processors as described indicates to me a possible reason Boeing allowed only one input sensor for the MCAS design.”
Nope! A decent design would use an additional CPU in the middle to process both sensor signal and send the result to the main CPU. If both sensor yield similar values, it would pass them right through, if they yield differing values, it would signal a fail.
Additional costs of goods: some cents.
Notwithstanding, the MCAS might generate some additional computation load, that cannot be outsourced to some peripheral CPU.

Posted by: Linus | Jun 27 2019 22:09 utc | 38

Its the same work model used by all the Aerospace Contractors for US Military and now Civilian aircraft. Opps, we totally fucked up. Guess the cost of each plane may double. Gonna need a total retrofit with new computer and auxiliary equipment. No worries, we can fix it. Don’t look at the costs, them of them as investments in the future. What you mean someone planned to use this piece of garbage? It’s a Carnival, Kabuki Theater and Reality all at the same time. God these people are so fucking stupid its an embarrassment to humanity.

Posted by: Delta G | Jun 27 2019 22:21 utc | 39

Boeing’s goal with the Max was to avoid pilot retraining and simulation to save time and cost.

Posted by: Jake | Jun 27 2019 22:23 utc | 40

Re #34
Well Don, whatever knowledge there is or is not here about embedded systems, you have done nothing to improve it. You apparently felt overwhelmed to comment despite the fact that all you are doing is guessing.
There is no reason why an x86 device can not be used in an embedded application and has been used for that frequently. The article specifically states that a 286 was used. If you know better, please provide references.
Many of the devices and companies you opine about did not exist when the FMC for the 737 was developed and old is not necessarily bad. I can not confirm if the Boeing supplier uses 80286 or not, but the latest similar FMCs use the Motorola 68000, another personal computer processor you would “never” use in an embedded application.
Thanks for playing.

Posted by: Pragma | Jun 27 2019 22:26 utc | 41

One good reason to use a 80286 is that they are very sturdy and in particular resistant to stray cosmic radiation at high altitude (they are widely used in satellites for the same reason).
I rather suspect it is some young buck fiddling with routines that shouldn’t be fiddled with, the expensive greybeards who knew how everything worked having been laid off or possibly retired years ago – to save money, natch.

Posted by: DomesticExtremist | Jun 27 2019 22:28 utc | 42

On a brighter note, this could spell the death knell for autonomous vehicles and widespread robotisation more generally.
The legal fallout from any widespread software failure, especially one that makes those ugly bags of mostly water (aka humans) dead in large numbers and the subsequent corporate failures will quickly dampen our overlords’ enthusiasm for such a dystopian future.

Posted by: DomesticExtremist | Jun 27 2019 22:39 utc | 43

Oh well … if Boeing 737 MAX jets are to be grounded for another year … and another year … and so on … they can always find a new lease of life as something else.
China’s first aeroplane restaurant in retired Boeing 737 in Wuhan
Ghana’s DC-10 Restaurant
India’s Airbus320 Restaurant
Boeing 707-348C restaurant in Damascus suburb, Syria, way back in 2006
At last aeroplane food doesn’t have to taste so bad.

Posted by: Jen | Jun 27 2019 22:49 utc | 44

@34
Asm and C are systems languages, and as such well suited for bottom layer of any embedded app, but i’m sure you know they have many unnecessary pitfalls for app logic. Even routine math is a chore if you want to do it 100% safely. To consistently make reliable code, you end up with something else, whether it be (1) a collection of macros to wrap pretty much every primitive op of the cpu and/or the low level language, or (2) a proper interpreter for a formally specified language and/or virtual machine (such as in PLC’s, for which I have no love, but they exist for a reason).
In any case, the chip and the io hardware it connects to are, I’m fairly sure, an inseparable part of the package whose replacement and re-validation were deemed too risky and costly.

Posted by: Ptb | Jun 27 2019 22:49 utc | 45

Re #40
DE:
As much as I would like to agree with you, I can’t. To the psychopaths, it is a numbers game.
I think the whole autonomous vehicle thing will proceed apace as long as the number of people killed and maimed by AVs is less than by humans. Considering the carnage on the roads, that shouldn’t be difficult. The biggest challenge will be the optics and spin.
Aviation is a different story, because the operators are professionals with a pretty good record, and unlike a car, an airliner can’t just roll to a stop if something goes sideways.
It will be interesting.

Posted by: Pragma | Jun 27 2019 22:50 utc | 46

I have a solution for the manual trim wheels. Airlines just need to hire weightlifters to be on all flights and available to turn the trim wheels when called on by the pilots.
No wonder they say it would take us 20 years or so to land a man on the moon , should we choose to do so again. Declining IQ’s are really being felt now. Not to mention the decline in morals and ethics that facilitate corporate and government joint malfeasance.

Posted by: Pft | Jun 27 2019 22:54 utc | 47

Linus @35,
Now the value combiner is a single point of failure. Not only that, it can’t tell which AOA sensor is lying without access to other observations of flight conditions and parameters. Back to the drawing board!

Posted by: Jonathan | Jun 27 2019 23:15 utc | 48

I laughed at Posted by: El Cid | Jun 27, 2019 4:46:39 PM | 24.
I had this image of Boeing getting back to basics and making tea strainers after the sale of their aircraft division to China.
Thanks for a solid analysis b and all the friends here. I would have thought a 486 to be a good answer to the data crowding. Why is that so hard? Any sane corporation would be carrying out succession planning at all levels of its technology and even more so when it is pressed by solid competition.
The penny pinching pay back has arrived for Boeing while their socialist comrades in government prepare a people’s bailout.

Posted by: uncle tungsten | Jun 27 2019 23:45 utc | 49

So what to do with all the 737-max already produced and those under construction?
How about turning them in to luxury apartments for the wealthy . It has electrical, wifi, entertainment, kitchen, bathrooms. Need to install showers, tubs, stoves and redo the interior. I bet they could sell them for 10 million or more. Maybe sell off the engines for another 10 million (perhaps to be used on a replacement model Boeing)
Sure they take an 80 million dollar haircut but the losses can offset future taxes on profits
To help them sell the house planes the government could issue the buyers tax credits.
Alternatively they could sell the planes to the military to be used as military transports. They would not need the MCAS in that case as pilots would be trained to fly the plane as is. The need to minimize pilot training costs to fly the same as older 737 would not be an issue.
They just need to offer cushy jobs to enough folks after retirement (politicians and officers alike), and buy a few of Trumps condos.
So dont write off Boeing yet. They are TBTF. Bailing them out will be a lot cheaper than bailing out the banks. Heck they give Israel 4 billion a year. That can buy over 30 Boeing 737 Max a year with that money and take on all of Boeings past deliveries and have them paid for in 12 years. Tough for Israel though , although maybe the US can send some to them to be used mobile housing for their settlers .
Ok, maybe too much sun today, but somebody will think of something

Posted by: Pft | Jun 27 2019 23:52 utc | 50

I’m pretty sure it has been mentioned here on MoA before, but one of the big selling points of the 737 MAX was that it shared a type certification with traditional 737s. Even swapping out a toggle switch on the control panel for one from a different manufacturer requires an appendix to the certification that itself has to be certified. Swapping out the processors on the FCC would require a mountain of re-certification work all by itself. Maybe that isn’t enough to require a whole new type certification, but along with new engines and another change or two you probably do start getting close to needing a new type cert.
There is a reason why brand new Cesnas still fly with engines designed many decades ago.

Posted by: William Gruff | Jun 28 2019 0:01 utc | 51

Pft #47 Absolutely brilliant idea, re-badge it as ‘the Golan’ and give it to the Israeli settler movement as a peace gesture. There would be a hollywood movie to package it just right for the USA audience.

Posted by: uncle tungsten | Jun 28 2019 0:02 utc | 52

It’s funny – in a sardonic way – to read the Moon of Alabama pieces and ensuing comments on the Boeing 737 scandal and then compare this knowledgeable commentary and discourse with the vacuous “Business Report” on my local MSM AM news radio station (that I listen to for traffic updates on my way into work every morning). If all I ever consumed was this “Business Report” on my way to work, I would be under the impression that the problem with the Boeing 737 Max was merely some kind of “glitch” and that the crashes it caused were only meaningful in terms of Boeing’s stock price. The subliminal message is that Boeing is the victim here and we should grieve that their planes could remain grounded for another “1 – 3 months.”
I grinned when I heard “1 to 3 months” – again, sardonically. Thanks, MOA, for helping to allow me to know better.

Posted by: Activist Potato | Jun 28 2019 0:25 utc | 53

Lemme guess: they used the cheap coders.

Posted by: Anon | Jun 28 2019 0:58 utc | 54

The reason avionics uses such primitive if you will hardware is the requirement for robustness and reliability. It must be able to work in high vibrations and temperature and pressure extremes. On top of that you have stray cosmic particles, ham-handed maintenance, and plenty of cycles of bump and grind. So the newer architecture of components has to be ruggedized and proven reliable before it can be used. As we know you cant pull over to the side of the road if it breaks.
Regards, Rob a retired 30 year avionics tech.

Posted by: viking3 | Jun 28 2019 1:06 utc | 55

Thanks for clarifying the muddy media reporting. Boeing took the cheap route to increase CEO bonuses and shareholder value by adding a new function, MCAS, to an old inadequate flight control system that adjusted the trim on the 737 NG. This saved recertification and pilot training costs. A toxic work culture ignored the safety concerns due the added costs of modern triple redundant unit flight control system. Boeing is TBTF. Pressures are building. If pilots could easily control the plane during takeoff without MCAS, it would be flying today. The Max must require an automatic flight control system to avoid crashing without waiting for the pilots to sort things out. A new system would take year(s) to design and install.
It is shocking how everything is culminating to a breaking point; the trades wars, Boeing. and the War with Iran. Corporate media can’t hide it any longer. FEDEX said that the trade wars are a Mike Tyson blow to its bottom line. The professional 10% are starting to realize they are being screwed by the oligarchs and if their families are to survive, they have to do something. FAA pilots reported the latest Max glitch. Sully Sullenberger got to fly a patched Max Simulator and reported to Congress that pilots must have Simulator training before flying the airplane. The acting Border Control chief resigned. Forced migration is overwhelming Europe and North America. The lure of the military contractor revolving door is enticing; but there are no safe places, supplies or troops to invade Iran. If attacked, thousands of ballistic missiles will blow up the Middle East.

Posted by: VietnamVet | Jun 28 2019 1:15 utc | 56

Anyone with a gram of brains would realise quickly the BS of the Boeing’s latest spin point. It is just not clear why b is falling for it (maybe a concern about being sued). Ok, so Boeing put MCAS in to create a pilot feel similar to the previous 737 and thus reduce the need for training/retraining. And the way to make the feel similar was to take the control away from the pilot (ROFL). What kind of cretin do you have to be to accept that control is taken away from the plane’s principal without a need to compensate for some major structural problem (comparison with autopilot is irrelevant)? Yet a fair number of years in sales and marketing have thought me that logic has nothing to do with a success of a spin point, only the amount of money spent. Boeing has pockets deep enough to create its own truth and its own reality in people with switched off brains.
Oh, and just as a mirror image to the troll here, I think that Boeing is a wonderful company but I will aim to never sit in one of their planes again because those with switched off brains should be given a chance to be culled out.
Ultimately, this is not a story about supposedly obsolete computer processors and faulty software, it is a story of how chomping at performance margins in life-critical systems will always lead to disasters of varying scale (close to 400 dead here). In other words, I am sure that any remaining capable Boeing engineer would suggest getting rid of the mush for brains MCAS altogether, because it should have never been there in the first place (the risk of giving full control to an automatic system reliant on one failable sensor clearly outweighs the risk of pilot error to stall). But try to imagine the outcry and the lawsuits that would follow such announcement. Therefore, fixing MCAS (the straw that broke the FCC camel’s back) correctly is probably not possible because MCAS is now baked in politically. The scumbags who forced it in to try to cover their management asses are not going to reverse themselves: “it just needs to be improved in the forthcoming version 7.5”.

Posted by: Kiza | Jun 28 2019 1:32 utc | 57

I flew as captain on 737 NG for major US airline for four years before retirement in 2008. Before that I flew the Boeing 727, 757 and 767. All were superlative airplanes. Wish I’d had a chance at the 787 and/or 777, but age intervened according to the FAA. BTW, at age 70 I think I could still do the job so put that in your pipe and smoke it. In decades past I believe Boeing had the expertise and leadership to right the ship and get back to business. Now the FAA/DOT bureaucracy is exposed for all to see and I’m thinking there’s a good chance Boeing implodes and takes the US economy with it. I hope I’m wrong.

Posted by: Vic | Jun 28 2019 1:35 utc | 58

I wonder what % utilization the FCCs were running at prior to the introduction of the original MCAS and the MCAS fix?…Seems like it must have been pretty high, MCAS doesn’t seem like it would require a huge amount of computation.
Also: Possibly a bad idea to let the FCCs handle the manual trim function in addition to everything else it’s doing–why not a simple relay circuit or semiconductor equivalent to execute the manual trim commands (control wheel thumb switch) regardless of what the FCC is doing?

Posted by: David Foster | Jun 28 2019 1:35 utc | 59

Pragma wrote “The article specifically states that a 286 was used.” Which article is your source for this statement? I can’t find that information anywhere. Please advise.

Posted by: Don Wills | Jun 28 2019 2:06 utc | 60

https://en.wikipedia.org/wiki/Intel_80286
“Produced by Intel from 1982 to 1991”.

Posted by: Don Wills | Jun 28 2019 2:17 utc | 61

@ # 57
The one at the top of the page written by b. The one that everyone is commenting on.
“The processors in question are said to be Intel 80286 type CPUs.”

Posted by: Pragma | Jun 28 2019 2:18 utc | 62

Perhaps Boeing should try some older, proven technology, like having an ancient analog screen for each sensor and task the co-pilot to watch them and tell the pilot if anything looks wrong. Humans can integrate several inputs, you know. And for trimming the tail flaps under high speed, it seems that they should have a capstan in the rear of the plain, operated by the cabin crew during an emergency. Capstan could be located in the plane kitchen, so the cabin crew would have to be train in pulling the capstan during emergency and to hop ever it during normal operation.

Posted by: Piotr Berman | Jun 28 2019 2:23 utc | 63

@Pragma #38
I would suggest you read up on the differences between a microcontroller and a microprocessor.
Among a few of the differences: a microprocessor has an operating system, has memory, has peripherals and has a bus. A microprocessor can handle more than 1 process at a time.
Yes, there are microcontrollers that a closer to microprocessor functionality such as the ECUs in a car (with the CAN bus), but microcontrollers are generally used for well understood processes that aren’t expected to radically change beyond an original specification.
I don’t know if the 737 uses a x286 for sure, but it makes sense since the control systems for an airliner can be expected to be under constant update and development.
In any case, I speak of this from the microprocessor side – I was actually on AMD design teams in the late 90s.
Perhaps you are a microcontroller expert, but then again, perhaps not.

Posted by: c1ue | Jun 28 2019 2:27 utc | 64

@ David Foster | Jun 27, 2019 9:35:36 PM #56
I’m so cynical these days – my first thought upon learning of this second flaw was that it might be a setup-job.
Newly vigilant FAA discovers serious flaw.
Humbled Boeing scrambles to fix it.
FAA looks it all over, declares all is well.
Result – FAA is again trusted to be on the job, and their blessing of the 737 MAX is accepted by all the world.
Your suggestion to disconnect the problem feature and connect it to something else might be a ‘solution’. Doing so would relieve the computer chip of the excessive workload, and likewise declaring another of the jobs it has been doing as “non critical” would allow the FAA to declare “all is well.”
Getting the airplane back in the air is Job One for all concerned, and if a few song-and-dance routines would help with that, we’ll be seeing them.
Yes, I’m cynical, especially when I run into sentences like this:

The problem occurred during a scenario that commercial pilots are highly unlikely to encounter, and doesn’t involve the flight-control software linked to the two 737 Max crashes, according to one of the people. However, the resulting diving motion created by the runaway trim was similar to the problem faced by the Lion Air and Ethiopian Airlines pilots on the flights that went down.

Not only would MCAS be fixed, but the “highly unlikely” concern is also gone.
All is well.
link

Posted by: Zachary Smith | Jun 28 2019 2:33 utc | 65

@ # 59
Link please. I still can’t find any reference to 286 or 80286 in any article anywhere. I will be happy to acknowledge I’m wrong if any semi-authoritative source mentions that ancient chip.
Note that the MCAS computer and the FMC are almost completely independent of each other.
On Googling around I did find that the original FMC on the Airbus 320 did use the 80286. That plane was certified in 1986 and continues to be produced. The article went on to say that the 80286 was replaced early on when the 80286 went out of production in 1991.

Posted by: Don Wills | Jun 28 2019 2:58 utc | 66

It is interesting to note the difference in posting and comments from The Register about this new information
https://forums.theregister.co.uk/forum/all/2019/06/27/boeing_737_max_control_bug_found/
The interesting tidbit from the Register comment conversation is that the 737 MAX is a flyable plane, albeit long in the tooth, and would not have had these failures if pilots had been trained properly but…….profit
Yes, I grew up in Tacoma when Boeing was a reputable company but it has been financialized like most everything and China is showing a better way by taking profit out of core things and regulating other sectors appropriately.
Private finance may survive the steamroller of China but it will not be the same jackboot (and company killer like Boeing) as now.

Posted by: psychohistorian | Jun 28 2019 3:05 utc | 67

@ # 63
Don, I don’t understand what you are asking me.
You stated firmly that the processor was NOT a 286, but if you scroll to the top of your screen, there is the original article from MOA about the Boeing 737 problem that we are all talking about here. It clearly states that a 286 was used. There is no external article that I am referencing, just what’s on this page. It is unclear to me whether the processor in question that is getting overloaded is in the the FCC or the Elevator Feel computer but the only device mentioned is the 80286.
I don’t know how I can make it any clearer.

Posted by: Pragma | Jun 28 2019 3:30 utc | 68

MCAS stands for Maneuvering Characteristics Augmentation System. What kind of f’ing “augmentation” is it to take the control away? More appropriate name: Maneuvering Characteristics Hijack System.
Solution?
Convert into MCMS – Maneuvering Characteristics Monitoring System. It should only be monitoring sensors (notice the plural) and warning pilots about the possibility of stall, not controlling anything. Cutting out the control functions, would cut out 2/3 of the processing load and would have a chance to satisfy the basic performance formula: processor_capacity – worst_case_other_load + MCMS_load > 30% (safety_margin). Top of the envelope, but probably not too far off the mark.
This would not remove the problem of the inherent dynamic instability, but it is probably the best “fix” possible considering that these planes must fly again once the spin and the memory hole do their work – wanna bet?

Posted by: Kiza | Jun 28 2019 3:31 utc | 69

When Boeing moved it’s headquarters from Seattle to Chicago the writing was on the wall. Now we see the fruits of their changed business philosophy.
Interesting how Russia has managed to modernize their military hardware to competetive levels over the past 20 years while spending less than 1/10 the funds available to the US military.

Posted by: the pessimist | Jun 28 2019 3:33 utc | 70

Oops forgot the brackets: processor_capacity – (worst_case_other_load + MCMS_load) > 30%

Posted by: Kiza | Jun 28 2019 3:43 utc | 71

@ #61
c1ue:
I’m not sure if I’m being trolled, but for now, I assume not.
A microprocessor is a single chip, usually, that contains just a CPU with the necessary data, address and control lines to fetch and store both instructions and data. Depending on whether it is Harvard or VonNeumann archtecture, it can have control lines for I/O functions as well. Examples of a microprocessor are the Intel 4004,4040,8008,8080,8085 and the x86 family, but it also includes AMD products, that emulate the Intel family and Motorola products such as the 6800 and 68000. The list goes on…
A microprocessor does NOT have memory, except for L1 and L2 cache on more recent designs, which are used only for intermediate storage and prefetch. Microprocessors do NOT have peripherals, such as UARTs, unless a designer wants to attach them via the data and address lines, or more recently via PCI lanes. A microprocessor can NOT handle more than one process at a time unless it is multi-cored (multiple CPUs) or it uses something like Intel hyper threading which presents 2 virtual cores attached to one physical core. To do this, it must have an operating system that is multi-thread capable.
A microcontroller, on the other hand, usually has a CPU, memory, peripherals and I/O lines all in one package, usually on one chip and are often referred to as SOCs, or System On a Chip. Usually they do NOT have an external bus and are limited to the on-chip memory, both for data and program, unless one uses a serial flash device like an SD card. An example of a microcontroller is the AT family by Atmel (now Microchip) that is used in Arduino products, and most ARM based products. The best known microcontroller by Intel is the venerable 8051 which is still found today. Recent microcontrollers can also have multiple cores such as the Snapdragon, and can execute more than one process at the same time. They can also have sophisticated o/s’s like Android.
Neither devices inherently have an operating system, although the internal microcode of recent CPU’s is very sophisticated. The o/s is at the discretion of the designer, for embedded products, or at the discretion of the manufacturer, or even the end user in the case of smart phones or personal computers. Embedded systems usually have a minimal core to handle context switching but are kept lean for best real-time response although it’s not a hard rule, as microcontrollers get more powerful.
What you are describing as a microprocessor is a typical system configuration for just about any computer, from a PC to a mini to a mainframe.
My apologies to MOA if this is too far off topic.

Posted by: Pragma | Jun 28 2019 3:53 utc | 72

Knowing almost nothing about computer programming, I assume that b’s knowledge of programming and the limits imposed by the clock-speed of a CPU are based on hands-on experience in working around those limitations.
However, from an outsiders pov, it seems that the flaws built into the MCAS from Day 1 only came to light BECAUSE of Computer Confusion caused by a faulty Angle Of Attack Sensor.
The factor which still amazes me is that tens of thousands of 737 MAX flights were completed safely and without incident. And one assumes that had no AAS failed then the 737 MAXs would still be flying happily and incident-free. I agree with the “band-aid on a band-aid” characterisation of this stage of solving the problem. I’d be curious to learn if any 737 MAXs with functioning AAS ever encountered conditions which resulted in the SUCCESSFUL activation of MCAS?
(PS: I haven’t read all of the comments)

Posted by: Hoarsewhisperer | Jun 28 2019 3:56 utc | 73

Good for Boeing. Nothing like opening up the market to competition.
https://www.youtube.com/watch?v=VuiYKUnanm0
I do not expect the MC-21 to take over the world market but airlines in any country under US sanction (The world minus the Vatican?) might be looking at such an alternative?

Posted by: jrkrideau | Jun 28 2019 6:42 utc | 74

In Dutch, program code meant to cover up a flaw in the underlying system is known as “sjoemelsoftware” (cheating software) since the VW emissions fiasco. Appropriate here?

Posted by: Ma Laoshi | Jun 28 2019 7:30 utc | 75

Mespal @4 & b @12
I have read before that the 737MAX is not areodynamically unstable and that therefore the MCAS is not really required, and that the only reason for the MCAS is to avoid the cost of retraining pilots.
But this really makes no scense to me at all. If this is the case then the MCAS is just a totally unnecessary level of abstraction which could fail and, in such an event, the pilot would have great difficulty in controlling the plane solely due to lack of training in piloting the 737MAX without MCAS.
And the solution to the 737MAX would be straight forward. Remove the MCAS and require that pilots are trained in how to fly the 737MAX without the MCAS.
Always going forward the MCAS will be a problem, and in times of failure cause far greater difficulty for the pilot.
Yet still Boeing persists with MCAS, when they could free themselves from this nightmare by just abandoning MCAS.
Ask yourself if you were a passenger would you prefer to be flown in a 737MAX without MCAS with a pilot suitably trained and compare that to how you would feel being flown in a plane where a completely unnecessary piece of software could fail and leave an untrained pilot at a loss about what to do.
So, the conclusion I would draw is that there is more to it. That aerodynamic stability doesn’t necessarily mean acceptable for pilots to fly the plane.
There is something more than the cost of retraining pilots that keeps Boeing wedded to MCAS.
——
Also, there has been a lot of focus on the weakness of the FAA approval, yet surely the EASA approval must also be similarly weak yet no attention at all has been given to the failure of the EASA process.

Posted by: ADKC | Jun 28 2019 7:46 utc | 76

I first met “IBM PC” as AMD 16MHz 80C286 🙂
Good CPUs for the time….
I guess modern technologies would allow producing 80286 at frequencies about 100MHz – but it will probably do little good, as there would be another bottleneck: “processor bus” connecting the processor to mainboard and those very sensors and memory. And that would be impossible to extend without redesign of the whole computer.
Another way would be to remove redundancy and make those two processors inside ever FCC to run DIFFERENT processes. At least in part. Classify processes into more and less critical and only duplicate critical ones, then non-critical processes (like allegedly was MCAS) would only run on one of CPUs. But – at the price of harming errors detection and mitigation.
One more thing to think about is “context switching” time lag. One thing is total data stream “speed”, and another is how fast it is required to process each new data packet. Processing 10 times 1000 bytes is significantly faster than processing 1000 times 10 bytes. Linux (or BSD) kernel even had special mode “interrupts polling” for such installations where data comes in very many very small packets. If MCAS requires such a processing, multitude of small data frames with shortest possible lag, then this alone can overwhelm the CPU.
Now good thing (seeking for a hope) might be if the MCAS fix was “debug build” with little optimization and lots of unneeded actions like extensive logging of intermediate results. THEN removing this instrumentation and making “release build” can bring it all back into 80286 performance envelope. However, this was just a quest for a hope.

Posted by: Arioch | Jun 28 2019 8:26 utc | 77

ADKC From my understanding it was FAA that required a fix for certification due to stall characteristics (as in when the aircraft had actually stalled). From what I can make of it, MCAS took control away from pilots when it thought a stall was imminent to ensure a stall would not occur.

Posted by: Peter AU 1 | Jun 28 2019 8:30 utc | 78

ADKC74
Some good points there. Instead of asking yourself as a passenger-how about asking yourself how you can sell more planes as a salesman – you know, the art of the deal etc.
Being able to present your latest toy as being “ready to go” with current pilot assets not requiring extra training is a massive sales benefit or “edge”. It boils down to the “all about the dollar syndrome” where everything, including safety, is a secondary consideration. It’s endemic throughout western culture and not only at the front end (sales). It also shows up strongly at the back end where companies refuse to do required maintenance on operating assets then wonder why things “blow up’ with subsequent loss of life. It’s a disease, driven by the american mindset of short-termism, just in time supply lines etc. I’ve been watching it all my life.

Posted by: m | Jun 28 2019 8:39 utc | 79

> The 286 series has a 24 bit address bus which can handle up to 2 megabytes of RAM.
16MB
8086/8088/80186/80188 have 20 bit (16+4) and it tops with 1 MB
80286 adds 4 more lanes over those 20 – so it becomes 16 x 1 MB
But only in “protected mode” which may be used by that OS or not, it probably would make “context switching” slower than in “real mode”.

Posted by: Arioch | Jun 28 2019 8:47 utc | 80

@B #12
> More I/O interrupts occur, more then current process data is pushed onto the stack, which thereby grows and in the end corrupts the rest of the memory.
A.S. The article does not quote “BSODs”, they quote “delays” and “lags”. So no memory corruption is evident, but the lack of peak processing power is.
Now to the “theory”:
1. This specific problem was long solved for example in MS-DOS 2.0 “device drivers” which consisted of two subroutines, interrupt handler and data handler. There should had been NO any data processing in the “interrupt handler”, only reading out the data packet and saving it in queue for further processing and then clearing interrupt condition, and with maximum possible speed.
If even MS-DOS 2.0 knew it – then I believe industrial OSes know it too.
2. You can not have interrupt with interrupt within interrupt finally exhausting stack by their unlimited nesting depth.
80286 uses Intel interrupt controller chip, developed for 8-bits 8008/8080/Z80 processors, actually two chips, cascaded (or emulation of those on later curcuitry). That maxes out at (8+8-1)=15 interrupt classes (IRQs). Plus there is NMI – non-maskable interrupt (at least there was on 8u0386). Which should only be signaled in extreme conditions, for example on Compaq Desk386, the first 386-CPU computer, it was signaled by “processor ‘frozen’, computer does not respond” watchdog as a last resort.
Those interrupts could be signaled in two modes: “by front” – the moment request lane turns from passive to active and requiring switching back to passive before re-signaling, or “by level” – by the very fact of lane being active, no matter how long it is.
The processor also has “Interrupt Flag” choosing whether the processor will or will not respond to those signals.
Of processor responds to an interrupt – the flag was dropped. So – !!! – while one interrupt handler is being processed there is no way any other standard interrupt (NMI being exception) can get in the way.
Only AFTER interrupt handler subroutine is finished (by the special return-from-interrupt opcode, was it RTI or IRET in Intel parlance I do not remember) this flag is restored so processor can now answer to another interrupt.
IOW, unless Boeing engineers coded insane interrupt handlers that re-enable interrupts processing before their termination – it should be absolutely impossible to “exhaust stack” by “unlimited” number of nested interrupts – for there can be none (except for NMI).
There are also CPU-internal interrupts, “exceptions” like “division by zero” or “debugging breakpoint” conditions, but they should not happen within properly written program, so should not exhaust stack either.
3. The above is true if CPU is working in the interruptible mode, typical for desktop. However the OS *might* prohibit interrupts reaction at all (sans NMI) and instead rely on “polling” – on scheduled querying hardware if “data is ready” (if interrupts conditions are fulfilled without interrupts themselves). I do not know if such a configuration is practically feasible for the specific FCC task, but it is definitely possible CPU-wise.
4. The above is true is CPU is working in “real mode” typical for single-task OS like MS-DOS was. For multi-threading OS like OS/2 or Windows or UNIX (Xenix) there was “protected mode” with clear separation of memory per-process. In particular, there was no more One True Stack shared by The System, but every process had their own one. Including interrupt handlers, which AFAIR (but I do not remember clearly this nuance today) had their own dedicated stacks too. Surely “exceptions” and “software interrupts” would use the process’ stack, but that is different case from external data-driven hardware interrupts. So, overflowing some particular stack was perhaps possible, but should only affect the wrongly coded interrupt handler that could not manage its own stack, without affecting rest of the system.

Posted by: Arioch | Jun 28 2019 9:16 utc | 81

Peter AU 1
Thanks.
For me, your info implies that the plane is aerodynamically unstable because it is prone to stall and cannot be prevented from doing so doing accept by MCAS.
In contrast, the info that you supplied led me to an article in
The Verge that states that the 737MAX behaved differently (from other 737) but only when it’s climbing steeply. And the MCAS was only required to make 737MAX behave in the same way as all other 737 for this scenario (climbing steeply) only. And MCAS was just fitted to ensure that no additional training (and cost) was needed. (It’s fairly likely that b has already covered all this).
So, if this is the case MCAS is an unnecessary piece of software. And Boeing could free themselves from MCAS by just abandoning it. But instead, they persist.
I wonder if an entirely different kind of FAA approval process and tests is required if the 737MAX is judged to be in a different class. And Boeing has reason to believe that 737MAX would not pass that approval process so they have no choice but to continue with MCAS to keep the 737MAX within the existing class?

Posted by: ADKC | Jun 28 2019 9:20 utc | 82

ADKC
From earlier on in the Boeing saga, and believe b mentioned it at that time, was the engine nacelles retained lift when the plane was at very high angles of attack.
If the horizontal stabilizer stalls before the main wing, the stall may be impossible to recover from and I thought this was the reason the FAA required a fix to certify the aircraft.

Posted by: Peter AU 1 | Jun 28 2019 9:44 utc | 83

@ADKC #80
There was some nuance in the history. And it was mentioned in media and at MoA few weeks ago. As far as we can trust those leaks, of course.
Initially MCAS was envisioned as “augmenting” system, helping pilots but not being essential.
That status let it run on one sensor alone and even that to do without error checking – because if “augmenting” system fails, it is not essential.
That status also let FAA take it easy upon MCAS – because it was just a comfy extra, not a core service.
That status also limited the effect MCAS could cause upon the aircraft – the speed and range of altering control planes.
However during test flights it was discovered that those limits were not enough to fully compensate for changed engines. And then the limits were relaxed. AFAIR control planes moving speed was increased three-fold and the range of MCAS-induced deviation was increased too.
By “normal” processing this should had triggered re-classifying of MCAS from “augmentation” into “essential” system, with total change of requirements for it (and for FAA).
Instead this information was buried deep inside some archives and obsolete, now wrong information about strict limits of MCAS effects was promoted.
Now back to your question
> And Boeing could free themselves from MCAS by just abandoning it.
Well, this looks as very reasonable proposal, assuming two pre-conditions:
1. MCAS really is just “augmentation” system that can be lived without. However the mere claim, that limits had to be extended so much after test flights suggest it might be more than that actually. Maybe not.
2. This would not expose Boeing to even worse legal charges. One thing is killing people by introducing an essential and needed security system (but screwing its design – we all make mistakes). Another thing is killing people by introducing system that was not necessary to start with.
Imagine two abstract situation.
a) I found a person in the street with obviously critical and rapidly worsening health condition. There was some loaded syringe near that person. Taking chances that the syringe was prepared by the person who did not had enough time to apply it, I take the syringe and do an injection. The person dies, later it turns out syringe was filled with poison rather than medicine. Technically I killed the person, but I did under emergent condition as a last resort risky attempt.
b) I found some syringe loaded with something, I also found a person sunbathing and obviously healthy. Without asking I made a sudden injection into the person, because why not. The person dies, the syringe was poisonous. Technically I killed the person in the perfectly same way as above. But now I do not have any plausible reason why I did it, without malice.

Posted by: Arioch | Jun 28 2019 9:48 utc | 84

I’ve programmed low lever process control hardware. With no operating system (no windows), and no graphics, a 286 has a lot of grunt. I expect there is a set of interrupt handlers from sensors and other input devices which post messages to a message queue which a state machine will process. But, it has been my experience that when the message queue gets >1 then you start to have problems. IE backlog and running out of processing power. Not clear this is the case here. Also, these embedded systems are very hard to verify and debug. I would not want to take on an existing system then try to apply a patch. Recipe for problems. This is assuming there are no inherent bugs in the existing software. Also, in my experience, there are always hardware problems.

Posted by: cdvision | Jun 28 2019 9:52 utc | 85

ADKC
I did a bit of a search on 737 MAX stall characteristics and run onto this.
“In 2012, veteran test pilot Ray Craig testing high-speed situations in a simulator in the early days of 737 MAX development, detected problems with the 737 MAX in certain high-speed cases. The FAA demands all passenger aircraft be able to handle a battery of extreme cases, like flying into another plane’s vortex. Boeing’s fix was the MCAS software.”
“But in 2016, after the 737 MAX took off in its first test flight, test pilots reported “something was off.” The pilots were having trouble in low-speed stall conditions. These problems were thought to be due to the bigger, more powerful engines and their placement more forward than in the previous 737 model. Rather than apply an aerodynamic fix or have pilots try to handle these situations, MCAS was offered as a solution.”
https://www.engineering.com/Hardware/ArticleID/19269/Boeing-737-MAX-Pilots-Had-No-Idea-What-They-Were-Up-Against.aspx
My memory was a bit out as FAA required a fix for high speed problems rather than low speed.
On the low speed stall characteristics this is worth noting – “or have pilots try to handle these situations”
Rather than ‘retrain’, the wording is ‘not have pilots try to handle these situations’.
Airbus has an auto stall recovery mode that is not essential for stall recovery, but on the MAX, if test pilots have difficulty recovering from a stall manually, the average pilot may not be able to do it even with retraining. The thing not mentioned in the piece is how much altitude it took to recover from the stall. Low speed stall testing would be done at relatively high attitude, whereas in practice, a low speed stall is more likely to occur at lower altitudes.

Posted by: Peter AU 1 | Jun 28 2019 10:45 utc | 86

I looked at the certification process for whenever a new aircrafe is added to a type certificate (a class).
What appears to happen is that only the changes between the new aircraft and the class need to listed and reviewed (it was from this list that MCAS was omitted).
If the 737MAX was treated as a new class then I presume everything would have to be reviewed.
So there are significant benefits to Boeing in getting a new aircraft within an existing class. The approval process is shorter and easier and the aricraft can go into production sooner. But, the risks might well be greater particularly if you “force” a new plane into an existing class by opting for sub-optimal designs and features.
MCAS therefore provides a method to ensure that the 737MAX faced a less rigorous approval process (as well as saving on training costs).
A feature that was unchanged from the existing class might be subject to different stresses and risks (because of other changes) but would appear to escape review because it has not changed.
Also, there is considerable evidence that Boeing did everything to prevent any change they could (even though such changes might well be beneficial) to ensure that the 737MAX remained within the existing class. In other, words innovations and improvements that could have reasonably (and easily) been added were kept out of the plane just the sake of keeping the 737MAX within the constraints of its class.

Posted by: Arioch | Jun 28 2019 10:55 utc | 87

Arioch – I used your handle by mistake @85

Posted by: ADKC | Jun 28 2019 11:01 utc | 88

Peter AU 1, Arioch, & cdvision
Thanks for providing info relating to my enquiry. You’ve gone some way to convincing me that the 737MAX problems may well be insurmountable.

Posted by: ADKC | Jun 28 2019 11:05 utc | 89

I keep on getting it wrong 🙁
It was definitely Arioch @85 but it was me @88

Posted by: ADKC | Jun 28 2019 11:40 utc | 90

Airlines and regulators meet to discuss Boeing 737 MAX un-grounding efforts
Airlines and regulators are gathering at a closed-door summit in Montreal on Wednesday to exchange views on steps needed for a safe and coordinated return of Boeing Co’s grounded 737 MAX jets to the skies following two deadly crashes.
<...>

Posted by: John Smith | Jun 28 2019 11:42 utc | 91

Sounds like the low speed stall problem would be the biggest killer, precisely because such stall recovery would typically call for increasing power as well as pushing controls for nose down. But in the MAX, increasing power exacerbates the nose-up forces. So essentially, without MCAS a pilot was going to have to be at full attention during power climb out, with his thumb on the trim and prepared for stall conditions to suddenly pop up at a flight condition you would not normally expect it and go immediately to serious nose down trim; and at low speed conditions (when you don’t have the altitude to recover), be ready not just to trim down but resist the urge to add power as trained his whole life, and perhaps even cut power at a time when they might or might not have the altitude to do that…what a mess. Sounds like it would have taken some serious re-design to correct these unacceptable issues, which is why Boeing decided to simply lie and hope for the best.

Posted by: J Swift | Jun 28 2019 12:27 utc | 92

Boeing’s 737 deadly imbroglio is a typical case of falling profitability (the law of the “tendency of the profit rate to fall”). The only thing that is worth to highlight here is that the spark that ignited the crisis was direct competition with another behemoth (Airbus).
The only way to permanently fix this clusterfuck is for the FAA to enforce a regulation over Boeing so strong that the losses would exceed the ones it would suffer if it put “band aids”. But Boeing cannot fail: if that happens, then the USA would lose too much ground in the capitalist race.
My suggestion would be a “bail out”: the USG uses taxpayer money to pay for the costs (investments) of doing a new model capable of competing seriously with Airbus’ new model. That way, Boeing could reap the profits without incurring the costs of productive investment, effectively recovering lost time.

Posted by: vk | Jun 28 2019 12:30 utc | 93

ADKC 91
If you are trying to get to the process behind how MCAS ended up in the existing format, and as well as what others have chipped in above:
https://leehamnews.com/2019/02/08/bjorns-corner-pitch-stability-part-9/
Has a look at the corner MCAS was extended to handle. What strikes me as unusual is that the stab trim cutout switch changes removed the option to manual trim from switches on the yoke with MCAS disabled. Also column overide of MCAS seems to have been removed.
Those could be just shortcuts to simplify the system, or those steps might have been taken because MCAS (or possibly other stab trim automation ) was deemed too essential/critical on the Max to allow pilots to counteract it. If they wanted to do that they would have had to have gone fully manual. I also remember somewhere there was adjustment that increased frequency and/or angle at some point in the design evolution. That might indicate the system was more critical than made out, possibly.

Posted by: gzon | Jun 28 2019 12:44 utc | 94

vk 94
Yes, I would agree that the only way is for the state to get involved in a new Boeing model, but this would also mean that the 737MAX is kaput and Boeing would need to have their market share protected.
But Airbus did not want really want Boeing to develop a new model because Airbus would be compelled to the same. It is believed that Airbus deliberately forced Boeing into opting for a 737 upgrade to stop Boeing developing a new model.
So now Airbus would also need to develop a new model (because their A320neo is also an upgrade and would not be able to compete against a brand new Boeing model) and they would also require state funding to do this.
And at the same time the West presumably would be actively blocking China and Russian aircraft from operating and taking market share.
Perhaps Boeing and Airbus need to become state controlled enterprises – they are really protected monopolies anyway and both have engaged in sub-optimal behaviour.
This is such a mess – the wonders of the private sector.

Posted by: ADKC | Jun 28 2019 14:24 utc | 95

@ Posted by: ADKC | Jun 28, 2019 10:24:09 AM | 96
Yes, my solution presupposes the American people still wants to preserve their capitalist system.
The other — more permanent, but harder — solution would be for the American people to do a socialist revolution and seize the means of production. But that would require a civil war, possibly with millions of dead, and from which victory would be uncertain. That’s why revolutions are extremely rare in History, only happening when every solution within the existing system are depleted.

Posted by: vk | Jun 28 2019 15:06 utc | 96

relevant from canada – Boeing falsified records for 787 jet sold to Air Canada. It developed a fuel leak

Posted by: james | Jun 28 2019 15:17 utc | 97

Peter Au1 is seriously mixing up an FAA requirement for a solution for certain flight situations and the requirement that MCAS be the solution. The starting statement was that FAA required MCAS – totally wrong and since when regulators mandate any particular solution? MCAS was Boeing’s solution which evolved into a flight control monster. It cannot be abandoned for the reasons of liability, because this would be proof of management incompetence, and also because this would open up the old can of worms of pilot retraining.
“I never had sex with that women (because I have my own definition of what sex is).” Thus, we could play the Boeing’s name game and stop calling 737MAX dynamically unstable, let us only imply that it has difficulties in some flight situations and improving this beautiful piece of software will get us through those. But honestly we are talking about a closed shit circle with no exit solution. 737MAX will remain a death trap BECAUSE IT IS DYNAMICALLY UNSTABLE under a non-Boeing meaning of the term. It will be an accident waiting to happen and it will be a budget airline plane until one comes into another, different set of bad circumstances and ends up the same. Until then, it will be flying beautifully and proving all detractors such as myself wrong.
Luckily, 737MAX is such an ugly bird that it is easy to recognise and gives us a chance to do our prayers if we find ourselves ever boarding one. The four engines stuck high and far forward brings about an image of a women with perky plastic tits, all four of them.

Posted by: Kiza | Jun 28 2019 15:37 utc | 98

Making sausage.

Posted by: jared | Jun 28 2019 16:56 utc | 99

Some things are said to be “too big to fail”. Other things can be said to be “too complicated to fly”. Limits of complexity capable of operation by humans seem to be being reached. One wonders on the necessity of such complexity and whether or not this is the ‘universe’s’ way of indicating a “no go” zone.

Posted by: stevelaudig | Jun 28 2019 17:55 utc | 100

Back to Main