Moon of Alabama Brecht quote
March 29, 2019

Regulators Knew Of 737 MAX Trim Problems - Certification Demanded Training That Boeing Failed To Deliver

A recently discovered document proves that Boeing ignored requirements international regulators made when they certified Boeing's 737 MAX airplane.

After the recent Boeing 737 MAX incident in Ethiopia we explained why it happened. Even before the plane type was grounded by the FAA we wrote:

Boeing, The FAA, And Why Two 737 MAX Planes Crashed

Our early take was confirmed by the reporting of other media which we also discussed:

Flawed Safety Analysis, Failed Oversight - Why Two 737 MAX Planes Crashed

The basic problem:

For commercial reasons Boeing wanted the new 737 version to handle like the old ones. But changes in the new version required an additional system to handle certain flight situations. The development of that system and the safety analysis of its implications were rushed through. Pilots were not informed of it and not trained to counter its failure.

The added 'maneuver characteristics augmentation system' (MCAS) depended on only one sensor. When the sensor provided false data MCAS engaged and pointed the planes towards the ground. Manual trim using the plane's trim wheel was required to regain flight stability. The pilots were not aware of that. The regulators who certified the plane as safe were unaware of the extend of the problem:

The MCAS system is poorly engineered and the design should never have been certified in the first place. But the issue is even worse. The certification that was given relied on false data.

The first MCAS design, on which the safety analysis and certification was based, allowed for a maximum trim movement by MCAS of 0.6 degree of a maximum of 5 degree. Flight tests proved that to be too little to achieve the desired effects and the maximum movement was changed to 2.5 degree.

No safety analysis for the much greater movement was conducted. The FAA and foreign regulators were not informed of it. Their certification of the 737 MAX was based on misleading data.

But even those certifications were only conditional. They required from Boeing to include relevant training material that explained the MCAS trim system and its potential problems to the pilots.

The original certification for the 737 MAX was issued by the U.S. regulator FAA. The European regulator EASA based its certification on the one the FAA provided but it added several of its own requirements. There is now documentary evidence that Boeing neglected to fulfill at least one of those requirements.

The one page document, first described by Reuters, is included in the Explanatory Note Issue 10 (pdf) to the EASA Boeing 737 type certification which was issued in February 2016.

Page 15 of the Explanatory Note discusses "Longitudinal trim at Vmo". Vmo is the maximum operational speed. The trim sets the nose of the plane up or down, independent of other pilot input. Too high up and the plane with lose lift and stall, too low down and the plane will hit terrain.

A failure of the MCAS system could trim the nose down. As a countermeasure the pilots would have to switch the trim system off. They would then manually trim the plane back into a level flight. This was a concern. The EASA note says:

Subsequent to flight testing, the FAA-TAD expressed concern with compliance to the reference regulation based on an interpretation of the intent behind “trim”. The main issue being that longitudinal trim cannot be achieved throughout the flight envelope using thumb switch trim only.

EASA considered the need to use manual trim "unusual". But it allowed it to pass because the required training material would "clearly explain" the issue:

The need to use the trim wheel is considered unusual, as it is only required for manual flight in those corners of the envelope.

The increased safety provided by the Boeing design limits on the thumb switches (for out-of-trim dive characteristics) provides a compensating factor for the inability to use the thumb switches throughout the entire flight envelope. Furthermore, the additional crew procedures and training material will clearly explain to pilots the situations where use of the trim wheel may be needed due to lack of trim authority with the wheel mounted switches.


Full document

While the EASA was convinced (by Boeing?) that those situations would be discussed in "additional crew procedures and training material", Boeing did not include it in the training materials for the airlines that bought the planes:

Those situations, however, were not listed in the flight manual, according to a copy from American Airlines seen by Reuters.

Without the additional procedures and training material the 737 MAX would not have been certified. By providing the plane without the required training material Boeing essentially handed incomplete planes to its customers.

The FAA is as regulator far too cozy with lobbyists and aircraft manufacturers. It outsources too much of the certification testing to the manufacturers. It should not have allowed Boeing to install a MCAS that depended on a sole sensor.

But the bigger culprit here is clearly Boeing. The plane was developed in a rush. Even its own engineers doubted that it was safe:

Rick Ludtke, a former Boeing engineer who worked on designing the interfaces on the MAX’s flight deck, said managers mandated that any differences from the previous 737 had to be small enough that they wouldn’t trigger the need for pilots to undergo new simulator training.

That left the team working on an old architecture and layers of different design philosophies that had piled on over the years, all to serve an international pilot community that was increasingly expecting automation.

“It’s become such a kludge, that we started to speculate and wonder whether it was safe to do the MAX,” Ludtke said.

MCAS was not the only change that made the 737 MAX a 'kludge'. The design errors were inexcusable. Boeing did not inform the regulators when it quadrupled the maximum effect the MCAS system could have. These changes had side effects that were not properly analyzed. Failure of the system was hazardous and extremely difficult to handle. Indicators lights showing that the system may have failed, a safety feature, were sold as extras.

And today we learned that Boeing did not even provide its customers with the "clear explanations" the certifications required it to deliver.

These were not 'mistakes' by some lowly technicians. These were breaches of legal requirements and of trust.

It will take quite long to certify the changes Boeing announced for the 737 MAX. Lawsuits were filed against the company. Orders were canceled. The company is under criminal investigation. The commercial damage to Boeing will likely be larger than currently estimated. It comes on top of a recent WTO ruling that Boeing illegally received billions of dollars in subsidies and will need to compensate its competition.

All these are consequences of bad management decisions.

The development and production of the 787 Dreamliner, announced in 2003, was outsourced all over the world. That led to years of delays and billions in development cost overruns. In 2010 Airbus announced the A-320 NEO as a better alternative to the 737 NG. Boeing was still busy to get the 787 into the air. It had neither the engineering capacity nor the money to counter the NEO with a brand new plane. It hastily revamped the 737, a design from the 1960s, into the 737 MAX. It promised to airlines that the new plane would not require to retrain their pilots. MCAS was specifically designed to allow for that. It was a huge mistake.

Boeing once was an engineering company with an attached sales department. It 2001, when it moved its headquarter to Chicago, it became a dealership with an attached engineering wing. The philosophical difference is profound. It is time for the company to find back to its roots.

Posted by b on March 29, 2019 at 13:29 UTC | Permalink

Comments
« previous page

This story reminds me of American auto manufacturers that kept building the same vehicles for years with slight changes to tail lights and headlights while the Japanese built better and better quality cars finally overtaking the market in trust.

Posted by: arby | Mar 30 2019 15:14 utc | 101

@92 jrkrideau - see @69 Dr. George W. Oprisko's comments.. he mentions the mc-21 and more... thanks for the link!

@94 cowboy... is that something a white supremacist would say? is that how they would see it?

@95 islander.. that is what it sort of looks like to me as well..

@96 ghostship.. it has already happened with accounting and rating agencies, enron, the 2008 financial meltdown and etc.. on one hand, it doesn't seem to matter.. when do we reach a critical turning point?

@99 dh - lol.. i am here to help! it was kiza's thought, but i agree with it..

Posted by: james | Mar 30 2019 15:20 utc | 102

@94 cowboy... is that something a white supremacist would say? is that how they would see it?

Could you please define "white supremacist"? It means different things to different people.

I am a race realist, and I have no qualms about saying that there is a hybrid war going on against whites, cis-gender males, and especially against white cis-gender males. Do you deny this?

Another good example of how low engineering has fallen in the US is Tesla and their model 3. What a cheap plastic piece of crap, it falls apart as soon as the temperatures go below freezing. Wompy Wheel is the phenomenon where cheap cast parts break at the worst time whenever the drive really uses all that electric torque. Holocaust by lithium battery burn is also a great past time for Tesla drivers.

But the way Tesla most closely parallels Boeing engineering is the break through that makes it the Yugo in the age of the green new deal: Autopilot. The thing is pathetic joke, with hundreds of idiots paying with their lives. Yet Musk still sells Tesla based on the lie that they can make it work. Just like Boeing. Just like the FED too.

Posted by: Cowboy | Mar 30 2019 15:39 utc | 103

I will forward your comment to the engineers at Boeing.

Posted by: dh | Mar 30, 2019 10:34:00 AM | 99

Forward it to management instead, do CC the engineering though.

In defence of engineers, if anything, these accidents prove failures at the management level. An engineer fails when he miscalculates a stress force or fails to report his results. He can also fail by missing specifications or misimplementing established norms. He can fail in the design phase and production phase, technically but also ethically if exploiting known regulation and control limits. There is also blind paper pushing involved in so called engineering work.

The bright idea of solving a hardware issue (nose-up aerodynamics) with a software solution (nose-down MCAS) is not taken by a proper engineer as it should, that kind of decision is rising above many levels of technical competency in the industry. Even if it is established the AoA vane is the ultimate culprit of both these accidents, its engineering failure (more likely production or maintenance) is an accepted given provided sufficient use instances, and would have been properly accounted for in its dependency up until now. Hence the generally accepted three point redundancy.

Engineers are abundantly zealous in their work, who regularly include comfortable "fear factors" in their calculations, it just so happens that, increasingly the provided margins are being lionized by management, while responding to competition, up to the point of unacceptable criminal consequences.

This is Capitalism in its terminal phase.

Posted by: Vasco da Gama | Mar 30 2019 15:46 utc | 104

@104 I'm not blaming the engineers. Seems to me they were given an impossible job when management told them to fit the larger engines onto the existing 737 airframe without compromising safety.

Posted by: dh | Mar 30 2019 16:01 utc | 105

@103 cowboy... i am not buying your viewpoint here.. and i am not playing word games either.. everyone is entitled to their viewpoint however..

Posted by: james | Mar 30 2019 16:14 utc | 106

@105 I thought so dh. But i felt one couldn't stress enough that point. I think there will be a great temptation to serve engineering with the failure. And as we already have indications, the issue was known before certification, and that can only be because there were warnings from the engineering side. If Boeing did not heed these warnings it must have been a failure at another level than engineering. Enter corporate management.

Posted by: Vasco da Gama | Mar 30 2019 16:17 utc | 107


@Bart #100

[Why do aircraft manufacturers have to make calculation of the AoA such a big problem? ]

Simply because Digital can do it. A lot of useless calculations are done for this reason. Some are necessary,,, some not.
We do a lot of unnecessary BS with digital that is not needed. Smart Phones for example. Many are using them to kill liberty. To the lemming it's convenient to use a telephone to make a payment. To Government it a great way to control your every move, purchase and to ensure you are paying your 'fair share' of their cut of your productivity. This is a text book description of slavery only it's a kinder and gentler slavery.

Posted by: ken | Mar 30 2019 16:21 utc | 108

In the Lion Air crash, both airspeed indicator and angle of attack were giving faulty readings. The AOA sensor had been replaced after the previous flight as incorrect readings had been showing. In replacing a sensor that feeds information to a commuter, checking and adjusting calibration so the computer reads the signals correctly is part and parcel of the job and I think this would have been done. Although it tested correctly, it did not prevent the problem from reoccurring on the next and fatal flight. Because of this, and the fact that two readings were showing incorrectly, the cause is more likely to be a computer problem - not holding calibration most likely, otherwise some sort of computer glitch causing it to read input signals incorrectly. The other part in the software is that MCAS appears to be delegated to whichever FCC is showing worst case scenario at any one time. The two FCC's each have their own sensors then compare readings. If the FCCs have different readings, from what I have read warning signal appear. ...
Posted by: Peter AU 1 | Mar 29, 2019 4:19:43 PM | 33

I am inclined to the interpretation that there is yet another layer of control that has been kept secret from the pilots (and everybody else) - that the raw sense data taken from the instruments is processed by the computer and compensated for various factors by software, and it is this compensated or virtual instrument reading that is displayed to pilots and used as "input" in the FCC deliberations (the compensation would be calculated within the FCC, but we can think of it as a separate unit conceptually, which gives its results as input to the FCC). If this compensation screws up in one FCC but not the other - for example by misinterpreting spurious conditions from other sensors (or even supposedly unrelated faulty sensors) and thereby imposing the wrong compensations to the main flight status instrument readings (especially airspeed, AoA, altitude), and as a result the airspeed AoA and altitude presented to the co-pilot conflict with those presented to the pilot. (To illustrate crudely: supposing the pitot tube gives an intermittently false raw reading due to turbulence, the FCC misinterprets this as a misbehaving AoA sensor and therefore gives false compensation to airspeed and AoA, and thinks altitude has changed as a result of the AoA so the altitude is also falsely compensated; the sensors then have readings at varience with the computer's expectations, so it further misinterprets the signals and gives further false compensation; there are probably scenarios that would be more realistic than this one).

If this obviously speculative interpretation were correct, it would imply that any tinkering with the MCAS by Boeing would fail to address the real issue and would be bound to lead to further crashes. Such automated sensor virtualisation - if it was being done - would be extraordinarily reckless even as a declared subsystem, but criminally negligent as an undeclared one.

As others have in effect pointed out above, Boeing are trying to make a grossly obese pigeon [i.e. a crap mis-engineered airframe] fly like a condor by relying on software and oversized engines to compensate for the obesity, oversized engines and poor airframe. I blame the management 80%, but also the engineers an additional (conservative) 20%: engineers in the US today are not being taught mathematics properly any more, unlike their counterparts in other countries.

As regards the flight characteristics (Posted by: Peter AU 1 | Mar 29, 2019 11:51:18 PM | 75, as I understand the critical difference between the 737NG and the 737MAX is that the lower surfaces of the further-forward engines act like an additional aerofoil when the AoA is very high, so that in stall conditions the tail stalls before the wings, resulting in the tail falling further, and an unrecoverable condition. Whereas for a well-designed stable airframe the tail will remain flying even after the wings stall, as the tail has a smaller angle of attack than the wings, so that the pilot can use the elevator to push the nose down and recover from the stall. With the 737MAX after the tail stalls the elevator does not work so the pilot has nothing to push the nose down with.

A plane well-designed for safety will naturally tend to stable and level flight if the pilot (or computer) lets go of the controls: if it stalls, the first thing that happens is that the nose will drop. If one wing drops slightly, the other wing will have less vertical lift and will drop to compensate. If it starts to enter the conditions for a spin, the nose will drop to make a spin less likely or in some cases even difficult. The 737MAX has been fudged (not designed!) in the opposite direction. They wanted bigger more powerful engines for more fuel-efficient cruise to compete with Airbus, but the plane was too low to fit them under the wings. Making more space under the wings would require a longer undercarriage, longer space to store the undercarriage when retracted, and thereby fundamental changes to the airframe. They didn't want to redesign the airframe because that would require new certification and they wanted (fraudulently) to use the old certification. So they pushed the engines too far forward, making the airframe intrinsically unstable instead of intrinsically stable. In doing so, I assert, they knew that certification under the old 737 class was fundamentally fraudulent, because one airframe is fundamentally stable whereas the other is fundamentally unstable.

That, in a nutshell, is the 737MAX. An overly obese pigeon with oversized engines and software attempting to compensate.

----

As an aside, I believe inside a cumulonimbus cloud it should be entirely possible under freak conditions for the tail to suddenly stall due to gusts. Perhaps this is possible even without the wings stalling, due to the very violent conditions inside a cumulonimbus [or if that were not possible, maybe the tail could still stall earlier than the wings, with similar results as the tail falls increasing angle of attack]. With the 737MAX, the tail should then fall, increasing the angle of attack for the wings. Can the aircraft recover from such a condition? (Obviously the aircraft has much more height in this case than shortly after takeoff). In this case probably the heavy engines will pull the nose down eventually. Or, the aircraft could be forced into a high angle of attack by a violent upthrust on the wings in the cumulonimbus, with the same result. What about that AirAsia flight that crashed after entering cumulonimbus near Indonesia a few years ago, could that have been a 737MAX? (my sketchy memory has it as an airbus, though).

Posted by: BM | Mar 30 2019 17:59 utc | 109

@104 I'm not blaming the engineers. Seems to me they were given an impossible job when management told them to fit the larger engines onto the existing 737 airframe without compromising safety.
Posted by: dh | Mar 30, 2019 12:01:35 PM | 105

There are various different ways you can respond when given an impossible task and pressured to fulfill it. The problem was created by management, but the engineers are not entirely without blame, even though the management blame is much larger.

Posted by: BM | Mar 30 2019 18:10 utc | 110

@all Boeing will not fall over this. It made $10 billion profit last years.
Posted by: b | Mar 29, 2019 3:05:04 PM | 23

I am quite sure $10 billion is trivial smallchange in comparison with the overall costs of this scandal. There are so many aspects which will compound with each other. In the end the losses will likely be many hundreds of billions or even in the trillions. Compensation to passengers, compensation to airlines for hundreds of aircraft grounded, fines to regulators for criminal misconduct (alone likely to be in the hundreds of billions), compensation for breach of contract, losses due to loss of trust, production of grounded aircraft, storage of unsold and unsaleable aircraft ... it will go on and on.

If it were not for the gross criminal misconduct and the legal liability to thousands of potential litigants TPTB could probably push it under the rug. The legal liability in numerous jurisdictions worldwide - compounded by the aggressive posture of the US towards everybody else - will make that impossible.

Posted by: BM | Mar 30 2019 18:48 utc | 111

Check this by Al Jazeera investigation: Head of 787 Program interview (a different plane, i know)

I would not trust this guy to boil an egg, much less the one which interrupts the interview. Nevermind now trusting these guys to interpret an engineering report on consequences regarding a certain battery propensity to catch fire.

Posted by: Vasco da Gama | Mar 30 2019 19:00 utc | 112

@110 Maybe some engineers complained......I don't know. Maybe some quit. Maybe some just kept their mouths shut because they needed a job. Maybe some were promised management jobs. Maybe some genuinely thought the problem was fixable.

The ultimate responsibility rests with management IMO.

Posted by: dh | Mar 30 2019 19:01 utc | 113

109 BM -

You are saying that the elevator can stall such that its surfaces cannot raise the tail in spite of flying at, say 300 knots at two or three thousand feet of relatively dense atmosphere?

Posted by: Bart Hansen | Mar 30 2019 19:17 utc | 114

How the FAA Ceded Aviation Safety Oversight to Boeing

Since it was formed over 60 years ago, the FAA has delegated some safety certification responsibilities to the aviation industry and to individuals. But in 2005, the FAA began delegating even more responsibility to industry and individuals as part of a plan dubbed ODA, or the “Organizational Designation Authorization” program.

Under this program, companies now play a larger role in approving the airworthiness—and thus the safety—of their own aircraft. As the Transportation Department’s Inspector General reported in 2015, “One aircraft manufacturer approved about 90 percent of the design decisions for all of its own aircraft.”

(...)

“Everyone assumes someone in the system will catch something like this,” said former Boeing physicist Stan Sorscher, referring to the suspected failings of the 737 Max.

(...)

The FAA’s stance on the ODA program is that the agency “doesn’t have the resources” to certify all new designs and that, by delegating responsibilities to industry, the FAA can “concentrate its resources on the most safety-critical matters.”

(...)

In 2012, the Inspector General’s office found that the FAA’s Transport Airplane Directorate (TAD), which is the part of the agency with authority over Boeing’s commercial airlines division, “and FAA headquarters managers have not always supported TAD employee efforts to hold Boeing accountable.” FAA employees “expressed concerns regarding a recent re-organization that merged Boeing's Certification Office with the ODA. In particular, TAD employees did not believe employees from the former Boeing Certification Office, given their previous role was to deliver airplanes, would effectively advocate FAA’s position.”

(...)

In the last few years, the FAA’s deference to the aerospace industry has apparently deepened, according to a 2017 scorecard on the ODA program, jointly created by the FAA, the Aerospace Industries Association, and another industry association. The scorecard examined 36 aerospace companies that the FAA and industry said did not have to inform the FAA in writing about “low risk design changes.” Before 2015, only 14 percent of the 36 companies had this authority, known as “No-Project Notification Letter” authority. By 2018, 89 percent did. And the number of aerospace industry projects involving written notification to the FAA about design changes dropped from 89 percent in 2015 to 59 percent in 2017.

Incredible resource. Check also their Federal Contractor Misconduct Database (FCMD) on Boeing

Posted by: Vasco da Gama | Mar 31 2019 1:16 utc | 115

... by delegating responsibilities to industry, the FAA can “concentrate its resources on the most safety-critical matters.”
Posted by: Vasco da Gama | Mar 30, 2019 9:16:00 PM | 115

Erm ... , like making sure that Boeing implementation of MCAS conforms to adequate safety standards? Or is an unrecoverable dive into the earth not mission-critical enough?

In 2012, the Inspector General’s office found that the FAA’s Transport Airplane Directorate (TAD), which is the part of the agency with authority over Boeing’s commercial airlines division, “and FAA headquarters managers have not always supported TAD employee efforts to hold Boeing accountable.” FAA employees “expressed concerns regarding a recent re-organization that merged Boeing's Certification Office with the ODA. In particular, TAD employees did not believe employees from the former Boeing Certification Office, given their previous role was to deliver airplanes, would effectively advocate FAA’s position.”

Lots of juicy material for the lawyers, both against Boeing and FAA. Anyone who still has shares in Boeing, now is the time to move that investment into aviation safety litigation lawyers - it's going to be a booming industry with thousands of highly paid jobs pushing paper to and fro.

Posted by: BM | Mar 31 2019 7:29 utc | 116

You are saying that the elevator can stall such that its surfaces cannot raise the tail in spite of flying at, say 300 knots at two or three thousand feet of relatively dense atmosphere?
Posted by: Bart Hansen | Mar 30, 2019 3:17:30 PM | 114

In a cumulonimbus we would not be talking about a plane immediately after takeoff at a few thousand feet, but more likely around 10,000 to 30,000 feet. Such clouds are huge and have extremely violent turbulence, and gusts can reach speeds exceeding 300 knots, in any direction and on any of the three axes - therefore yes, the elevator could stall. If a gust at 300 knots hit the elevator in the direction of travel, then it would instantaneously have an airspeed of zero. If that had anything to do with the AirAsia crash I have no idea, I haven't followed it since shortly after the crash (was the black box ever found?). On the other hand, the turbulance in such clouds can be so violent it can smash an aircraft to pieces, so the AirAsia crash does not necessarily have anything to do with stalls or spins. With substantial height there is a lot more margin for recovery; nevertheless in a tail-stalling 737MAX with MCAS problems the question still has to be addressed what happens to the aircraft after both wings and tail stall? Will it eventually reach a nose-down position or not, and how much height does it need to do so?

Posted by: BM | Mar 31 2019 7:47 utc | 117

@ PavewayIV #79

Your link author Bjorn Fehrm posted three previous articles about the 737 disasters on his blog - I learned of them a the Naked Capitalism site this morning.

"Bjorn’s Corner: The Ethiopian Airline’s Flight 302 crash" March 15, 2019

Bjorn’s Corner: Why did Ethiopian Airlines ET302 and Lion Air JT610 crash? March 22, 2019

and

Bjorn’s Corner: The Ethiopian Airlines Flight 302 crash, Part 3 March 23, 2019

The end of the third piece:

In the life of a commercial 737 MAX pilot he should never experience an MCAS augmentation, its use case was so remote. Instead, it became the most known and explained function of all on the 737 MAX. And for the wrong, very sad reasons.

There are only a few airliner OEMs in the world. There is a reason for this. It’s a challenging product to get right and the stakes are very high for any mistakes. In today’s very safe air transport system mistakes of this scale are non-acceptable.


In my opinion the people responsible for the production and marketing of the 737 MAX ought to never have a chance to do anything like this again. Maybe they could be trusted as managers at fast food franchises. And maybe not.


Posted by: Zachary Smith | Mar 31 2019 17:53 utc | 118

arguing the angel population on a pinhead again eh.
sorry psychohistorian but imo, It is foolish to assert that this type of rort couldn't happen in a publicly financed enterprise as any Soviet citizen of a certain age could attest. USSR did many great things but the safety durability record of the bigger state capitalist enterprises such as Zil, Ilyushin or Atlas was comparable to capitalist enterprises in that as well as 'lemons' produced by extremes in tolerance, there were notorious design failures that customers were forced to accept. A socialist enterprise which valued input from all staff is of course less likely to fail but that was not the consultative model used in the really big (and later 'privatised') state capitalist enterprises.

Posted by: whatever | Apr 2 2019 6:03 utc | 119

https://www.reuters.com/article/us-ethiopia-airplane-software/boeing-software-under-scrutiny-as-ethiopia-prepares-crash-report-idUSKCN1RF0YU
"Boeing anti-stall software forced down the nose of a doomed Ethiopian jet even after pilots had turned it off, sources told Reuters on Wednesday, as investigators scrutinize the role played by technology and crew in the fatal March 10 crash."

Posted by: Peter AU 1 | Apr 3 2019 16:00 utc | 120

« previous page

The comments to this entry are closed.