Moon of Alabama Brecht quote
November 1, 2022
Down Time

Recently the provider of this Website, Typepad, moved their systems to a new datacenter. It then announced an additional maintenance period to update the system's architecture. That ended up in a mighty screw-up. For the last days Moon of Alabama was a casualty of it.

After a while the Typepad engineers recognized the problems and decided, correctly, to roll everything back to the old version. It took a while, but finally most stuff is working again.

There are two maxims in Information Technology.

I. Never change a running system.

Unfortunately there are circumstances, like growth pain etc, where one HAS to change things. That usually ends up in trouble.

Another IT maxim is:

II. Never change multiple things at a time.

This is where the recent Typepad screw-up happened. Multiple components of the system were changed at the same time and did not interact properly with each other.

Been there, done that. Back around 1995, I was working for a large international access provider who's systems always had growth pains. Everything was well prepared for a down time and a major update of the architecture. A short maintenance period was announced and we proceeded with it.

Every element had been tested before. But unbeknown to the software engineers the network providers, those with the physical access lines, had decided to use our announced down time for a change in their systems too. The two changes collided with each other and it took more than 24 hours to even find out what had happened. The roll back, not well planned, created more of a mess. We were down for 72+ hours. We had several million customers at that time. They weren't happy.

Typepad is now back and with it all the blogs that are running on it. I am pretty sure that there are still some bugs that will have to be cleaned up over the next days.

But the service has, in general, been good over the many years Moon of Alabama existed and ran on it. It is relatively cheap and relieves me of setting up and administrating my own servers.

That's why I will stick to it. For now.

Comments

Very true….
And I thought there is massive attack on provider. Because of nature of persons which comment on this portal. Free persons on free portal.
Thanks for explanation, so theories of hackers attacks we can put aside…

Posted by: preseren3 | Nov 1 2022 6:22 utc | 1

test.
It happens, glad site is back and seemingly functioning normally now, thanks for the update.

Posted by: knighthawk | Nov 1 2022 6:24 utc | 2

Hmmm… back around 1995 I was running some major Internet backbones (AS 1239 and 1800 among others, LOL). Now the old-time Internet nerds know exactly who I am;)

Posted by: averros | Nov 1 2022 6:37 utc | 3

I. Never change a running system.
Unfortunately there are circumstances, like growth pain etc, where one HAS to change things. That usually ends up in trouble.

Hence redundancy and failover.

Another IT maxim is:
II. Never change multiple things at a time.

I’ve been preaching this for years but millenials and genz will never stop believing all the things can be done all the time all at once.
This is why we can’t have nice things.

Posted by: Arch Bungle | Nov 1 2022 6:40 utc | 4

Took me a while to work out that it was actually Typepad and not someone spamming MOA. Once I realised that all I had to do was check the Typepad Twitter page to see where they were up to. Didn’t realise how much I relied on MOA to keep informed on the non-MSM view of what is happening in the world. Not just B’s version but all the well informed posters with their own contacts and access to alternate sources.

Posted by: Cyberhorse | Nov 1 2022 6:46 utc | 5

Already expected the worst when my RSS feedreader alerted me that it couldn’t catch your feed updates anymore. Apparently the link to the feed changed and now everything is working again. Looking forward to your posts as usual.

Posted by: mrfruehling | Nov 1 2022 6:58 utc | 6

IT – not all IT’s cracked up to be, eh.

Posted by: ThusspakeZarathustra | Nov 1 2022 7:01 utc | 7

Welcome back !
And yeah… that infallible maxim in software dev + IT – change two (main) things at once and all things go pear-shaped. Been there as well, small cog in bigger machine, but the whole team watching in horror at the monitoring tools that turn red, all of them.
Hopefully Typepad will manage to implement their necessary changes and server moves bits by bits next.

Posted by: phiw13 | Nov 1 2022 7:10 utc | 8

Thanks b.
Probably a good time to acknowledge how valuable this site is.
Thank you.
And thank you to the many posters who respect the site and post in good faith.

Posted by: Melaleuca | Nov 1 2022 7:11 utc | 9

Thanks for the explanation. Clearly it was/is a technical issue combined with poor decisions. I was starting to suspect something more nefarious, but all in all the explanation given is the truth.

After a while the Typepad engineers recognized the problems and decided, correctly, to roll everything back to the old version.

I guess this means another attempt to upgrade in the near future…? Maybe change one thing at a time to avoid a repeat of the recent days.
I am fine with the blog layout and technical features when it works normally. It is an important meeting place and we don’t need too many experiments to disrupt it, especially not in these turbulent times.

Posted by: Norwegian | Nov 1 2022 7:21 utc | 10

I posted a comment on technical issues in this thread 5 minutes ago. That post is missing, so there must be technical problems ongoing still.

Posted by: Norwegian | Nov 1 2022 7:24 utc | 11

dear B and Typepad,
You had me worried for a day. As in Piping, one should never fix what is not broken, but tuning is a constant requirement.

Posted by: Andrew S MacGregor | Nov 1 2022 7:25 utc | 12

Thanks for the information and many thanks for your work.
I was lucky to have a full outage of MoA for about 18-20 hours only. Frankly, I missed it a lot 🙂

Posted by: aquadraht | Nov 1 2022 7:33 utc | 13

Search function is working again.

Posted by: SeanAU | Nov 1 2022 7:43 utc | 14

At least they had an old version to roll back to.

Posted by: Bemildred | Nov 1 2022 8:13 utc | 15

Good to have you back. I was checking multiple times….only to realise how much I depend on your commentary.

Posted by: Lubica | Nov 1 2022 8:39 utc | 16

If it ain’t broke, don’t fix it.
(From experience in the early 70’s)
Glad you are “back” b.

Posted by: Stonebird | Nov 1 2022 8:53 utc | 17

I’m damned glad it wasn’t something more serious, and my motto is “cheap and easy to use trumps a lot of features.”

Posted by: pretzelattack | Nov 1 2022 9:13 utc | 18

Two suggestions:
1. moonofalabama.org (typed into the browser by careless or lazy users) should redirect to http://www.moonofalabama.org as before (currently, it leads to a landing page of the hoster)
2. please don’t follow the trend of providing castrated rss feeds with only a snippet of article content in the body. Instead, please continue to include full article content in the feed

Posted by: herrsimon | Nov 1 2022 9:21 utc | 19

It’s good to have you back Sir.

Posted by: Mark Grostate | Nov 1 2022 9:31 utc | 20

Good to see you back, MofA is an essential source of information. When I was seeing the “verification” stuff I suspected the site had been hacked in some way. I did not think it was a Typepad technical problem only. I tried to post last night, but it did not appear, so perhaps some residual difficulties persist, hopefully not.

Posted by: Gabriel in Ireland | Nov 1 2022 9:35 utc | 21

Test… Glad you’re back

Posted by: Sean in Ireland | Nov 1 2022 9:37 utc | 22

I never realized how much I’ve come to rely on MoB since the beginning of the year until it wasn’t there, I was worried for a moment the Ukrainians had put you on the Kill List, (it wouldn’t surprise me if they probably already have done so), and someone had taken you out.
Anyway, glad to see you’re up and ruining again.

Posted by: YosserHughes | Nov 1 2022 9:44 utc | 23

There was lots of duplicate, triplicate and quadruplicate comments on the last thread, I thought: “damn, Typepad is having a seizure right now”

Posted by: Ricardo Ramirez | Nov 1 2022 9:46 utc | 24

My mental wellbeing is restored 😉

Posted by: Kaiama | Nov 1 2022 9:57 utc | 25

I worked for a major bank 1970 to 2000’s … I understand!

Posted by: roberto | Nov 1 2022 10:15 utc | 26

Glad to see you back. I need my daily feed of moronic posts from most of the people here, and the few, like karlif1, who aren’t.
I’ve done a lot of tech support in my time, including for the cash management division of Bank of America, so I know how incompetent literally everyone in the computer industry is. And the advent of the Internet only made it infinitely worse because now every moron who knows HTML and JavaScript thinks he’s a “programmer”, let alone a “system designer.” Don’t even get me started on my bailiwick of computer security.

Posted by: Richard Steven Hack | Nov 1 2022 10:40 utc | 27

Richard Steven Hack | Nov 1 2022 10:40 utc | 28
I know how incompetent literally everyone in the computer industry is
that really isn’t fair and I doubt you truly understand how complex the inter-tubes really are. for me it is a miracle the stuff works at all. so many things can go wrong when you think that a single bit can change the whole shooting match.
granted there are people who should not be allowed to be near computer keyboards but there are a lot of really smart people who make the magic happen. You never hear about them because everything works and there are no problems. It is only when something inconvenient happens that you even know they exist.
stuff that matters stays up and works all the time.

Posted by: dan of steele | Nov 1 2022 11:20 utc | 28

Cyberhorse #5
” Didn’t realise how much I relied on MOA to keep informed on the non-MSM view of what is happening in the world.”
The earlier thread had a well argued post headed up ‘Curate’. Apologies for not recalling the posters moniker. I felt they made a good point; that we should do our own research and not rely on second or third hand perspectives. Despite my general agreement I would like to take issue with myself.
You can’t be all places at all times doing everything. Inevitably we all end up dealing in second hand information. I understand that Australia exists. However, unless I walk to South-East Asia and get in a boat heading south-east, navigating using a sexton and star chart all the way, I can’t unequivocally prove its existence. Even if I done all that, the vast majority of things Australian would still be beyond my knowledge.
So effectively everything is hearsay; its all a story.
Why do we choose to beleive in some stories and not others?
I think we must be drawn to ideas as if they perform as a centre of gravity.
I’m a new entrant into the MOA community. Many years ago I witnessed the potential power of such forum as this when a local sporting controversy erupted that brought a blogger to the fore. The comments section of this blog invited a myriad of contributors that were expert, insightful, misguided, humorous, irreverent and varied. Contributors learned from each other how to source material, how to decipher main stream media, how to speculate and most of all, how empowering such a forum appeared to be. Its very presence appeared to put a brake on the calumny the forum was created to challenge.
A story with a substantial grain of truth concerns a statistician at a rural market over a century ago. There was a ‘guess the weight’ of the bull exhibit, where for a fee you could guess a weight, the prize being the animal on display. After the winner was announced, the statistician gathered up the discarded ticket counterfoils, each having a guessed weight on them. It was found that the average of all these guesses was the true weight of the animal.

Posted by: Cyclops | Nov 1 2022 11:54 utc | 29

@b
FYI: rollback doesn’t seem to have been completely successful.
I am getting a truncated request to enter a CAPTCHA style text even when loading some pages – in particular, the last OT thread after I posted a response to bevin. This is accompanied by that MOA attempting to load for a long time – presumably because the CAPTCHA image isn’t readily available.

Posted by: c1ue | Nov 1 2022 12:13 utc | 30

Posted by: dan of steele | Nov 1 2022 11:20 utc | 29
“You never hear about them because everything works and there are no problems. It is only when something inconvenient happens that you even know they exist.”
Which is precisely why I know they’re incompetent – because “inconvenience” – poor usability, poor reliability and non-existent security – is epidemic. They call themselves “software engineers” but there is virtually nothing of actual “engineering” that takes place. “Engineering” means taking components of known properties and organizing them in a provably correct manner to produce a result with known properties. Software doesn’t even come close to that definition. Only the people who design CPUs and similar hardware come close and even those usually have bugs which fortunately most of the time are mostly unnoticeable.
Ted Nelson back during one of the West Coast Computer Faires back in the ’80s said there was no acceptable software on the market. That remains true, if not more so, today. That applies to both closed source and open source software. I suspect we’re still at least two decades or more away from properly engineered software.

Posted by: Richard Steven Hack | Nov 1 2022 12:17 utc | 31

dan of steele | Nov 1 2022 11:20 utc | 28
“Miracle”… well, are you in the industry, or just a member of the audience? Those of us inside have no time for childish worshipful mystification or any other kind of enchanted nonsense (though we still type the sync command twice before shutting down the server). RSH’s claim is accurate, for some value of literally.
Richard Steven Hack | Nov 1 2022 12:17 utc | 31
Pity that design-by-contract didn’t take off. On the other hand, the browser is more of a grift than a platform.

Posted by: sippy the shot glass | Nov 1 2022 14:19 utc | 32

sippy the shot glass | Nov 1 2022 14:19 utc | 32
childish worshipful mystification
that is a bit harsh. I guess you can say I was more inside than outside. I began my journey with an AT&T 3B2 back in 1989.
One of the things that impressed me a lot were the VPNs we used to transfer medical data over unsecured networks. They were managed by a small team and there were literally 2 or 3 guys who knew how it all worked. When we had circuit problems we had to get them involved because the guys doing routing could not even see the traffic on those circuits.
another frustrating thing is if you do your job well and everything runs smoothly people think you don’t do anything. If something goes wrong people think you are a screwup. hard to come out ahead.

Posted by: dan of steele | Nov 1 2022 14:58 utc | 33

welcome back b and thanks for the details here…

Posted by: james | Nov 1 2022 15:00 utc | 34

i’ve only done IT work since around 2004 but i also recall every area – from repair and building to deployment to upgrading/testing networks to automation – had the same rule: do one thing at a time or you’ll have to guess which one effed up everything. also have a hot back up site if people NEED the services but i guess they didn’t get that memo either.
kinda funny that awful avalanche just recently had a post about the ukies upgrading their entire IT infrastructure to use NATO wank. he made a lot of the same points you do but then applied that to an active battlefield. so…could be worse?

Posted by: the pair | Nov 1 2022 15:46 utc | 35

The 5-eyes folks have taps coming from all the major telecommunication access points around the world and I posit that Cloudflare is the tool that is/will be used to control access to the internet world….kinda like China’s Great Wall but dressed up as Freedumbocracy and charging you for the violation.
I find it interesting that there is no mention by Typepad nor b of any potential outside component to this outage event. We will see how flawlessly the site works with the “new” environment and the world keeps turning…happy to eat some crow if proven wrong but others have noted MoA “special treatment/problems” within the Typepad world.

Posted by: psychohistorian | Nov 1 2022 23:28 utc | 36

Error 503 Odamnrod meditation

Posted by: Test | Nov 1 2022 23:52 utc | 37

So effectively everything is hearsay; its all a story.
Why do we choose to beleive in some stories and not others?
I think we must be drawn to ideas as if they perform as a centre of gravity.
Posted by: Cyclops | Nov 1 2022 11:54 utc | 29
Yes indeed! Excellent points Cyclops. I personally would add in ‘ principles’ and ‘ ethics’ to the notion of ideas, fwiw, but no biggy. Great anecdotal ref point/story. Good stuff. A ‘Bull’ story that’s not BS. 🙂

Posted by: SeanAU | Nov 2 2022 2:13 utc | 38

Server Pilot is $5/month: https://serverpilot.io/pricing/
It automates server administration on any cloud provider. Been using it 6+ years. No problems.
A 1GB Digital Ocean droplet is $6 per month: https://www.digitalocean.com/pricing/droplets#basic-droplets
WordPress supports TypePad imports natively. There are plugins to sweep up the loose ends with a click. You can use WordFence for a free WAF then just use a regular CDN instead of CloudFlare.
Just sayin’

Posted by: Jonathan | Nov 2 2022 3:54 utc | 39

Just got the message below on Some Ukraine… thread
Your comment could not be posted. Error type: undefined
Error 503 Service Unavailable
Service Unavailable
Odnamrod Meditation:
XID: 1977296410

Posted by: psychohistorian | Nov 2 2022 5:13 utc | 40

Thanks b and whatever you do, do NOT take them to a USA court or you might experience this:
https://wallstreetonparade.com/2022/11/three-2nd-circuit-judges-all-in-their-80s-decide-traders-rigging-libor-wasnt-really-a-crime/
A systemic failure.

Posted by: uncle tungsten | Nov 2 2022 6:19 utc | 41

Orage | Nov 2 2022 7:44 utc | 42
“Interesting piece in {mainstream rag}. Please demonstrate and acknowledge my proposal of its validity by engaging with it.”
That form of psyop has been debunked already.

Posted by: sippy the shot glass | Nov 2 2022 11:37 utc | 43

Regarding posting problems.
I am seeing a xml page reporting an error when hitting the post button. As per following.
“DTD/xhtml1-strict.dtd”>


IQUI
503 Service<br /> Unavailable
‘g/”>
leak

Error 503 Service
Unavailable

Service Unavailable

1ks d
O, It

Odnamrod Meditation:
XID: 1986894901”
If I click ok and then try a few reloads of the pages it seems the comment is actually posted.
Multiple posts must be happening because people hit the post button thinking the error must mean it hasn’t been. Or if they have refreshed but didn’t see their comment immediately.
Give it a minute and multiple refreshes.

Posted by: DunGroanin | Nov 3 2022 14:32 utc | 44

Here on the West Coast US MoA is more not available than it is available and it is way past software update time…..Hmmmmmm

Posted by: psychohistorian | Nov 3 2022 18:03 utc | 45

Now that I complain about the availability, it improves, thanks!
I hope it stays stable.

Posted by: psychohistorian | Nov 3 2022 19:42 utc | 46