Live Event Windows Outage Event - Can't buy food. Can't fly to Bali...

[Xtreme]

Welcome to your all digital future. Be careful what you wish for.

 

[Gralin]

I wonder if it pushed ITSELF LIVE??

It's the singularity.

Someone edited crowdstrikes wiki page to say this was AI taking over

I see a bunch of USB sticks with updates going out via bicycle messenger tomorrow.

For sure

 

[Chief]

I wonder if someone got in and infected the update?

 

[Pakenhamsaint]

I wonder if someone got in and infected the update?

I'd wager it's the old Friday special

 

[Gralin]

You'd think they'd have them in a key escrow or a backup secrets manager somewhere.

That they might be locked out of as well

You'd think. Unless that's their next project just after this anti-malware update...

I'd say 95% of company wouldn't have a clue what they're doing most of the time

The big thing at the moment is airgapped recovery environments, but most places don't have them yet

And they don't protect everything, it's all app data not end user compute

This has hit end user compute hard

 

[Gralin]

I'd wager it's the old Friday special

 

[Gavin Excell]

Young people facing starvation with issues Uber Eats reported

 

[Gralin]

Young people facing starvation with issues Uber Eats reported

Did you get this from a boomer Facebook page or is this all you

 

[Gavin Excell]

Did you get this from a boomer Facebook page or is this all you

I tried to.work avocado into it but lacked the elegance

 

[Gralin]

I tried to.work avocado into it but lacked the elegance

Decades of experience with the English language and you fumbled it

 

[nut]

Albos fault

 

[MarcMaverick]

You were warned what would happen if the whole world went digital.

You went full digital. Never go full digital.

 

[Gralin]

You were warned what would happen if the whole world went digital.

You went full digital. Never go full digital.

We're not full digital and the issue is capitalism not computers Marc

 

[Gralin]

Albos fault

He's definitely an enabler of the system but no

 

[CD Xbow]

Looks like it's the Crowdstrike update specifically affecting Azure VM'S, not Windows servers generally. It will be interesting to see how a buggy patch got through. Got a laugh at MS advice.

'Microsoft's top piece of advice to fix your broken Azure VMs is to turn them off and on again - repeatedly. No, even more than that.'

Reboots best fix for Azure VMs CrowdStrike borked, says Msft

Have you tried turning it off and on again, like, a bunch?

www.theregister.com

 

[Gralin]

Looks like it's the Crowdstrike update specifically affecting Azure VM'S, not Windows servers generally. It will be interesting to see how a buggy patch got through. Got a laugh at MS advice.

'Microsoft's top piece of advice to fix your broken Azure VMs is to turn them off and on again - repeatedly. No, even more than that.'

Reboots best fix for Azure VMs CrowdStrike borked, says Msft

Have you tried turning it off and on again, like, a bunch?

www.theregister.com

Their main advice is restore from backup

The reboot 15 times is probably related to triggering a last known good configuration rollback as a possible quick fix now that crowdstrike aren't pushing out a brick patch

 

[CD Xbow]

Their main advice is restore from backup

The reboot 15 times is probably related to triggering a last known good configuration rollback as a possible quick fix now that crowdstrike aren't pushing out a brick patch

It sounds silly doesn't it?
I saw this from the MS Azure server farm.

 

[bzparkes]

Just in case anyone's still affected you can go to C:\Windows\System32drivers\CrowdStrike and delete a file from th 19th(yesterday) that looks like C-00000291*.sys

Having said that, I didn't have one that matched, so I deleted the next closest, which was C-00000029.sys, which fixed my login loop. I also had the Bitlocker ****up from Microsoft this morning, which made it more annoying.

 

[bzparkes]

the file was installed at 14:29, which was a couple of minutes before everything went to shit.

 

[Gough]

Went to get a paper, my local servo is still cash only and no petrol.

 

[peetoo]

We don’t even let Microsoft free rein with auto updates

Crazy governance out there

 

[peetoo]

Have you ever deleted an important file which proved too hard to get back?
How about a whole virtual server (yes I put my hand up for that)?

Spare a thought for unisuper who’s entire cloud based system got deleted (oops!)

UniSuper members worried about retirement funds as week-long outage continues

Some of UniSuper's 630,000 members are losing sleep over whether they have lost their money put away for retirement as the superannuation company's outage enters its 7th day.

www.abc.net.au

 

[peetoo]

Honestly is we spent half the time we do telling the heirarchy what a great job we are doing ACTUALLY doing a great job we might get somewhere (but not promoted obviously)

 

[Gavin Excell]

That they might be locked out of as well

I'd say 95% of company wouldn't have a clue what they're doing most of the time

The big thing at the moment is airgapped recovery environments, but most places don't have them yet

And they don't protect everything, it's all app data not end user compute

This has hit end user compute hard

Reckon there's a bit of bluff and bluster in the IT industry

 

[peetoo]

Disaster strategy

1 keep resume current
2 keep a copy off site