Any of you lot work in IT? Cancel your weekend

[warshipadmin]

Crowdstrike's latest update is causing BSOD reboot loops. Apparently to get into safe mode to fix it you might need physical access to the machine. ruh roh.

We've got many large companies frozen as of about 3 hours ago.

Solution is "we were able to get our systems/security teams back online by rebooting into safe mode and renaming the: C:\windows\system32\drivers\crowdstrike folder and rebooting"

OTOH Crowdstrike recommend the following

Boot Windows into Safe Mode or the Windows Recovery Environment

Navigate to the C:\Windows\System32\drivers\CrowdStrike directory

Locate the file matching “C-00000291*.sys”, and delete it.

Boot the host normally.

We've got banks airlines the ABC and service stations down, just asked my wife to fill up the diesel tank.

[Simon Darkshade]

Additional “downs” are SkyTV, NSW Police, the major supermarkets, power and water and government departments.

It is in Aus, Japan, India (won’t somebody think of the poor scammers!), Britain and the US.

[Nik_SpeakerToCats]

HSBC site seems to be okay, but card-handling in-store may be hurt...

I'm taking enough cash for this afternoon's 'weekend' shop...

[David Newton]

I work in IT. No problems at all so far as I am aware with our systems. Of course we don't use the offending software!

[jemhouston]

Retired IT. Due to some back scene drama, we had to switch MSPs. We went to Crowdstrike. I didn't like. My new manager had things locked down, so he had to give permission to do anything with it if there was an issue.

We had an issue, so I had to work with one of their people. The person they assigned me was in Sydney AU. I'm in Houston, TX, other side of the planet.


That is one of many reasons I'm retired.

[Nathan45]

Hmmm
https://x.com/anammostarac/status/1814229225580204255

[Rocket J Squrriel]

According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.

Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.

[jemhouston]

If they were onsite, did anyone go to their office with an axe or impact object in their hands?

[Micael]

It seems that the file that Crowdstrike pushed out as part of the update, and that caused this issue, was full of just zeroes:


Bit of a boo-boo that this wasn’t caught before pushing it out.

[Johnnie Lyle]

Everyone who shuts their computer down at the end of the work day was fine.

We were able to pull backup laptops out for staff who were bricked until IT could unbrick them.

[Johnnie Lyle]

According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.

Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.

Sounds like typical IT idjits.

The (expletive deleted) are absolutely shit about communicating, let alone actually talking to the users before they roll out something new, so we can identify the problems and prevent them from happening. I just got done working with one vendor to fix an issue caused by a partial rollout of Good Idea Fairydom, and still have two key systems down.

If they were onsite, did anyone go to their office with an axe or impact object in their hands?

You joining Pooh on his mercy and leniency tour?

[Rocket J Squrriel]

According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.

Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.

Sounds like typical IT idjits.

The (expletive deleted) are absolutely shit about communicating, let alone actually talking to the users before they roll out something new, so we can identify the problems and prevent them from happening. I just got done working with one vendor to fix an issue caused by a partial rollout of Good Idea Fairydom, and still have two key systems down.

Fortunately my place of employment is fairly good at not rolling something out without making sure it doesn't crash the system or if it does, they can pull it back. We went to Windows 10 several years after it came out because we had so many legacy apps out there. Lots of patches, replacements, etc.

My section has enough people that worked in the unit spread out to other areas that back channels work pretty well.

[warshipadmin]

Ford used to be anal about testing stuff before releasing it. Obviously the software I used was of little urgency, so we were years behind the current release. So we'd be raising bug reports, and they'd come back with oh we fixed that in last year's release. Now they've gone the other way and we run the current version.

[Poohbah]

According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.

Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.

Had that happen to me while I was working a high priority tasker that needed to be done in fifteen minutes (supporting a deadline deadline for submitting the RFP through the government e-commerce portal).

I managed to get it done, but the official receipt showed we had three seconds left.

[Poohbah]

According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.

Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.

Sounds like typical IT idjits.

The (expletive deleted) are absolutely shit about communicating, let alone actually talking to the users before they roll out something new, so we can identify the problems and prevent them from happening. I just got done working with one vendor to fix an issue caused by a partial rollout of Good Idea Fairydom, and still have two key systems down.

If they were onsite, did anyone go to their office with an axe or impact object in their hands?

You joining Pooh on his mercy and leniency tour?

I was once told that decapitating the IT shop would only be a momentary pleasure, and people would talk.

[Kunkmiester]

https://x.com/Perpetualmaniac/status/18 ... 8095754753

Thread on it, is the actual code actually out there though?

[jemhouston]

According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.

Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.

Sounds like typical IT idjits.

The (expletive deleted) are absolutely shit about communicating, let alone actually talking to the users before they roll out something new, so we can identify the problems and prevent them from happening. I just got done working with one vendor to fix an issue caused by a partial rollout of Good Idea Fairydom, and still have two key systems down.

If they were onsite, did anyone go to their office with an axe or impact object in their hands?

You joining Pooh on his mercy and leniency tour?

I'm unfamiliar with the terms and leniency, please provide real life examples

[jemhouston]

According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.

Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.

Sounds like typical IT idjits.

The (expletive deleted) are absolutely shit about communicating, let alone actually talking to the users before they roll out something new, so we can identify the problems and prevent them from happening. I just got done working with one vendor to fix an issue caused by a partial rollout of Good Idea Fairydom, and still have two key systems down.

If they were onsite, did anyone go to their office with an axe or impact object in their hands?

You joining Pooh on his mercy and leniency tour?

I was once told that decapitating the IT shop would only be a momentary pleasure, and people would talk.

It would also tick off the maintenance staff responsible for cleaning up hazardous waste. They would also make you fill out the needed paperwork.

Thought about doing drop tests using the H/W Engineering Department at JSC.

[Johnnie Lyle]

According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.

Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.

Sounds like typical IT idjits.

The (expletive deleted) are absolutely shit about communicating, let alone actually talking to the users before they roll out something new, so we can identify the problems and prevent them from happening. I just got done working with one vendor to fix an issue caused by a partial rollout of Good Idea Fairydom, and still have two key systems down.

If they were onsite, did anyone go to their office with an axe or impact object in their hands?

You joining Pooh on his mercy and leniency tour?

I was once told that decapitating the IT shop would only be a momentary pleasure, and people would talk.

You’re getting soft in your old age.

[Paul Nuttall]