Any of you lot work in IT? Cancel your weekend
-
- Posts: 397
- Joined: Mon Nov 28, 2022 4:16 am
Any of you lot work in IT? Cancel your weekend
Crowdstrike's latest update is causing BSOD reboot loops. Apparently to get into safe mode to fix it you might need physical access to the machine. ruh roh.
We've got many large companies frozen as of about 3 hours ago.
Solution is "we were able to get our systems/security teams back online by rebooting into safe mode and renaming the: C:\windows\system32\drivers\crowdstrike folder and rebooting"
OTOH Crowdstrike recommend the following
Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.
We've got banks airlines the ABC and service stations down, just asked my wife to fill up the diesel tank.
We've got many large companies frozen as of about 3 hours ago.
Solution is "we were able to get our systems/security teams back online by rebooting into safe mode and renaming the: C:\windows\system32\drivers\crowdstrike folder and rebooting"
OTOH Crowdstrike recommend the following
Boot Windows into Safe Mode or the Windows Recovery Environment
Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.
We've got banks airlines the ABC and service stations down, just asked my wife to fill up the diesel tank.
-
- Posts: 1049
- Joined: Thu Nov 17, 2022 10:55 am
Re: Any of you lot work in IT? Cancel your weekend
Additional “downs” are SkyTV, NSW Police, the major supermarkets, power and water and government departments.
It is in Aus, Japan, India (won’t somebody think of the poor scammers!), Britain and the US.
It is in Aus, Japan, India (won’t somebody think of the poor scammers!), Britain and the US.
-
- Posts: 1126
- Joined: Sat Dec 10, 2022 10:56 am
Re: Any of you lot work in IT? Cancel your weekend
HSBC site seems to be okay, but card-handling in-store may be hurt...
I'm taking enough cash for this afternoon's 'weekend' shop...
I'm taking enough cash for this afternoon's 'weekend' shop...
-
- Posts: 845
- Joined: Thu Nov 17, 2022 9:37 am
Re: Any of you lot work in IT? Cancel your weekend
I work in IT. No problems at all so far as I am aware with our systems. Of course we don't use the offending software!
- jemhouston
- Posts: 3837
- Joined: Fri Nov 18, 2022 12:38 am
Re: Any of you lot work in IT? Cancel your weekend
Retired IT. Due to some back scene drama, we had to switch MSPs. We went to Crowdstrike. I didn't like. My new manager had things locked down, so he had to give permission to do anything with it if there was an issue.
We had an issue, so I had to work with one of their people. The person they assigned me was in Sydney AU. I'm in Houston, TX, other side of the planet.
That is one of many reasons I'm retired.
We had an issue, so I had to work with one of their people. The person they assigned me was in Sydney AU. I'm in Houston, TX, other side of the planet.
That is one of many reasons I'm retired.
Re: Any of you lot work in IT? Cancel your weekend
You do not have the required permissions to view the files attached to this post.
-
- Posts: 531
- Joined: Thu Nov 17, 2022 5:23 pm
Re: Any of you lot work in IT? Cancel your weekend
According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.
Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.
Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.
- jemhouston
- Posts: 3837
- Joined: Fri Nov 18, 2022 12:38 am
Re: Any of you lot work in IT? Cancel your weekend
If they were onsite, did anyone go to their office with an axe or impact object in their hands?
Re: Any of you lot work in IT? Cancel your weekend
It seems that the file that Crowdstrike pushed out as part of the update, and that caused this issue, was full of just zeroes:
![Image](https://pbs.twimg.com/media/GS2vJgsbUAAjnOj?format=png&name=large)
Bit of a boo-boo that this wasn’t caught before pushing it out.
Bit of a boo-boo that this wasn’t caught before pushing it out.
-
- Posts: 2710
- Joined: Thu Nov 17, 2022 2:27 pm
Re: Any of you lot work in IT? Cancel your weekend
Everyone who shuts their computer down at the end of the work day was fine.
We were able to pull backup laptops out for staff who were bricked until IT could unbrick them.
We were able to pull backup laptops out for staff who were bricked until IT could unbrick them.
-
- Posts: 2710
- Joined: Thu Nov 17, 2022 2:27 pm
Re: Any of you lot work in IT? Cancel your weekend
Sounds like typical IT idjits.Rocket J Squrriel wrote: ↑Fri Jul 19, 2024 9:33 pm According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.
Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.
The (expletive deleted) are absolutely shit about communicating, let alone actually talking to the users before they roll out something new, so we can identify the problems and prevent them from happening. I just got done working with one vendor to fix an issue caused by a partial rollout of Good Idea Fairydom, and still have two key systems down.
You joining Pooh on his mercy and leniency tour?jemhouston wrote: ↑Fri Jul 19, 2024 10:00 pm If they were onsite, did anyone go to their office with an axe or impact object in their hands?
-
- Posts: 531
- Joined: Thu Nov 17, 2022 5:23 pm
Re: Any of you lot work in IT? Cancel your weekend
Fortunately my place of employment is fairly good at not rolling something out without making sure it doesn't crash the system or if it does, they can pull it back. We went to Windows 10 several years after it came out because we had so many legacy apps out there. Lots of patches, replacements, etc.Johnnie Lyle wrote: ↑Fri Jul 19, 2024 10:19 pmSounds like typical IT idjits.Rocket J Squrriel wrote: ↑Fri Jul 19, 2024 9:33 pm According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.
Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.
The (expletive deleted) are absolutely shit about communicating, let alone actually talking to the users before they roll out something new, so we can identify the problems and prevent them from happening. I just got done working with one vendor to fix an issue caused by a partial rollout of Good Idea Fairydom, and still have two key systems down.
My section has enough people that worked in the unit spread out to other areas that back channels work pretty well.
![Very Happy :D](./images/smilies/icon_e_biggrin.gif)
-
- Posts: 397
- Joined: Mon Nov 28, 2022 4:16 am
Re: Any of you lot work in IT? Cancel your weekend
Ford used to be anal about testing stuff before releasing it. Obviously the software I used was of little urgency, so we were years behind the current release. So we'd be raising bug reports, and they'd come back with oh we fixed that in last year's release. Now they've gone the other way and we run the current version.
Re: Any of you lot work in IT? Cancel your weekend
Had that happen to me while I was working a high priority tasker that needed to be done in fifteen minutes (supporting a deadline deadline for submitting the RFP through the government e-commerce portal).Rocket J Squrriel wrote: ↑Fri Jul 19, 2024 9:33 pm According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.
Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.
I managed to get it done, but the official receipt showed we had three seconds left.
Re: Any of you lot work in IT? Cancel your weekend
I was once told that decapitating the IT shop would only be a momentary pleasure, and people would talk.Johnnie Lyle wrote: ↑Fri Jul 19, 2024 10:19 pmSounds like typical IT idjits.Rocket J Squrriel wrote: ↑Fri Jul 19, 2024 9:33 pm According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.
Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.
The (expletive deleted) are absolutely shit about communicating, let alone actually talking to the users before they roll out something new, so we can identify the problems and prevent them from happening. I just got done working with one vendor to fix an issue caused by a partial rollout of Good Idea Fairydom, and still have two key systems down.
You joining Pooh on his mercy and leniency tour?jemhouston wrote: ↑Fri Jul 19, 2024 10:00 pm If they were onsite, did anyone go to their office with an axe or impact object in their hands?
-
- Posts: 232
- Joined: Thu Nov 17, 2022 1:16 pm
Re: Any of you lot work in IT? Cancel your weekend
https://x.com/Perpetualmaniac/status/18 ... 8095754753
Thread on it, is the actual code actually out there though?
Thread on it, is the actual code actually out there though?
- jemhouston
- Posts: 3837
- Joined: Fri Nov 18, 2022 12:38 am
Re: Any of you lot work in IT? Cancel your weekend
I'm unfamiliar with the terms and leniency, please provide real life examplesJohnnie Lyle wrote: ↑Fri Jul 19, 2024 10:19 pmSounds like typical IT idjits.Rocket J Squrriel wrote: ↑Fri Jul 19, 2024 9:33 pm According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.
Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.
The (expletive deleted) are absolutely shit about communicating, let alone actually talking to the users before they roll out something new, so we can identify the problems and prevent them from happening. I just got done working with one vendor to fix an issue caused by a partial rollout of Good Idea Fairydom, and still have two key systems down.
You joining Pooh on his mercy and leniency tour?jemhouston wrote: ↑Fri Jul 19, 2024 10:00 pm If they were onsite, did anyone go to their office with an axe or impact object in their hands?
- jemhouston
- Posts: 3837
- Joined: Fri Nov 18, 2022 12:38 am
Re: Any of you lot work in IT? Cancel your weekend
It would also tick off the maintenance staff responsible for cleaning up hazardous waste. They would also make you fill out the needed paperwork.Poohbah wrote: ↑Fri Jul 19, 2024 11:52 pmI was once told that decapitating the IT shop would only be a momentary pleasure, and people would talk.Johnnie Lyle wrote: ↑Fri Jul 19, 2024 10:19 pmSounds like typical IT idjits.Rocket J Squrriel wrote: ↑Fri Jul 19, 2024 9:33 pm According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.
Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.
The (expletive deleted) are absolutely shit about communicating, let alone actually talking to the users before they roll out something new, so we can identify the problems and prevent them from happening. I just got done working with one vendor to fix an issue caused by a partial rollout of Good Idea Fairydom, and still have two key systems down.
You joining Pooh on his mercy and leniency tour?jemhouston wrote: ↑Fri Jul 19, 2024 10:00 pm If they were onsite, did anyone go to their office with an axe or impact object in their hands?
Thought about doing drop tests using the H/W Engineering Department at JSC.
-
- Posts: 2710
- Joined: Thu Nov 17, 2022 2:27 pm
Re: Any of you lot work in IT? Cancel your weekend
You’re getting soft in your old age.Poohbah wrote: ↑Fri Jul 19, 2024 11:52 pmI was once told that decapitating the IT shop would only be a momentary pleasure, and people would talk.Johnnie Lyle wrote: ↑Fri Jul 19, 2024 10:19 pmSounds like typical IT idjits.Rocket J Squrriel wrote: ↑Fri Jul 19, 2024 9:33 pm According to my IT its not impacting our systems directly but there are a couple external apps that are used that might down. I don't think we use Crowdstrike but then again Information Security doesn't tell us anything.
Real conversation with security:
"You need to auto-expire and force reset the passwords for 2000 people?"
"Yes, possible exposure so just to be on the safe side...."
"Gotcha. When are you planning on doing this and are you sending out an email informing them of it?"
"We did it a half an hour ago and why would we send out an email?"
"...." Notices that the phone queue just exploded.
The (expletive deleted) are absolutely shit about communicating, let alone actually talking to the users before they roll out something new, so we can identify the problems and prevent them from happening. I just got done working with one vendor to fix an issue caused by a partial rollout of Good Idea Fairydom, and still have two key systems down.
You joining Pooh on his mercy and leniency tour?jemhouston wrote: ↑Fri Jul 19, 2024 10:00 pm If they were onsite, did anyone go to their office with an axe or impact object in their hands?
-
- Posts: 163
- Joined: Thu Nov 17, 2022 5:19 pm