Skip to main content

I mostly agree here. I haven't read the Doctorow piece. But I've been having a similar conversation within my professional circles. Yes Crowdstrike screwed up. But humans are gonna screw up. We know this. So rather than discussing who to blame, the better discussion is how so many companies found themselves exposed with no way of taking control of what was happening to their systems.
hachyderm.io/@jenniferplusplus…

Jenniferplusplus (@jenniferplusplus@hachyderm.io)

Other companies have been gutting their operations and security teams. I don't care what they say, the large majority of jobs they eliminated were actually necessary. They've been outsourcing to vendors to cover the gap. Sometimes bad things happen.
Hachyderm.io
in reply to Marco Rogers

Marco Rogers
Marco Rogers
• •
I would make a stronger statement. For a long time, I've been feeling like this is the major flaw in our current trajectory. The vendorization of everything is a trap. Not only because of the huge "blast zones" as Jennifer puts it. But because we've also created an environment where dumping risk onto vendors effectively means accountability is diffused to the point of uselessness.
in reply to Marco Rogers

Marco Rogers
Marco Rogers
• •
Crowdstrike is gonna see some consequences for this. But I suspect they won't be as severe as people think. Okta is still around after failing at their one job multiple times. So is Lastpass. Vendor failures are becoming normalized at a rapid pace. And meanwhile the companies who are delivering direct service largely dodge accountability by just blaming their vendors.
in reply to Marco Rogers

Marco Rogers
Marco Rogers
• •

My mom has been trying to fly back from San Francisco to Atlanta since Friday. Delta Airlines has been totally hosed by this issue. I guess they were deeply invested in windows and Crowdstrike.

But what decision does Delta get to make now? What can they change that won't expose them to a potential Crowdstrike or a similar vendor exposure? I don't think they have that option. The whole ecosystem is set up to shed risk in a way that makes accountability impossible.

in reply to Marco Rogers

jr conlin
jr conlin
• •

Yeah, non-tech folk grossly underestimate the complexity and scale big co's like Delta have invested in a given platform. Switching would take decades.

All rooted in basically arbitrary decisions made years ago because some exec bought a sales pitch and went all in on a marketing demo.

Not slighting anyone, though. This crap is hard and impossible to predict. This was (in Delta's eyes) a minor vendor.

in reply to jr conlin

Dave Lane 🇳🇿
Dave Lane 🇳🇿
• •
@jrconlin I'd slight them for allowing that unqualified 'exec' to make that decision. That, in a nutshell, is why we (as a human society and global environment) can't have nice things.
@jr conlin
in reply to Dave Lane 🇳🇿

Marco Rogers
Marco Rogers
• •
@lightweight @jrconlin who is supposed to make the decisions if not "execs"? Are you coming imagining some kind of all-knowing Uber exec that could do this better but doesn't for some reason?
@Dave Lane 🇳🇿 @jr conlin
in reply to Marco Rogers

jr conlin
jr conlin
• •

@lightweight

Nope. Just drawing from personal experience.

Execs make decisions based on different criteria, and often, don't stay long enough to see the consequences. In this case, I'm betting there was some level of due diligence done, but ultimately it came down to price and features.

Then it got forgotten about since it didn't "catch fire".

Granted, spending money on infra is really hard to justify, so it's usually starved anyway.

@Dave Lane 🇳🇿
in reply to jr conlin

Marco Rogers
Marco Rogers
• •
@jrconlin @lightweight I'm not saying you're wrong about what happens sometimes. But I think it's way too easy to just blame "some exec". In my experience, engineers do not stand up and say "we shouldn't do this". And even if the decision is made for them, they still have autonomy to do lots of things that might mitigate the worst case scenarios. We can't fix "some exec". We can look critically at our own actions and what we do have control over.
@Dave Lane 🇳🇿 @jr conlin
in reply to Marco Rogers

Richard Johnson
Richard Johnson
• •

@jrconlin @lightweight

Case in counter-point: you could call it broken ticket-toss buck-pass subculture perversely incentivized.

We said "no agent complexity, or if you must, it will have phased and tested/metered roll-outs of changes". We were adamantly overruled. They said "we accept the risk of total revenue outage if this agent breaks catastrophically" and (Catch-22) "you must still ensure no outage" and "you must get budget elsewhere to completely re-engineer your service".

@Dave Lane 🇳🇿 @jr conlin
⇧