Could it be that the only large safety-first companies are the ones forced by law (either proactively, or due to reliable legal accountability if things go wrong) to be safety-first?
> There are only so many safety-first companies and products
There are only so many companies that think of themselves as safety-first. In practice, basically all companies work on things that should be safety-first.
Does your software store user data? Congrats, you are now on the hook for GDPR and a bunch of similar data handling regulations.
Does your software include a messaging component? You are now responsible for moderating abusive actors in your chat.
Does your software allow users to upload images? Now you are a potential distribution vector for CSAM.
And so on... safety isn't just for things which can cause immediate death and dismemberment
There’s a difference between "safety matters" and “safety is the primary constraint".
Most companies manage risk to an acceptable level while optimizing for speed and cost. Aerospace companies optimize for minimizing catastrophic failure, even at extreme expense.
Treating a potential GDPR fine as equivalent to a flight-control failure ignores that society, regulators, and markets treat those risks very differently.
The inconvenience and economic cost of your Discord messages leaking is not the same category of harm as your pacemaker controller failing.
And because the majority of economic activity sits in that lower-criticality category, it would not be surprising if highly specialized, safety-critical human software engineering becomes more of a niche, while much of routine software development becomes increasingly automated or commoditized.
> Treating a potential GDPR fine as equivalent to a flight-control failure ignores that society, regulators, and markets treat those risks very differently
Agreed, though I think that if GDPR fines were actually being levied at the recommended 4% of global revenue, we'd start treating them more similarly to a 737 crash.
> The inconvenience and economic cost of your Discord messages leaking is not the same category of harm as your pacemaker controller failing
Sort of depends who they leak to. Your teen classmates who bully you to suicide? Your abusive ex who is trying to track you down to kill you? The 3-letter agency who is trying to rendition your family to an internment camp?
There are a lot of seemingly benign failure modes that become extremely lethal given the right circumstances. And because we acknowledge the potential lethality of something like a pacemaker failure, we have massive infrastructure dedicated to their mitigation (EMT teams, emergency external pacemakers, surgical teams who can rapidly place new leads, etc). For things society judges less important, mitigations are often few and far between
That's a very anthropocentric view. Technology isn't a series of deliberate inventions by us, but an autonomous, self-organizing process. The development of a spear, a bow, or a computer is an evolutionary step in a chain of technological solutions that use humans as their temporary biological medium.
The human brain is not the starting point or center of this process. It is itself a product of biological evolution, a temporary information-processing system. Its limitations such as imperfect memory, are simply constraints of its biological origin. The tools we develop, from writing to digital storage are not just supplements to human ability, but the next stage in a system that is moving beyond its biological origins to find more efficient non-biological forms of information storage and processing.
Human pride in creation is a misinterpretation. We are not the masters of technology. We're just the vehicle of it. Part of a larger process of technological self-improvement that is now moving towards an era where it might no longer require us
I think your understanding of the words "autonomous" and "self-organizing" is somewhat lacking. If there were no humans, those things would not happen.
Further, if it were a byproduct of the presence of humans, then the backpath of invention would be repeated multiple times and spread out across human history, but, for instance, despite the presence of saltpeter, sulfur, and charcoal, magnetite, wood and ink across the planet, the compass, gunpowder, papermaking and printing were essentially exclusively invented in China and only spread to Europe through trade.
The absence of the four great inventions of china in the Americas heavily implies that technology is not a self-organizing process but rather a consequence of human need and opportunity meeting at cross ends.
For instance, they had the wheel in America, but no plow animals, so the idea was relegated to toys despite wheelbarrows being a potentially useful use for the wheel.
reply