This is certainly the worst case scenario - your security officer installing remote access software on developers machines, stealing bitcoins from production, then selling the company source code, access credentials and access to the internal network to a Russian hacker.
Building a security system to handle this level of attack is a whole level beyond stopping even determined external attackers. Are there any best practices guides on this?
One thing that the article showed is the importance of external security review to deal with the threat of internal incompetence or evil.
It's extremely expensive. Many banks and companies in the finance industry (hedge funds) do this:
Hire at least 2 or 3 people for every job. Have them watch each other whenever touching systems that connect to production or deploying code to production. Never trust any one of them with the private keys or passwords to anything - they can only get half of a secret and their co-worker gets the other half.
To do this effectively, you have to build a zero trust environment, and rely on surveillance to ensure that nobody is a bad actor. It really makes CIA/NSA level security look somewhat weak.
This is actually the key one - the more people who need to be corrupt, the harder it is to get away with being a crook. A surprising amount of internal fraud can be completed simply by requiring people to take solid blocks (2 weeks plus) of leave every year.
That "inefficiency" the "lean 10x disruptors" congratulate themselves over does not always end well.
I've heard that's because a lot of those scams/tricks fall apart without constant gradual intervention, so an important part of it is that the employee is prevented from accessing most work-resources during that time.
I've kinda thought about this in the context of relatively complex technical jobs. One side of my family tree worked a bunch in the consumer banking (non-investment) industry, and I remember hearing story after story of someone being caught after their mandatory leave.
But at a job where the ideal is automation, rather than manual verification and analysis, I wonder what additional obstacles must be placed in a potential attacker's way? Rotating keys based on employee schedules? I'm sure there are many well-studied ways to do this, but it's interesting to armchair analyze.
Well the traditional banks have been dealing with exactly this kind of threat for a long time.
There's a load of practices that are designed to stop this kind of problem in that world.
Things like :-
- Enforced holidays. Frauds are hard to maintain when you're not there to keep cooking the books.
- Audit reviews. an independant function with the ability and authority to review key processes
- split authority. Key actions need multiple people to complete (statistically fraud incidence drops a lot when you need multiple people involved)
- Strong background checking. When I worked at a bank they went back 10 years of employment history and required accounting for any gaps. + credit checking + criminal record checking etc.
Ultimately, the founders of bitcoin companies need to understand security themselves or they won't be able to hire and manage a competent team.
Despite all the theoretical advantages of blockchain over conventional banking protocols, they are not doing categorically better than banks at avoiding major losses. Although, the numbers at http://www.risk.net/operational-risk-and-regulation/opinion/... are shocking.
I am not sure why those who essentially believe in utopia through an organized "war of all against all" wouldn't expect that anyone in position of trust wouldn't try to rip them off as much as possible.
"Damn you, you've betrayed our trust and prevented us from building a world where we didn't have to trust people..."
I'm not aware of any, but we did have quite a lot of procedures meant to mitigate this sort of things when I worked in banking. For one, none of the developers had any sort of access to production machines.
We learn some more things. Bob has prior police records in Florida, where he’s from.
So they didn't even do a background check before hiring "Bob"? For a position where he would have access to systems that handled financial data? That's just grossly incompetent, in my book. I've worked for 5-man startups and Fortune 500 companies. In every case, the offer letter has stated that the offer is conditional on the successful completion of a background check, and none of the positions I've held have been remotely as sensitive as the position that Bob was hired for.
That's sort of boilerplate though, isn't it? They reserve the right to do all sorts of terrible shit, but because most of that shit costs money they don't actually do it.
There are few positions that merit a hiring background check more than ones directly involving the financial transactions of a company. Even if it costs a lot of money, it is absolutely money well spent.
Criminal background checks are actually surprisingly hard to do. There are cheap ones that will search a subset, but doing it thoroughly actually requires physically going to courthouses in the county for all prior addresses. Even then, you can miss records if they are in counties where the person doesn't live.
This surprises me about America. Other countries have national criminal record databases. Is it that America hasn't spent the money to build one, or that counties/states/police stations don't want to share information, or that they're not allowed to share information?
But access to those databases is not necessarily available to all. In the UK we have the 'Disclosure and Barring Service' (which used to be the 'Criminal Records Bureau', but there are now a bunch of non crime (or non conviction) reasons you might be on the barred list) which will perform a background check which discloses any convictions and any allegations of sexual abuse. All this for the bargain price of £59.
As that last suggests, however, only certain roles are eligible for the checks, those being primarily roles that involve working with children or vulnerable adults, though the full list [0] is quite interesting.
Last I checked you're out about a grand for a semi-decent background check, one that would possibly have turned this up. Compared to the total cost of hiring someone that's peanuts.
They do. At least, my latest position involved submitting my employment data for the past seven years to a firm that conducted background checks. Again, this was for a position that was several layers removed from any kind of direct access to customer data or financial data.
For those interested in security engineering in the financial industry a good reading source is the banking section of:
http://www.cl.cam.ac.uk/~rja14/book.html
One interesting and relevant take away (of many) -- The greatest fraud threat to a financial firm are insiders. About 1% of staff across the industry is fired every year due to fraud. Within the fraud incidents, the most damaging fraud is perpetrated by senior and trusted individuals.
I'm surprised that anyone would take this at face value. The cryptic black hat responding to e-mail with "one word: bob" is straight out of the most teenage of hacker fanfics.
It seems odd to me that "Bob" hasn't been outed. It almost makes me suspect that someone isn't sure how much of "Bob's" role as portrayed in TFA is true and how much is a frame-job by an untouchable hacker [EDIT:] or wishful thinking by a frustrated executive.
"We had changed almost everything, but hadn’t scrapped our personal computers used while Bob had been part of the team. Would that have been the paranoid thing to do? Yes."
At my humble and refreshingly drama-free place of work we have standard client images. Anything weird and the techies re-image the client. Assuming 'Bob' wasn't in charge of the images, would such a procedure have sorted the rdp?
The larger question was why did Bob have root access to people's individual laptops? He could have done a "snowden", grabbed their SSH keys including passphrases. That would have been much harder to detect.
Let's not lose sight of the fact that their cold wallet was untouched and all they lost was on the order of a hour's worth of turnover. That's more than can be said for a lot of the other bitcoin hacks.
This reads like a case study in pure incompetance at every possible level. Lack of vetting, no third party auditing, poor segregation of customer funds. It's a total shit show. This should permanently damage their business and reputation, but the Bitcoin community has always been forgiving of people who lose their money. Fool me once...
That's just some shit that ancaps like to say. Also note the constant references to fiat currency. I don't think he was actually being racist with that.
Man, calling a social security number a "social serfdom number" is really dumb and off putting. So is the continual reference to 'fiat money' constantly.
I always love the irony of people so against the basic social contract are always so quick to turn to authorities when things predictably go wrong.
Idk, I'm so tired of NewYorker-style liberal's attempt at humor about other political groups. It's disrespectful because they don't show that they've attempted to actually think about the issues, they're just knee-jerking with their emotions. Their treatment of Trump is similarly ridiculous. They're just waving their blueTribe flag as hard as they can.
And when liberals talk about Ayn Rand? It's like she's the worst depraved demon ever. But if you read some of her quotes, they're kind of motivational. Sure it can be a little too selfish, but I know a lot of people who would benefit if they internalized that THEY are the main driver in their own lives, rather than playing victim their whole life. http://www.goodreads.com/author/quotes/432.Ayn_Rand “If you don't know, the thing to do is not to get scared, but to learn.”
I've been watching the Rubin Report b/c Dave Rubin is a liberal that will actually fucking sit down and talk it out with someone on the otherside. Great show. https://www.youtube.com/user/RubinReport
I stopped at 1:20 when he stated that many on the far-left endorse terrorism. Which is apparently the intellectual equivalent of when conservatives paint all Muslims as terrorists.
No worries. Just added Rubin to my ever growing twit filter. Thanks for the tip.
Whole article tl;dr: randian libertarian idealist buttcoin people are extremely surprised to discover that a black market economy turns out to actually contain real blackhats.
The article seemed pretty open about major mistakes that ShapeShift made and lessons learned. It's a good postmortem to learn from, and far more open than most would have posted.
One of the striking things in this article was when he said they might have been compromised by their "CloudCo" (Cloud Provider). If I'm going to build any systems that handle money or bitcoin in a cloud provider, I will make damn sure I don't trust the cloud provider with anything.
Everything should be fully encrypted such that even a breach of trust from the hosting provider would not compromise your data/funds. I know this is hard to do, but it's mandatory when you're handling digital currency.
Encryption won't protect you - the cloud provider has access to executables (in ram and perhaps on disc), your keys (ram and disc) and the data both pre and post encryption (in ram).
Because they control the hypervisor, they control everything. That means they have as much access and authority as the code that you are running on their servers have. So the only way to protect yourself from them is to limit what your servers (deployed on their cloud) can actually do.
So for instance you could have a secure backend server on a dedicated host in a trusted environment, with the cloud servers using an API to the backend server. If the API is suitably secure then the cloud servers could be compromised without allowing them to directly issue invalid commands in the same way the backend server could. Then you could use the cloud to scale out your web frontend without compromising yourself.
The same is true of hardware on the dedicated host (such as the "Trusted Computing" Module) that you do not control. If that (or the BIOS) gets compromised you might not even know that your host is no longer secure.
You can use hardware security modules in datacenter space you physically control to store the private keys used to encrypt your data at "CloudCo". Amazon even offers this service and calls it Cloud HSM.
There's always the in-memory vulnerability, which is harder to mitigate, but requires an attacker with physical access to the hypervisor, so it's much more difficult to execute (as most meat-space hacks are).
In a limited way, this is what Apple does with iCloud. The customer data is encrypted and stored at AWS and Azure, and presumably decrypted on servers Apple runs and controls.
It's not just hard to do, it's impossible. For a cloud provider anyway. Given that your software has to be capable of executing Bitcoin transactions itself, and the VM manager sits at a higher level and controls allocation and access to your VM's memory (to isolate it from other VMs), you are at the very least trusting that (1) when your cloud provider says they are using VMware or whatever standard software they aren't lying, and (2) that VMware itself is securely designed to limit its own access (and access of its own operators) to the underlying VMs.
If you don't want to trust your hosting provider with anything, you have to own the hardware.
IMHO if you're doing anything financial and don't own the bare metal hardware that the hypervisor runs on and 100% control physical access to it, and run your own network gear (right up to your border with transit providers), you're doing something fundamentally wrong.
To the contrary - their business model depended on it. Spending money on security and procedures would make them unprofitable. It's gambling. And losing.
Not quite. I suggest that a CEO should assume his responsibilities. He is quick to trash Bob[1] several times, but I see assumption of any responsibility at all.
[1]
Despite our note to all employees to come into the office urgently, Bob, our head IT guy, the one responsible for security and infrastructure, arrives at 11:30am.
Soon after, Bob decides it’s time for his lunch break, and we don’t see him for an hour, during the worst incident in ShapeShift’s history
It's the difference between heroic responsibility and actual causality.
"Of course it's my fault. There's no-one else here who could possibly be responsible for anything".
The whole subtext of this was "Here's how I fucked up in leading this company", but then the actual text is causality. It is his responsibility to make sure employees are trustworth? Yes. Is he the cause of employees abusing trust? No.
He then also determined that his responsibility was to let people know what had happened, so that's what he did. He told us the things they used to figure out what happened in order to attempt (and fail, and then attempt again...) to prevent it from recurring.
No. There are Bobs everywhere. Do your job, and Bob doesn't rip you off. Leave Bob to the courts - blaming Bob is like blaming your dog for stealing your lunch, or a wave for soaking you on the shore. You will not hurt Bob's feelings.
edit: Bob didn't betray you. Your friends and family betray you. Bob stole from you.
Semantics aside, if this story is to be believed the unknown hacker theif was even a bit put out by Bob. Especially if they guy robbed his employer after compromising & sabotaging key infrastructure and tech, then selling that to other people to exploit.
I mean, that's baseline treachery right there.
I like shapeshift and have used it a bit. If you(generic) hate them, alt currency, or the CEO, I think it is fair to say that Bob is pretty shitty. Yes there are sociopaths and criminals everywhere. If you have met a sociopath in real life, it isn't simply a myth that they are charming and appear normal. So I agree with the sentiment that you should expect bad things at your company, but you seem to imply:
1. You're reading something I didn't say. Edit: What should have been obvious is that some people are criminals.
2. This is a religious question. Whether Bob goes to hell or not is irrelevant to whether you've done your job well. Bob's responsibility was to be a good criminal. The CEOs responsibility was to be a good CEO. Both of them failed.
edit:
"If you have met a sociopath in real life, it isn't simply a myth that they are charming and appear normal."
It's absurd to diagnose someone as a sociopath because they steal from you. The problem is not that there are thieves in the universe, it's that you handed one the keys to the henhouse. That they lie about it, and manipulate you, is evidence that they are rational, not that they are crazy. If you feel betrayed by people other than your friends and family, you have boundary issues. Blaming your own failures on others is a good way to repeat the same mistakes over and over again.
Are you really saying that criminals can't be blamed because "thieves gonna thieve"? Sure ShapeShift can take some blame for bad security, which he readily admits in the post, but the bulk of the blame is owed to the person who did the crime.
Eh, when CEOs lay off tens of thousands of people to pocket a sweet bonus, they're "just doing their job", regardless of whether it ruins their ex-employees lives. I think it's a bit of a reach, but "just doing my job" seems to be a modern get-out-of-jail card for pretty much any action that harms others.
Building a security system to handle this level of attack is a whole level beyond stopping even determined external attackers. Are there any best practices guides on this?
One thing that the article showed is the importance of external security review to deal with the threat of internal incompetence or evil.