"Some will not, and we’ll need to wait for the vast majority of those to cycle out of the Web ecosystem. We expect this will take at least five more years, so we plan to use a cross signature until then."
So half a year ago they expected to continue cross-signing for 5+ years. What changed?
For what it's worth, you'll still be able to use the old roots for another two and a half years (until September 29, 2021), which is not quite five years from the date of that old post, but also way longer than half a year.
There are no configuration options of any kind in Google App Engine with managed SSL security enabled (since the point is to let Google manage it), which I'd rather keep enabled if feasible to avoid having to worry about renewals etc.
So what motivates one CA to cross-sign another? I would have thought, if you were a CA you'd prefer not to enable your competitors - especially one who's planning to give away the product for free.
I don't think anybody at Let's Encrypt has spoken on this topic, but in their case specifically there are both moral and pragmatic reasons to choose to cross-sign.
Morally is the easy one. If you work for a public CA you presumably think that the Web PKI is a good idea, and Let's Encrypt helped bring that benefit to lots more users, so that's a good thing. Consider the question of whether McDonalds should support a local soup kitchen. McDonalds thinks food helps bring people together, so why not?
Pragmatically, there are a number of benefits to Let's Encrypt for a commercial CA. It creates a "brand halo" for the "SSL Certificate" product class that you benefit from, where positive experiences with Let's Encrypt result in more customers for you. Growing the market means more opportunities for you as a seller. I see some misleading analysis of the "SSL Certificate" market that doesn't include "Does not have a cert" as one of the options. So they see Let's Encrypt crushing other outfits and assume that's got to hurt profits. But a site that goes from nothing to a Let's Encrypt cert makes no difference to sales at the for-profit CA. Even if 100 sites do that, if just one copies them but chooses to buy a cert, that's an extra sale they would not make otherwise.
AIUI IdenTrust was set-up as a sort of self-servicing entity by banks mostly for banks and their clients, so IdenTrust probably doesn't really care too much about classic web CA business.
I was going to add this. The choice of IdenTrust was not an accident. It was and is a very reputable CA but for banking reasons and so is not threatened by LA's entry into the market.
It's worth noting here, too, that the vast majority of commercial CAs do not make a lot of money from their public SSL business. The public SSL business is viewed as a loss leader that provides a public profile and security assurance for the company, but most of them make far more money from their private PKI engagements (providing all of the certificates for a company's Active Directory infrastructure, e.g.). As such, Let's Encrypt enhances people's desire for certificates, but doesn't at all compete in the private-CA space where most CAs make bank.
I dunno, if you look at the legacy of DigiNotar it seems like you're dealing with a lot of potential headaches for a couple thousand bucks that your customers hate paying you anyway.
(DigiNotar, of course, famously gave out a fraudulent * .google.com cert and is now defunct.)
As a CA you're assuming a whole lot of liability for not that much money (not that much at the scale of even a small business, anyway), and that just doesn't seem like it'd scale to a wildly profitable venture, especially considering the kinds of people who are actually well equipped to run a CA can probably make a lot more money doing basically anything else in web security. When you add up the contingency risks, the opportunity costs, and whatever actual day-to-day business expenses, it does seem like you'd be looking for other ways to make more comfortable profits.
That doesn't mean CAs should charge more or anything, just that I could accept that SSL certs for standard websites isn't what anyone with a good vision for their business is really trying to hold onto.
Let's Encrypt is a nonprofit so all expenses on cross signing would be tax deductible. Also LE only does DV (not OV or EV, and only recently wildcards) and it has purposely short expiry times so it's not a 100% direct competitor.
But that wasn't the question. The question was "How much money do you want from us before we destroy your business?"
For one, there are so many CAs that could potentially cross-sign, it's unlikely none will "defect" and take the opportunity to earn some money, but also, if none had cross-signed, they simply would have started later with their own root-cert, destroying the business anyway, so that's another reason to earn some money while you can.
CA market was so competitive, and with a free and open CAs coming in the future, IdenTrust probably grabbed the opportunity. It would be someone else if it wasn't IdenTrust.
IdenTrust has OV, EV, managed PKI, and even DV certificates with 2 year validity that some legacy organization infrastructure requires.
This is in their latest blog post "Christine expands our board’s global perspective with her career experience. She worked for many years in the Australian government"
I was wondering what impact if she, a board member of ISRG, has to comply with the Australian encryption laws?
CA have zero control over your encryption keys as is. As for the MitM there likely several CAs actually based and doing business in Australia. Yet with certificate transparency being mandatory it's almost guaranteed that entity doing something shady will go out of business really fast.
Not that I actually would be at all, but hypothetical I would be more concerned with them hiring an Australian engineer than adding an Australian board member. What is the board member going to do to compromise operational security?
Influence hiring to get the engineer spy in. Also CIA docs have been published that said they had a system where people would purposefully tie down a company by being inefficient and causing beaucratic messeshttps://www.cia.gov/news-information/featured-story-archive/...
They haven't really published a list of good/bad clients. I'm interested in what's the practical cutoff point with mobile phones? I expect desktop browsers will be less of an issue.
On Android, the root was first added in Nougat (~half of devices according to Android Distribution Dashboard). But I think that browsers (like Firefox and Chrome) on Android tend to bring their own cacerts rather than using the device's, so it's probably not as bad as it looks.
To that end, it was added to NSS 3.26/Firefox 50 (November 2016) and to Chrome 57 (March 2017).
I was eating breakfast with a multitude of Android phones around me and four "older ones" could not access that site. The oldest that could connect was ~5 months old, all using new Mobile Chrome versions.
The reality is, for a bunch of usecases, you're gonna need to support 15 plus year old devices.
So Windows XP...
There are a lot of old systems out there running API's, automation, industrial systems, etc. They never get updates, and are expected to last decades. Most of them aren't on the public internet, but HTTPS would still be a good idea. This change is going to mean a bunch of them just get changed over to having no encryption.
Work for a university. We have a number of students in Africa, southeast Asia, and eastern Europe taking online courses. We do not support Windows XP or Vista, except for in the case of these students. Some of these guys don't have the resources for anything else.
I also know that we get a number of students connecting with Android v3 or earlier from these same locales.
Most industrial devices and even consumer kiosks probably access sites that are under the control of the device's owner or manufacturer. So the sites can just use self-signed certificates in the first place.
Or, for industrial use a hundred bucks on a certificate from an older CA is nothing.
Or, as 'vbezhenar hinted at elsewhere, the operators of the systems can distribute and install the ISRG root certificate to the devices that need them.
> I built an app that's used on thousands of Android devices in an industrial setting
If its that important to you or if its a commercial offering in an 'industrial setting', you should have no problems acquiring a cheap SSL certificate from another source. You can literally get them as low as $6 a year right now.
LE provides a great service and continues to do so. If you want to nitpick, then jump to a 'competitor'.
> If you want to nitpick, then jump to a 'competitor'
Sure, I can solve the problem by switching to a different CA, or by adding the ISRG root cert to each device.
But this is a problem that didn't need to happen. I blame myself for not anticipating it when I selected LE.
And I blame LE for cross-signing with a root cert that expires so soon. Not a good choice for a new CA that will take many years to be trusted on most devices.
So iPhone 4S is out of luck. That's terrible move. Suddenly half of the Internet becomes inaccessible to a perfectly working device. Though I think that user can install new CA certificate, so it's possible to manuall work around that problem.
The iPhone 4S has not received a security update in >2 years, and has >700 known security vulnerabilities, so it is not a "perfectly working device," it's a "probably compromised device."
Tested on a few tablets my company sell / used to sell :
- FAIL Galaxy Tab 4 7" (SM-T230) Android 4.4.2
- FAIL Galaxy Tab A 7" 2016 (SM-T280) Android 5.1.1
- SUCCESS Galaxy Tab A 9.7" (SM-T550) Android 7.1.1
- SUCCESS Galaxy Tab A 10.1" (SM-T580) Android 8.1.0
I don't have any Android 6 device at hand, but this is consistent with @regecks statement "On Android, the root was first added in Nougat" (which is Android 7).
This is going to be problematic, as there are still devices currently for sale on Android 5/6 (such as the aformentionned Galaxy Tab A 7", which doesn't have a replacement on some markets).
Tried the test site on a Nexus 7 running Android 6.0.1, Firefox was ok (seems it ships with its own list of roots), but latest Chrome rejected it.
My wife runs a blog which generates substantial income and uses certs from Let's Encrypt. It's a non-tech blog with primarily US readership. Checking stats for this month, 7% of all visitors were using Android 4/5/6 (20% of all Android users). The percentage of users on old Android running Firefox was basically nil. Losing all these users would be very costly.
Hopefully certbot will be modified so it is possible to pick the current intermediate during automatic renewal. If I have to do a manual operation to switch intermediates each time the cert renews (currently done by cronjob) then it is probably safer (operationally speaking) to just buy a cert.
I don't really understand why Let's Encrypt is making this change now. Sure, the current root is expiring "soon", but not until September 2021. Switching roots could be safely pushed off to early 2021 at which point hopefully most of these older Androids would be cycled out.
As a tip to you/your company: unlike Chrome and most Chrome-based browsers, Firefox for Android has a separate root certificate store and your old devices will still be able to access Letsencrypt websites if you switch browsers.
The issue is not so much about the browser, as I'm an app editor (which happens to also sell tablets with our apps preinstalled to reduce friction). The issue is that apps that rely on the device certificate store aren't going to be able to use https with a server using a Let's Encrypt certificate issued with the new root CA.
Shipping a root certificate store would be (for my scale) a bad practice. I made the mistake of pinning a SSL key in the past, never again (you run into issues when your clients never even connect the device to the internet in 3+ years, and then your updater doesn't work anymore).
Fortunately for me, I don't currently use Let's Encrypt for my API servers, and that news was the last straw to make the boss decide we will stop selling Android 5 devices.
Unfortunately, this means our users who recently bought those devices will have some third-party apps might be broken starting 9th of July, and some sites will give a scary warning.
> This is going to be problematic, as there are still devices currently for sale on Android 5/6
If I recall correctly, Android >=5 has pretty decent support for modern crypto (such as TLS V1.2, ECC). So, Android 5/6 could still just work, if the vendor is bothered to update the CA root store.
This will indeed be a problem. Many old devices still run fine; outside of a password submission page, is https really worth the hundreds of dollars it would cost to replace my devices? I really don't know why people feel the need to get latest Android; outside of security, it's just gimmicks.
That site should have some text on it like "If you did not get any errors or warnings while opening this web page, your computer or device knows about the ISRG root certificate and Let's Encrypt will continue to work for you." Currently, this isn't immediately clear to (relative) laypeople like me.
That test site, though they do link it in their own announcement was actually created as part of their compliance with root trust programme conditions. Specifically Mozilla's conditions require them to prove their setup actually works (modulo the trust they're requesting) by setting up a web site with certificates that would work once that trust is granted. They were also required to provide example sites with e.g. expired certs so that a Firefox developer could check that does what you'd expect.
The good news is that unlike some of the required test sites, which took a bunch of advanced planning (you can't ask Let's Encrypt's service to mint you an expired cert so the expired cert was produced by requesting a valid cert and then waiting for it to expire and making sure not to lose it...) this test is easy for anybody knowledgeable to manually reproduce in a few minutes, as it's just the "wrong" certificate chain with the good trusted leaf certificate you got from Let's Encrypt. So even if they don't heed your call I'm sure someone else can.
Since, according to the article, the two different Lets encrypt intermediates have the same key, and the same name, they are interchangeable?
As in, could I just replace the intermediate in my cert-chain and have everything continue smoothly?
The end of the article suggests that this can't happen, but I can't quickly find what would stop this from happening.
On my blog ( https://theandrewbailey.com/ ), I have a "health check" page that includes all available trust chains. For my Let's Encrypt certificate, it shows 2: one through an intermediate to the DST Root, and another intermediate to the ISRG Root. I can verify that both exist and are used (though one certificate and intermediate are loaded and served): the current Firefox release (and all other browsers I've tried) uses the DST root, but Firefox Developer Edition uses the ISRG root.
It seems to make sense: certificates don't sign certificates, keys sign certificates (more specifically, certificate requests). I don't see an obvious reason that a single certificate request can't be signed more than once. If one is, trust should flow through either certificate, since the certificate says that the signer verified that the holder has the associated private key, and if that private key signed another certificate, it should not matter which intermediate the trust goes through, so long as the root on the other end is trusted.
Thanks! I can't seem to find the direct 'health-check' page though.
Also, according to the spec, certificates sign a 'tbsCertificate' which contains all data of a certificate except for the actual signature and the field that determines what signing algorithm was used.
> If you get a new certificate issued from the X3 intermediate that chains to the ISRG root, but you serve the old, cross-signed X3 intermediate in the TLS handshake, the connection will still work (* for now, keep reading).
They now support wildcards, but in order to verify you are authorised to get a wildcard certificate you need to be able to pass the DNS challenge. Unfortunately there simply isn't another universal, trust-less (and automatable) way of verifying that someone owns a domain -- other than DNS.
However, because the DNS check doesn't require writing to the webroot, you could run this on any server you like and then distribute the certificate to your edge nodes (meaning your edge nodes don't need to have access to write to your DNS). Some clients even have scripting hooks which could make this significantly easier.
(I assumed the "store my cloudflare API keys on my web host" aspect was your main concern with this method -- not necessarily who wrote the client because there are plenty of other clients.)
From the Caddy release announcement that supports wildcards[1]:
"This release introduces support for wildcard certificates, a new offering from Let's Encrypt. Getting a wildcard certificate requires enabling the DNS challenge. Fortunately, that is extremely simple with Caddy, and it works with over 20 different providers!"
In this case the third-party software implements a public protocol, comes with a full source written in a very readable language and is sufficiently small to permit a casual review.
It's not trivial but it's not that hard. In my case I modified AcmeeSharp to handle acmeev2 with a custom script.
The problem I found is that you need a DNS provider that not only has an API, but also has a fairly predictable propagation of DNS changes. I noticed through trial & error that contrary to the specs, let's encrypt doesn't wait and retry if it doesn't find the right DNS entries when it validates the DNS authorization request, instead it just fails the whole thing. Which means that you need to be pretty certain the change has propagated before you tell let's encrypt to complete the authorization. In my case I am using OVH, which has two tiers of service, a regular DNS which is fairly predictable, and a "Anycast DNS" offering which I found to be very hard to get let's encrypt to work with.
Also a rookie mistake to avoid: “* .domain.com” doesn’t cover “domain.com”. So you kind of need to get two authorizations, one for “* .domain.com” and one for “domain.com” and then get a cerficate that covers both (if that’s your need).
Is there still a valid use-case for wildcard certificates when using Lets Encrypt? AFAIK wildcards were used for financial reasons and laziness (since the traditional method of acquiring a cert was cumbersome), but with LE none of those arguments make sense.
Why not just fetch a different cert for every subdomain you use? It's also better security practice as this allows you to use different key per subdomain and the possibility to revoke bad or unused certs.
Also, if you own a very big amount of subdomains (say > 100), you want to minimize the number of certificates you want to manage/renew for easier maintenance and renewal.
I don't agree on this one. The whole point of the ACME protocol is that it allows for automated certificate management. Thus, it shouldn't matter if you manage 1 of 10000 subdomains, because you should automate it anyway.
Also, if for some reason the automated process fails, I'd rather have one subdomain go down, than all of them.
> If you have a lot of subdomains, you may want to combine them into a single certificate, up to a limit of 100 Names per Certificate. Combined with the above limit, that means you can issue certificates containing up to 5,000 unique subdomains per week.
Unfortunately CloudFlare doesn’t provide limited-scope API keys so every server requesting certs needs your global API key which is the keys-to-the-kingdom...so be careful.
You can also CNAME your _acme-challenge record to another zone and perform the updates on the target zone, avoiding overprivileging your webserver.
https://github.com/joohoi/acme-dns is a server implementation of that, and a number of popular clients support it. Or if you don't want to run acme-dns, some ACME clients support an "alias" mode that essentially does the same thing using generic DNS CNAMEing.
Annoyed that Route53 IAM still doesn't let you limit the record label ...
No need for individual servers, let alone public ones, to have your DNS API key.
Have a separate machine that isn't publicly addressable generate the keys and request the certificates, and then push them out to where they need to be. You don't even need to give that box privileged access to the places the certificates need to be, if you arrange things right. Only one box then needs your API key.
That separate machine doesn't need the private keys.
Depending on your scenario it may well make more sense for the DNS privileged machine not to know the keys, and be given Certificate Signing Requests (CSRs) for the certificates it's to go get from Let's Encrypt. Most good ACME clients will accept a CSR you provide as an alternative to making their own keys or using a private key you generated.
A CSR is a signed document like a certificate, but instead of being signed by a CA (Let's Encrypt) to certify something, they're signed by "you" (ie using your private key) to prove you know the key and you want a certificate with these names on it.
Under the hood the ACME protocol always uses CSRs, but since tools to make them are often clumsy and nasty clients tend to follow Certbot (the reference client) in defaulting to making their own CSR without you needing to know about it, a client can do this because it knows your private keys. By providing your own CSRs you don't need the box speaking ACME to know your private keys.
Let's Encrypt doesn't care if you provide the same CSR repeatedly for renewals, unless your private key is known to them because somebody leaked it, in which case subsequent requests for a certificate will fail, which is what you want since that key is now compromised and you need to replace it. You should rotate keys anyway, but can do so at a pace which is appropriate to your risk factors e.g. once a year is fine for most people, and not dictated by the rate at which Let's Encrypt renewals happen.
It's not that hard, I've written a fairly short bash script that automatically issues all of the wildcard certs I use with the correct credentials (and switches to normal HTTP verification when a wildcard isn't requested).
You don't need to "use" the old root, you want to configure the chain of certificates provided so that it links back from your leaf cert to Identrust's "DST Root CA X3" not "ISRG Root X1". Specifically the chain will be just one cert, an "intermediate" which you want to ensure is the one cross-signed not the new one.
This provides a hint to the client that it should trust this certificate because it can follow the trust back down the chain to DST Root CA X3 (which it trusts) not to ISRG Root X1 which is too new-fangled for it to have heard of.
You want the one labelled: Let’s Encrypt Authority X3 (IdenTrust cross-signed)
In nginx you need to concatenate the leaf certificate from Let's Encrypt (often a file named "cert.pem") with the file you downloaded from that site, to produce a chain, which you could call mychain.pem, and then tell nginx that's your certificate chain with a config line like:
"Some will not, and we’ll need to wait for the vast majority of those to cycle out of the Web ecosystem. We expect this will take at least five more years, so we plan to use a cross signature until then."
So half a year ago they expected to continue cross-signing for 5+ years. What changed?