Load averages can be very useful, but it's problematic to combine them with something like readiness checks. I must point out the corresponding comment in loadavg.c which is required reading:
The parent's concept of reporting "hotness" is really powerful. It lets a central authority (the load balancer in this scenario) make an informed decision about where to send traffic: the coolest backend. In the case where all systems are actually quite hot, it probably makes sense in most scenarios to continue sending traffic as best-effort.
In the case of readiness checks, nacking a check removes the backend from the pool of eligible new-traffic recipients. At least this is true for systems I've encountered. The solution you propose is problematic as it's possible for all systems to decline traffic resulting in no available backends -- rarely a desirable state -- which the load balancer usually can't do much about.
A good system will allow you to decouple the signals (loadavg, connections, latency, CPU, etc.) and the health decision (readiness checks) and the interpretation of that decision (load balancer policy) to provide the best of both these worlds.
Now, there are many scenarios where simply excusing yourself works fine -- processing expensive batch jobs, for example -- so this can work nicely, but for typical production traffic scenarios I'd advise against this approach.
presumably you want your service to be up. If you can smooth the load out to other nodes, your service stays up. If you have a bad strategy, you start losing capacity as nodes get removed.
Let's say each node can handle 5k rps and you have 10 nodes. You can handle 50k rps. If you are receiving 40k rps, a good strategy will put each node at 80% capacity. A bad strategy will knock a node out, reducing your total capacity, putting extra pressure on the rest of the system, causing more failures, and more pressure. This is called a thundering herd.
At some point, your only option is load shedding. But with a bad LB strategy, you start load shedding much much earlier than you should. This is a bad experience for customers that is avoidable with good LB strategies.
Fair. Our solution is to make sure a biased load balancer (like in TFA) isn't sending the workload towards a select few machines while others may be not be as over worked, in as simple way as possible.
We run load balancers in a fail open mode. As in, if every backend is excusing itself, then none are excused.
But as you point out, load balancing is a hairy beast unto itself.
https://github.com/torvalds/linux/blob/b97d64c722598ffed42ec...
The parent's concept of reporting "hotness" is really powerful. It lets a central authority (the load balancer in this scenario) make an informed decision about where to send traffic: the coolest backend. In the case where all systems are actually quite hot, it probably makes sense in most scenarios to continue sending traffic as best-effort.
In the case of readiness checks, nacking a check removes the backend from the pool of eligible new-traffic recipients. At least this is true for systems I've encountered. The solution you propose is problematic as it's possible for all systems to decline traffic resulting in no available backends -- rarely a desirable state -- which the load balancer usually can't do much about.
A good system will allow you to decouple the signals (loadavg, connections, latency, CPU, etc.) and the health decision (readiness checks) and the interpretation of that decision (load balancer policy) to provide the best of both these worlds.
Now, there are many scenarios where simply excusing yourself works fine -- processing expensive batch jobs, for example -- so this can work nicely, but for typical production traffic scenarios I'd advise against this approach.