This was difficult to accomplish when server usage was high. There were a couple of ways to deal with this.
For the first chunk, a diff of 2 meant the first chunk server returned false, as the change in port #s meant no other network traffic had occurred between the primary server's request and reporting false from the chunk server. The chunk servers were called synchronously, so if the first chunk was correct and the second chunk server returned false, the difference would (ideally) be three. It's easy to see how a high network load would cause a lot of jitter here.
Some in IRC suggested the best way to reduce jitter was to improve the processing speed of the successive curl calls by using a faster language, such as C. My initial script was in Ruby, which felt qualitatively slower than Python, so I was stymied by jitter too. (I started out rejecting 2 in 10, and it got progressively worse.) I simply checked back every one in a while. By Monday night, the server was either empty or Stripe had increased processor resources, so 9/10 of my calls were effective.
For the first chunk, a diff of 2 meant the first chunk server returned false, as the change in port #s meant no other network traffic had occurred between the primary server's request and reporting false from the chunk server. The chunk servers were called synchronously, so if the first chunk was correct and the second chunk server returned false, the difference would (ideally) be three. It's easy to see how a high network load would cause a lot of jitter here.
Some in IRC suggested the best way to reduce jitter was to improve the processing speed of the successive curl calls by using a faster language, such as C. My initial script was in Ruby, which felt qualitatively slower than Python, so I was stymied by jitter too. (I started out rejecting 2 in 10, and it got progressively worse.) I simply checked back every one in a while. By Monday night, the server was either empty or Stripe had increased processor resources, so 9/10 of my calls were effective.