I think the main question is - do second order methods enable faster training on a single GPU? Not in terms of epochs, obviously, but wall time.