Is there any reason not to just roll your own auth, with something like http://railscasts.com/episodes/250-authentication-from-scrat... and add omniauth if you want to use OAuth? It's not much more work than getting devise running, and it gives you the flexibility to do pretty much whatever you want in terms of fitting auth into your app.

(Also, it works fine with CanCan, which is brilliant and does a good job at staying out of your way.)