If it works, you could use the llms for the first few thousand cases, then use these annotations to train an efficient supervised model, and switch to that.

That way it would be both efficient and cost-effective.