"if you're working outside the job, then you're not fully committed to me".

And if you're not working outside the job, then you're not fully committed to your career.

And for that matter, why should anyone be fully committed to a job? Life is more than a job, and most lives last longer than most jobs.