It’s not incompetence. These large AI companies have (or can hire) the competency to engineer proper crawlers. This is deliberate due to lack of accountability and “who’s gonna stop them?”