I'd believe that noise universally causes more stress, but that doesn't mean that more noise is always bad. An interesting thing about stress is that we're most productive with some stress; not too much or too little. (Sorry I don't have a citation for that). So it could be that Von Neumann used that noise to reach the sweet spot of stress.