I almost always ask how they are doing too, and never have I noticed (or don't remember) someone being annoyed by me asking.