My grandson went in the women’s bathroom with me (or in my absence, his mother) along with his younger sister until he was about 8 1/2 years old. That is when my daughter and I allowed him to go in the men’s room alone but with one of us standing outside, carefully scoping anyone who went in. At 9 years old, at public places we frequented and felt comfortable about, we would not wait outside. Unfamiliar places, we hovered near by.
That man was way out of line. You were completely within your rights to take your son in the restroom with you.
You are a good mom. End of story!