It really depends on frequency and tightness. Pulling back the hair in a ponytail once in a while is fine but the cumulative effects of pulling the hair back, even if it’s not very tight, can be thinning and breaking around the hairline.
Pulling it back tightly is never a good idea. Besides the pulling, which will thin the hairline pretty quickly, it can also cause headaches. So at the end of the day I will say yes to loose periodical ponytails.