Is ma’am offensive?

Although Florida isn’t really the same as southern states like Georgia Tennessee Louisiana and the Carolina’s…. We are a mix of Florida culture and southern culture. I say y’all religiously. But another thing I say is ma’am. Yes, it was taught to me out of respect but it’s also something I just say naturally. Well I’ve met a few people who talk about being referred to as ma’am as a negative thing because they think it means the person is thinking they’re an old lady but I can assure you that for me that’s not the case. I call my 3 year old niece ma’am all the time. Do y’all take offense to being called ma’am? I think nothing of it when I get called maam or say yes or no maam.