I'm not a Daddy Dom but I think I might be able to throw a couple of possibilities at you about it that may help.
One is that people develop their own personal preferences for certain other people throughout their entire lifetime of interacting. Basically, the idea is that who you grow up as being friends with and having positive interactions with may mold you to have an increased desire for as a partner in the future. For example, if only lean, fit men were kind to you as a child then perhaps you subconsciously seek out those types of partners. This is also why a lot of people say things like men end up marrying women like their mother's--because they grew up being cared for by primarily their mothers and subconsciously revert back to that idea (however their mother acted/treated them) as caring/loving. As time has passed Americans have grown larger in overall body-size (meaning, you're likely to be larger than your mother/your Daddy's mother since you are decades apart).
Also, and quite generally so definitely not an always, men like to feel dominant and will often seek a partner smaller than them to feel like they are physically capable of protecting. When people say things like, "Pick on someone your own size!" it's because they feel like the person who is larger can cause more damage to the other person strictly based on their size difference. The same goes for protecting someone important--body guards are usually not short, thin people, but, rather, quite tall and broad so they, at the very minimum, give off a feeling of intimidation. So, what I'm kind of saying is that being larger than your partner can be a subconscious feeling of domination too.
Then there's the meeting-needs aspect of it all. Previous littles of mine have expressed the desire to be picked up in some manner. I'm sorry if it hurts your feelings but if you're very, very heavy then most (but not all!) Caregivers are probably not going to be able to fulfill that part of your needs if it's in your list of most-desired capabilities. Sometimes if the little so very heavy it's even difficult for some Caregivers to properly hold them and still be comfortable. So, perhaps these Daddy Doms that you're seeing also desire something like that (to pick up their partner) and, thus, seek someone that can physically meet that need of theirs. I mean, personally, I want to fulfill as much as my little's needs as I possibly can, and when I'm not able to it does make me feel like I'm not a good Caregiver.
Some Caregivers want to be able to use real pull-ups on their partners. Perhaps they want to use real children's clothing to dress their little up in sometimes too. Pull-ups and brand-names like that just weren't designed to fit a large adult body.
I don't think anybody is hating on larger folks though. I don't think the Caregiver/little community is in disinterest of larger littles. If you can, I'd try not to worry too much about your body--I'm sure you're perfectly wonderful and special exactly how you are right now.