Wouldn't it be nice if everyone could just get along with one another like you and your husband do? Sadly, things happen in life that create hard feelings between friends. Maybe one friend's son got the last place on the fifth grade baseball team, cutting off the chances for your other friend's child. Or maybe their kids just never got along and the parents had to get involved to settle arguments over the years, creating friction among them, as well. Whatever the reason for the ill will between your friends, it's finally become your problem, too.
If both groups of friends have been important to your family over the years, you certainly don't want to exclude any of them from your party. As you said, it would be hard to choose, and could hurt your future friendships with the guests you excluded. But you also don't want to have an ugly scene at such a happy event for you and your son. Therefore, while I'd recommend you count on the maturity and sense of common decency among your friends and invite all of them, only you know whether it could turn ugly. That's why you need a few strategies to try to head off any potential unpleasantness.
I believe it helps for guests to be forewarned of any potential trouble. That's one of the secret advantages of the current trend toward using online invitations for casual parties. Guests can see exactly who has been invited and who has RSVP'ed that they will attend. If you choose to use a written invitation instead, try to find a casual way to mention who else is on your guest list prior to the party. This avoids any spontaneous negative reactions when unprepared guests run into one another at your event. It gives each side a chance to come up with their own strategy for dealing with one another in person.
The other helpful thing you can do as the host is to be careful with your seating arrangements. If you will have assigned tables, seat your guests at tables as far apart as you possibly can.
Finally, if any of these guests decline your invitation because of anticipated issues, accept it gracefully and don't hold any hard feelings. It's better that they know themselves and avoid creating any issues for you, than if they attend your party out of a sense of guilt.