You are playing with fire, girlfriend, and you WILL eventually get burned. There is nothing good that can come out of a friendship between a married woman and a married man that does not include the other spouses. Do not listen to anyone who tells you otherwise. They are flat wrong.
As you continue to spend time together, you both will start to share more and more of your personal feelings. It will start out with basic frustrations about your kids, your spouse and other things that you are less than happy about at the moment. Eventually, as you grow closer, you will each probably even discuss your wants and needs that have gone unmet by your spouse. One thing will quite possibly lead to another and you will develop an emotional connection, even if you do not mean for it to. Then, one or both of you may develop significant feelings for the other that are not appropriate. This will put you in a very bad position and will either damage your friendship or, more importantly, your marriage. At some point, you may find yourself in an affair. Do not say, 'This would NEVER happen to me or my spouse. We are too strong/close for that!' Everyone says it but EVERYONE is capable of having an affair.
PLEASE, think about this before you continue. Are you doing it out of pure friendship or is there something beneath the surface already? I have been right where you are and it went way farther than I ever meant for it to go. Thankfully, I had a spouse who loved me enough to work through it with me. Will yours?
I'm glad you are planning to keep the other two spouses involved. However, I strongly suggest that if you already have any 'feelings' for this man, break all ties with him NOW. If you don't, your feelings will continue to grow and then sometime in the future, you and he WILL find a time and a way to be alone again.
It just sounds as though you already know where this may be headed. Am I wrong?
I don't see it as being an issue at this point. To me, it's like working closely with a man at a 9 to 5 job. Are the risks there at the 9 to 5 job to have an extra-marital affair? Absolutely, but I don't think befriending a SAHD puts you at greater risk. Does that make sense? I'm reading it back to myself and it seems like gobbledy gook. LOL
I personally see nothing wrong with it. I have male friends, my husband has female friends, we both trust each other enough and love one another that much to know that the other wouldn't do anything rash. If you are secure in your marriage, and see it simply as friendship that also benefits your children. I see no harm in it. There are a lot of SAHD now a days, more than people realize. Your spouses both know as well, so once I again, I see nothing wrong with it.