I work from home with a 3 year old and 4 month old. It really hasn't been an issue for me. As long as you have a place where you can close the door[1] it's fine and it's really nice to be able to see the kids during lunch/break. It's just important to set ground rules: while you are working kids/wife don't interrupt you.

[1] I do need to wear headphones most of the time since it can be loud sometimes :)