Two lines of argument: social opposition (people are scared by the technology and many religions strongly oppose human GMO) and lack of adequate technology (we couldn't actually make a permanent change that does what we want without side effects).
Note, germline modification means making changes to future children, and all their cells would permanently be modified, and they would pass the changes on to their children. Look at what happened with Jianku He- who ignored all the advice to do this (amazingly, they eventually let him back in the lab).
You missed the part where I'm describing germline modifications for non medical uses. Like, intelligence augmentation for your children. Very controversial.
Public acceptance of those modifications could increase if such treatments are marketed as new or alternative approaches to what the cosmetic surgery and erectile disfunction treatment industries are making money off today.
Note, germline modification means making changes to future children, and all their cells would permanently be modified, and they would pass the changes on to their children. Look at what happened with Jianku He- who ignored all the advice to do this (amazingly, they eventually let him back in the lab).