Code signing is a trip alright. I make a build tool called Conveyor that simplifies all this way down for developers [1] as part of making shipping desktop apps a single command, and damn has it been a lot of work.
First challenge: the tools provided by the OS developers only run on their OS and are of questionable quality, with bad error messages, poor documentation and unintuitive defaults. So I used a mix of open source and custom code to make signing work for every OS on any OS with good errors, docs and sample apps. The open source libs out there are slowly catching up with this, but they don't solve every part of the task and aren't integrated with everything else you need to ship apps (like laying out a working app tree, adding a software update engine etc). It's nice because it lets you ship from cheap Linux CI workers.
Second challenge: you get a proliferation of private keys. Keys to sign your Mac and Windows binaries, keys to sign your Mac update feeds, keys to sign apt repositories. So, Conveyor can derive them all from a single root key that you can back up on paper with a pen. Except for Windows where this worked up until Microsoft started requiring all keys to be held in an HSM.
Third challenge: said HSMs. They're a pain. As the Mozilla blog alludes, some tools like to require things to be done with a GUI. So, Conveyor can detect PKCS#11 drivers and drive the signing process itself using passphrases supplied via CI secrets or similar. It can also store the root key in the Mac keychain if you want to use that, which is a pretty nice way to keep it secure as macOS will ensure that only Conveyor can read it.
Fourth challenge: Windows CAs decided to squeeze extra revenue from devs by exploiting the new HSM requirement's incompatibility with cloud CI. They started offering "cloud HSMs" that can be used without any hardware, just a textual credential again (so removing much of the security an HSM is supposed to create!) and these are popular because everyone wants to ship from CI. Unfortunately these services all charge extremely high prices per signature once you go over a fairly small monthly allowance. If you want to ship builds on a daily basis, for instance, you can easily blow hundreds or even thousands of dollars on signatures alone. So Conveyor has a disk cache that stores signatures for binaries, and separates them from the underlying DLLs/EXEs so the cache doesn't fill up too fast even if you're shipping very large binaries, and a GitHub Action to integrate that cache with GitHub's own. All this saves a lot of signatures and therefore money.
Fifth challenge: Windows chooses to identify apps by the hash of their X.500 name as decided by their CA, but CAs constantly change their mind on how to format this name and change it for pointless reasons. Because the Windows ecosystem hates developers this can easily break app reputations/updates that are checking code signatures, and nobody at the CAs or Microsoft cares. So Conveyor has a whole system that recognizes when this has occurred and can drive an automatic reinstallation to repair things. The user doesn't notice except for a short delay on app startup (and seeing a progress bar).
Sixth challenge: platforms like the JVM that like to hide native code inside ZIP files, which may also contain native libs for the wrong CPU architecture or platform. Conveyor finds them, deletes the irrelevant binaries, signs the ones that remain and puts them back (or extracts them into the app directory and ensures they get found).
Seventh challenge: enterprises that have custom internal signing servers. So you can write a little script that submits binaries to the server and gets them back again. This works with all the above stuff too, when relevant.
Eighth challenge: MSIX. Microsoft's only non-deprecated packaging tech has a lot of benefits, and allows non-admin users to install packages, but the signature format is pure madness. By far the craziest scheme I ever saw. Reverse engineered and implemented.
Next challenge on the list: Azure Trusted Signing, Microsoft's new CA. It's marginally cheaper than the others, so some users want to use it, but it also doesn't provide a standard driver or use standard protocols. It'll require custom support.
Looking to the future, the path forward is to use kernel sandboxing tech to eliminate signing requirements for our users when possible. For apps that can fit inside the sandbox it should be possible for us to sign and distribute their apps for them, with full code signing only required if the app has special needs. That would give a much more web-like experience for non-web apps, where a developer can just create a scaffolded app with one command, ship it with a second and not think about any of the platform specific details or code signing at all. Unfortunately this is a way off yet.
First challenge: the tools provided by the OS developers only run on their OS and are of questionable quality, with bad error messages, poor documentation and unintuitive defaults. So I used a mix of open source and custom code to make signing work for every OS on any OS with good errors, docs and sample apps. The open source libs out there are slowly catching up with this, but they don't solve every part of the task and aren't integrated with everything else you need to ship apps (like laying out a working app tree, adding a software update engine etc). It's nice because it lets you ship from cheap Linux CI workers.
Second challenge: you get a proliferation of private keys. Keys to sign your Mac and Windows binaries, keys to sign your Mac update feeds, keys to sign apt repositories. So, Conveyor can derive them all from a single root key that you can back up on paper with a pen. Except for Windows where this worked up until Microsoft started requiring all keys to be held in an HSM.
Third challenge: said HSMs. They're a pain. As the Mozilla blog alludes, some tools like to require things to be done with a GUI. So, Conveyor can detect PKCS#11 drivers and drive the signing process itself using passphrases supplied via CI secrets or similar. It can also store the root key in the Mac keychain if you want to use that, which is a pretty nice way to keep it secure as macOS will ensure that only Conveyor can read it.
Fourth challenge: Windows CAs decided to squeeze extra revenue from devs by exploiting the new HSM requirement's incompatibility with cloud CI. They started offering "cloud HSMs" that can be used without any hardware, just a textual credential again (so removing much of the security an HSM is supposed to create!) and these are popular because everyone wants to ship from CI. Unfortunately these services all charge extremely high prices per signature once you go over a fairly small monthly allowance. If you want to ship builds on a daily basis, for instance, you can easily blow hundreds or even thousands of dollars on signatures alone. So Conveyor has a disk cache that stores signatures for binaries, and separates them from the underlying DLLs/EXEs so the cache doesn't fill up too fast even if you're shipping very large binaries, and a GitHub Action to integrate that cache with GitHub's own. All this saves a lot of signatures and therefore money.
Fifth challenge: Windows chooses to identify apps by the hash of their X.500 name as decided by their CA, but CAs constantly change their mind on how to format this name and change it for pointless reasons. Because the Windows ecosystem hates developers this can easily break app reputations/updates that are checking code signatures, and nobody at the CAs or Microsoft cares. So Conveyor has a whole system that recognizes when this has occurred and can drive an automatic reinstallation to repair things. The user doesn't notice except for a short delay on app startup (and seeing a progress bar).
Sixth challenge: platforms like the JVM that like to hide native code inside ZIP files, which may also contain native libs for the wrong CPU architecture or platform. Conveyor finds them, deletes the irrelevant binaries, signs the ones that remain and puts them back (or extracts them into the app directory and ensures they get found).
Seventh challenge: enterprises that have custom internal signing servers. So you can write a little script that submits binaries to the server and gets them back again. This works with all the above stuff too, when relevant.
Eighth challenge: MSIX. Microsoft's only non-deprecated packaging tech has a lot of benefits, and allows non-admin users to install packages, but the signature format is pure madness. By far the craziest scheme I ever saw. Reverse engineered and implemented.
Next challenge on the list: Azure Trusted Signing, Microsoft's new CA. It's marginally cheaper than the others, so some users want to use it, but it also doesn't provide a standard driver or use standard protocols. It'll require custom support.
Looking to the future, the path forward is to use kernel sandboxing tech to eliminate signing requirements for our users when possible. For apps that can fit inside the sandbox it should be possible for us to sign and distribute their apps for them, with full code signing only required if the app has special needs. That would give a much more web-like experience for non-web apps, where a developer can just create a scaffolded app with one command, ship it with a second and not think about any of the platform specific details or code signing at all. Unfortunately this is a way off yet.
[1] https://hydraulic.dev/