We use it for hooking our JS unit tests for http://lanyrd.com/ in to our Jenkins continuous integration server. Our tests are written in QUnit (so we can run them in a regular browser), but Phantom is used to execute them from the command line as part of a Jenkins task.
We also have a separate Phantom/Casper script which we use to test that our Twitter login flow is working.
At my job, we do A LOT of UI testing using selenium. We mainly use the FireFoxDriver, and i did some basic tests comparing it to phantomjs, and phantomjs is much faster!
If we use the plain HtmlDriver than it might be as fast, but that doesn't use WebKit, nor has good support for C#... :(
