I heard it said that OpenGL drivers are essentially run-time compilers :) Working on a driver like this would not actually involve much knowledge of graphics. It's more an exercise of careful spec interpretation and deep knowledge of hardware-level interface.

Have a read through https://fgiesen.wordpress.com/2011/07/09/a-trip-through-the-... and see if you'd find that space interesting to work in.