Before I read up on how Alien works, I have to guess: it uses dlsym() (or the Windows equivalent) to find a function by name, and has a number of prototypical function pointer types which it uses to call the symbol returned by dlsym(), based on the number and types of parameters passed in Lua. When calling C++ it could even check the parameter types, provided they know how name mangling works for whatever platform the run on.

Now to see how close I was...