AFAIK (and I'm not a Windows person, so feel free to correct me) I/O completion ports are analogous to something like Linux's kernel AIO (asynch I/O). The model is to start an I/O operation on an fd (providing a buffer) and then be informed of its completion (with the data already copied into your process space), rather than the traditional Unix polling model of being informed when there is data waiting on an fd an then do a (non-blocking) read on it. If you're used to writing traditional Unixy event-based network daemons this seems pretty backwards.
One advantage of the aio model is that it at least theoretically allows "non-blocking" I/O of a normal disk file, whereas none of the polling-based systems I am aware of work with regular files. However, at least on Linux the kernel aio is notoriously fidgety (as it still may block the process if you take certain kinds of page cache misses; the solution is apparently to use O_DIRECT to bypass the page cache which has its own bag of problems), and most people fall back to using blocking reads with thread pools, or something like libeio or libuv that handles all that crap for you. (There's also the POSIX AIO that's built into glibc, but that done entirely in userspace [presumably with thread pools] and uses signal-based completion notification, which is kind of gross and probably pretty slow if you have a ton of I/O events happening.)
One advantage of the aio model is that it at least theoretically allows "non-blocking" I/O of a normal disk file, whereas none of the polling-based systems I am aware of work with regular files. However, at least on Linux the kernel aio is notoriously fidgety (as it still may block the process if you take certain kinds of page cache misses; the solution is apparently to use O_DIRECT to bypass the page cache which has its own bag of problems), and most people fall back to using blocking reads with thread pools, or something like libeio or libuv that handles all that crap for you. (There's also the POSIX AIO that's built into glibc, but that done entirely in userspace [presumably with thread pools] and uses signal-based completion notification, which is kind of gross and probably pretty slow if you have a ton of I/O events happening.)