my guess is that the cache tags or tlb entries loaded on failed speculative accesses are wrong (maybe the valid bit is set but the address wasn't changed, or the user/supervisor protections are munged), that could leave you with a cache line or page tagged as user accessible but really protected kernel data