Spanish speaking people tell me they find the Cuban accent difficult. Of course, it's all inter-comprehensible, but you're not refuting the OP's point.
A kind-of-rule based in personal observations, is that spanish people closer to the sea talk FASTER than the rest.
We have somebody in the family that is know as:
"Blah ahk guak fiuya sdsds caramañola ñame!". And " caramañola ñame" is the only thing you will understand 100% of him ;) -- the other text is not a perfect rendition. In fact, nobody agree how exactly him talk when left him at full speed!
How fast is the talking is not the only issue. Is the slang. We can have the same language... but, o boy! the slang is diverse even in the same country.
And some words have reverse meanings, or don't make much sense.
Oh man, don't get me started. I'm trying to learn Spanish, and if I watch a movie from Spain, there's no "v" sound, everything is "b". If I watch a movie from Mexico, it's the other way around! Trying to get the accent right on my trips to Spain are pretty frustrating to me, although it's a pretty minor difference, so it ends up fine.
Surely both sounds exist in both vatieties? Mexicans don't pronounce 'bueno' with an initial fricative, and people from Spain don't pronounce 'estuvo' with a plosive at the beginning of the final syllable.
Probably it has to do with socioeconomic classes and level of education. Usually uneducated people or people from rural areas sound, for the lack of a better term, "crude" to big city or urban ears in universally all languages and Spanish is no exception. You have to judge a language or dialect based on how they communicate in official mass communication media like Radio or TV and not some guy you met on the street.