Our app was recently rejected specifically for this reason, though we had a "skip" button, contacts weren't just automatically 'farmed'. So we had to add a popup with explicit allow/deny buttons and then the app passed subsequent reviews.