Why can't I scroll through my contacts faster

I can only scroll 4 or less contacts
per scroll

Hi, welcome to iMore.

I just tried it out and I am able to scroll 15 or more contacts at a time (didn't actually count but it's way more than the 4 than you mention).

Join our forum and reply in this thread as to which iPhone you're using.