No — consonants trip people up just as much
Vowel pairs like ship and sheep get the attention, but consonant swaps cause real confusion too: TH vs S in **thin and sin**, or F vs P in **fan and pan**. Whether a pair is hard for you depends on which contrasts your first language never used.
