Their land came to be known as Kafiristan

C. M. Naim in Outlook India:

When in 1888 Rudyard Kipling sent off his two rascally heroes to become kings in Kafiristan, this is how he described their first sighting of the local people: “Then ten men with bows and arrows ran down that valley, chasing twenty men with bows and arrows, and the row was tremenjus. They was fair men, fairer than you or me, with yellow hair and remarkable well built.” (Sadly, the 1975 film based on the story was shot in Morocco and not in Chitral, and John Huston’s “natives” were swarthy and dark-haired, true only to Hollywood anthropology.) Linguists who studied the relevant languages have declared them as old as the time when Aryan and Iranian languages had not branched away from each other—even older. These people made their home in a remote region, extremely picturesque but not possessing the wealth that attracted marauders and empire builders. Various invading hordes seemingly skirted them. And when the diverse people around them became Muslim, they collectively came to be known as “Kafirs,” and their land as “Kafiristan.”

However, what could survive ancient marauding failed against the combined might of 19th century colonialism and nationalism. The British in India came to terms with the Pathans in Kabul in 1893 and put down the infamous Durand Line (1896) that cut through the land of the Kafirs. Soon after, the Amir of the new nation of Afghanistan invaded his portion of the divide to establish his sovereignty. Those who could do so fled to Chitral, whose Muslim ruler let them settle near their brethren.

More here.