2 to 3 IQ points not highly significant? 
Especially if compounded through generations. Never seen that brought up in any debate.