Ibn Taymiyyah said (Majmu' al-Fatawa 12/93):

It has been established by both textual evidence and consensus that the one who spoke with the speech of men, deliberately, [with speach that is not for the] beneficial interest of the prayer, knowing of its unlawfulness, his prayer becomes invalid.

And in the Majmu Fatawa (6/47) of the Permanent Committee (here):

Deliberate speech during the prayer invalidates it except in the case of the ignorant one and the forgetful. In that case, it does not invalidate it in the stronger view due to the hadeeth of Mu'aawiyah bin al-Hakam when a man sneezed next to him so he invoked Allaah's mercy upon him whilst in prayer and the Companions (radiyallaahu anhum) showed rejection against them. When he finished the prayer he came to the Prophet (sallallaahu alayhi wasallam) and asked him about that. And the Prophet sallallaahu alayhi wasallam) said to him, "Nothing of the speech of people befits the prayer, it is but tasbeeh, takbir and recitation of the Qur'an." And he did not command him to repeat the prayer. So this shows that speech in the prayer does not invalidate it when the one who speaks is ignorant of the sharee'ah ruling. And likewise speaking in the prayer for some benefit in the prayer, it does not invalidate it due to the hadeeth of Dhul-Yadayn.

And Shaykh Ibn al-Uthaymeen said (here), after mentioning the same hadeeth of Mu'aawiyah:

This is a general principle through which Allaah bestowed a favour upon His servants, it comprises all unlawful matters, when a person commits [anything from] all of the unlawful matters due to forgetfulness, or ignorance or compulsion, there is no sin upon him and there is no ransom or expiation.

And in Liqaa Baab al-Maftuh (no. 235), the Shaykh, whilst discussing the issue of man who fearing giving the imaam who leads the prayer the evil eye due to being impressed by his recitation, says "Maashaa'Allaah, Tabaraak Allaah ar-Rahmaan," said - as it relates to speech made out spontaneously (afawiyyan:

A person is not blamed for what is spontaneous, he is not held to account for it, even if he spoke, if a stone fell upon him whilst he was prayer and say "Ah", spontaneously, without intent, then nothing is upon him."