Her Son Not Your Son
The end of פרשת ואתחנן has the Pasuk from where we learn that one is Jewish if his mother is Jewish. This is learned from the words, כי יסיר בנך מאחרי, when your son is removed from Me.
The Gemara in Kiddushin learns from here that a child born of a Jewish father and non-Jewish mother, is “her son, not your son.”
This is carried forth to such an extent that if one died childless, but had a half brother, both had a Jewish father, but the half brother had a non-Jewish mother (and he later converted), it is considered that he had no brother, and there would be no Mitzva of Yibum.
Similarly, if one had a son of a non-Jewish woman, and then he passed away, it is considered that he died childless, and his surviving brother, would need to fulfill Yibum.
This is clearly what Roth Gemara meant with the words, “her son, not your son.”