“Nephew” at one time was a gender neutral term, but since around the 17th century has referred nearly exclusively to male children of one’s siblings or brother/sister-in-law’s children.  The word “nephew” comes from the Old French “neveu” meaning “grandson, descendant”, which in turn comes from the Latin “nepotem”, meaning “sister’s son, grandson, or descendant”.  The first documented case of “nephew” being used in English was in the 14th century.

Today, because we now lack a gender neutral word to refer to a niece or nephew, it has been suggested that the slang word “nibling” be adopted, derived more or less from “sibling”.

References:

• Etymology Nephew
• Slang Words
• Oxford: Nephew
• Nephew