This is a broader fundamental question which you are better off asking AI to give you an overview. It involves various techniques like;

1) Paleo genomics i.e. DNA study of any surviving material.

2) Statistical study of Human Morphology i.e. variation w.r.t. current day human (and close primates?) biological structures. Eg. Current-day human cranium sizes vary within known limits. So if we find a fossil cranium which is outside these limits then there is a high chance it might be a different species.

3) Fossil dating shows different species (w.r.t. above data) occupying the same geography and within overlapping time periods.