Outgroup (cladistics)

In cladistics or phylogenetics, an outgroup is a group of organisms that serve as a reference group when determining the evolutionary relationship among three or more monophyletic groups of organisms. The outgroup is used as a point of comparison for the ingroup, the set of organisms under study that specifically allows the phylogeny to be rooted. Because the polarity (direction) of character change can be determined only on a rooted phylogeny , the choice of outgroup is essential for understanding the evolution of traits along a phylogeny.

The chosen outgroup is hypothesized to be closely related to the other groups but less closely related than any single one of the other groups is to each other. The evolutionary conclusion from these relationships is that the outgroup species has a common ancestor with the ingroup that is older than the common ancestor of the ingroup. An outgroup may be a sister group to the ingroup or may be more distantly related.

The best outgroups satisfy two characteristics:

Therefore, an appropriate outgroup must be unambiguously outside the clade of interest in the phylogenetic study. An outgroup that is actually nested within the ingroup will, when it is used to root the phylogeny, result in incorrect conclusions about phylogenetic relationships and trait evolution. In molecular phylogenetics, satisfying the second requirement typically means that DNA or protein sequences from the outgroup can be successfully aligned to sequences from the ingroup.

In each example, a phylogeny of organisms in the ingroup may be rooted by scoring the same character states for one or more members of the outgroup.

...
Wikipedia