Disclaimer: I am not a biologist. I'm interested in this topic but I haven't studied it that much. These are my random thoughts on how a species can form:
It seems to me that we only “need” new species when things are going badly. In these stressful situations, the population of an existing species will decline. A small population is more likely to lead to inbreeding/incest. What happens with inbreeding? We get increased mutations. The history of royalty in Europe has some examples (hemophilia, six fingered folk, etc.). It's exactly when things are going badly with a species that a new species has a chance, and I think it's no coincidence that mutation rates are higher then.
More specifically, I think susceptibility to mutation is an evolved characteristic. Species that “fixed” the problem with genetic errors would not evolve, and would eventually be wiped out. Only the species that had mutations would survive. Secondly, I think the mutation rate varies, and it responds to stress and inbreeding, not by accident, but because a variable mutation rate evolved as well. When a population no longer fits well into the environment, it needs to increase the mutation rate so that it can turn itself into a new species.
A consequence of this line of thinking is that when populations are large, we should rarely see new species form. We shouldn't see many new species forming until the environment changes drastically.
I also think in extreme cases, a very small population might lead to asexual reproduction with a high mutation rate. Species that allowed for asexual reproduction in rare cases are more likely to survive.
If small populations lead to new species, what would we observe?
- If there are several populations of a species, and one of them mutates into a new species, we will see a new species and call it a “branch” on the tree of life.
- If on the other hand the small population is the only surviving instance of a species, then as it turns into a new species, the old species will be wiped out. There may be no record of it. In the tree of life, we would only see two distinct species if the creatures are physically different and if both populations left fossils.
I think most new species are of the latter form, and never show up in the fossil record. How could I call this a new species then? If we had a time machine and brought a creature of the first form forwards in time, and it tried to breed with a creature of the second form, we'd be able to decide whether the old and new creatures are genetically compatible. If they are, I'd say they are the same species. But in a lot of cases, they won't be able to interbreed, and we have a new species. We can't really test this without a time machine.
To summarize: I think that variable-rate mutation is an evolved behavior that shows itself when populations are small and stressed, and that there have been a lot more species than the ones we see in the fossil record.