If it is necessary for retirement or the kids' college, it might make much more sense to invest—a payoff is much more specific down the road, even if it does not quantity to a sexy six-figure verify.