Reducing infamy PS3 oblivion?

I have started the KoN quest for the relics and then decided to do others. Consequently, I have racked up 62 brilliant infamy points. I am at 101 fame and every time I go around to try to do the Pilgrimage quest again, each alter tells me: "Continue your penance to earn to gods' forgiveness". I know I must do the quest again, but is there somewhere I should start or something I am missing? Ugh, I ultimately found this heartbreaking news during the Path of Righteous quest when I could not put on the Boots of the Crusader to cross to get the mace. HELP!!