When you end the dream chosen quest by giving up, you get the a journal entry for the slaver goblins saying you gave up that quest as well.