By default, VisionLib collects and stores further poses on-the-run, in order use them as fallback poses, when tracking gets lost. Sometimes, these poses might be corrupted or inappropriate, though.
You can either call a “soft reset”, which sets VisionLib tracking state to “lost” and resets to the original init-pose. If this doesn’t help, you can further call a “hard reset”, which on top clears the collected pose cache. More on this in this docu article on (re-)initialization.