Aside from the quality threshold ( minInlierRatioInit ), which may be set too high in the -.vl config file for a good initialization, this usually might indicate bad calibration.
Whenever you feel that tracking and initialization are working only moderate, or your line model and your tracked object appear misfitting “here and there” while trying to match them, check if you have set the correct calibration for your (mobile) device or camera. During development, e.g. if you use a USB camera, VisionLib will output, if the calibration data for the device is missing.
There is a video tutorial on YouTube regarding calibration on desktops and calibration on mobiles, and an article in the documentation.