OK now I understand what you are hoping to accomplish but I am not convinced it will help. I think it could just as easily cause more problems than it solves.
First of all, when you move from a situation where you were getting high prefix confidence to a situation where you are getting low prefix confidence (let's say you have moved to a position further from the microphone, or maybe there is some new background noise) then it won't listen to you any more. You will have to repeat yourself several times to get the prefix confidence requirement low enough again.
Also, if VC is hearing a false prefix with a low confidence, then this will also cause the required confidence to go down which will increase the chance of false positives. (learning from mistakes is not good when the machine thinks the mistake was not a mistake
)
Personally I think that finding a reasonable average value that works in most situations will be more stable, but I guess you have to try it out for yourself to see what works best for you. I'm just thinking about the logic of it, I haven't actually tried it, and I could very well be completely wrong!