Members of the HBF trade body delivered an 89% vote against the skills body.
The HBF is launching its own skills training initiative this week committing companies to working together and with subcontractors.
Federation members said they were fed up with the “overly bureaucratic nature of the process builders have to go through” to access CITB training funding.
HBF Executive chairman Stewart Baseley said: “The vote demonstrates the frustrations house builders feel with CITB and the training regime currently in place.
“The industry simply does not feel that CITB is providing the support and framework it needs to train its staff despite the huge amounts being paid in levy by home builders.
“House builders desperately need a training body focussed on its requirements with which it can work closely to develop training regimes that are easily accessible to companies large and small.
“We hope this will be the wake-up call CITB needs to drive root and branch change through its entire organisation.”