The minister blamed his decision on the withdrawal of support by local councils and significant local protests.
Highways England was due to announce the preferred route for the controversial A27 scheme after six options had been considered.
Under original plans work was expected to start in 2019. The project involved upgrades to four junctions along the existing route south of Chichester.
In a letter to Highways England, Grayling said: “I have considered all the representations and understand that this is a controversial scheme.
“The support and funding contribution by the local authorities are both vital to its successful delivery. “After the withdrawal of support by the local councils for the shortlisted options and significant local campaigns, I am cancelling the scheme.
The Chichester scheme is one of a package of four schemes to relieve congestion on the A27 in West Sussex. Other projects are planned at Arundel as well as Worthing and Lancing.
Jim O’Sullivan, Highways England chief executive, said: “We are obviously disappointed at this decision as the improvement would have brought significant strategic benefits to the region. But any improvement had to be right for Chichester and there was no overall consensus.