Tom now heads up the AIM-listed investment company Marwyn with Simon Vivian, another industry heavyweight and ex-chief of Hanson Materials.
The two are understood to be talking about buying the Tarmac business completely or investing cash for management control, leaving Anglo with a substantial stake in the UK aggregates business.
HSBC is reported to have been hired to advise them on a possible deal.
Peter Tom, the founder of Aggregate Industries and chairman of Leicester Tigers rugby club, stepped down after selling the aggregates business to Holcim, the Swiss industrial giant, for £1.8bn in 2005.
The veteran businessman re-emerged from retirement in 2008 to become chairman of Marwyn Materials, a shell that is planning to make acquisitions in the building materials industry.
Anglo declined to comment on the takeover reports about Tarmac, which it has been seeking to divest for years.
“As we have said, we will divest of Tarmac in a manner and on a timetable that maximises value for Anglo American’s shareholders,” said a spokesman.
Analysts value Tarmac at £2.6 – £3bn.
Earlier this year, Anglo sold the European operations of Tarmac for just over £250m as part of a restructuring programme that includes billions of pounds-worth of cost savings.