Nigel Graham, the former co-owner of a Yorkshire carpet manufacturer bought by flooring conglomerate Victoria in 2019, has left the company.