In their [workers'] social relations, too, there has been an entailed retrogression rather than a progression. The wage-earning factory-hand does, indeed, exemplify entirely free labour, in so far that, making contracts at will and able to break them after short notice, he is free to engage with whomsoever he pleases and where he pleases. But this liberty amounts in practice to little more than the ability to exchange one slavery for another; since, fit only for his particular occupation, he has rarely an opportunity of doing anything more than decide in what mill he will pass the greater part of his dreary days. The coercion of circumstances often bears more hardly on him than the coercion of a master does on one in bondage.
From Chapter IX of Principles of Sociology. The mainstream critics of Spencer, ignorant as they are of what the man actually wrote (see this New York Times article), would be surprised by this passage.
Spencer does not appear to attribute the workers' predicament to government intervention, but rather treats it as something inherent in the nature of things. At the end of the chapter he write, "...there appears to be no remedy."