I agree that blue and brown asbestos have risk levels an order of magnitude above that of white asbestos in terms of levels of exposure to the correct lengths of fine particles and that the identified risks from white asbestos are not fully quantified.
However it is fairly certain that a risk is still present for the white asbestos to cause lung problems, though generally at high concentrations of exposure.
Because that risk is present and because it is not possible to tell without analysis the type of asbestos and therefore if there are contaminants of the other types of asbestos (dependent on the source of the mineral) it is better to assume that "asbestos" in general does have a risk factor and to handle the material appropriately without taking account of the actual type, if you are considering asbestos from a layman's point of view.
However since the risk factors from undisturbed asbestos, i.e. there are no fibres present in the air, is at an absolute minimum there is no need to take extreme measures. prevention of the release of fibers is the best way to limit the risks.
Exposing asbestos removal workers to additional risk by removing the material and also presenting a potential risk to later occupants / users of an area should the containment not have been complete at the time of removal does not seems to me to be a good idea.