Ask Mr. Smarty Plants

You are correct that plants in the Genus Prunus (includes wild plums, cherries and peaches) are toxic to livestock and, in particular, ruminants.  It is cyanide poisoning that is the problem and cyanide poisoning can kill quickly—within 15 minutes.  Ruminants are particularly susceptible because the release of cyanide from the cyanogenic glycosides that reside in the seed and leaves of Prunus plants occurs more easliy in their stomachs compared to the acidic environment of non-ruminant stomachs. Here is information from the Texas Toxic Plant Database and University of Pennsylvania Poisonous Plants Database.

You do want to be certain that the plum trees aren't in the pasture with your goats.  Either you need to fence the goats out of that area—good luck with that since goats are notoriously difficult to discourage by mere fencing—or you need to remove the roots of the trees from the ground to keep them from resprouting.  As long as there are still roots there, they will continue to sprout. Five to ten acres-worth is going to be a lot of work if you do it by hand so probably your best bet is to find someone who can dig up the roots mechanically (e.g., backhoe or skid steer).