Ask Mr. Smarty Plants

The Poisonous Plants of North Carolina database says that all parts of Asclepias species are poisonous, but are "Toxic only if large quantities are eaten."  The Toxic Plant Database of Purdue University Veterinary School agrees with this assessment.  Moreover, the plants are generally unpalatable.  Grazing animals don't readily eat them unless they are confined in a pasture with large numbers of plants or the plants have been included in hay given to them.  The compounds that are toxic in the milkweeds are cardiac glycosides.  The monarch larvae eating the plants sequester these toxins in their exoskeletons where they remain even after they metamorphose into butterflies.  Not only do they make the butterflies taste bad, but the glycosides have an emetic effect that causes a predator, such as a bird, to vomit.  Predators apparently remember and associate the butterflies with the unpleasant taste and learn to avoid eating the monarch.

According to the Botanical Dermatology Database, several species of the family [e.g.,  Asclepias viridis (Green antelopehorn)] can cause dermatitis from the milky sap so you should probably protect the plants to keep them from being broken and exuding the sap.

Since milkweeds are unpalatable and they don't contain attractive fruits that might tempt the children to eat them, I don't think that you should be overly concerned about including them in your butterfly garden.