We wrote an article on this issue some time ago and noted at that time that most tea is made by a mold-free process.
In fact the same would go for most plant foods as Aspergillus is an important component of soil and most plant material will have been in contact with soil and windblown soil. The amount of mold contaminating the plant material is probably very small and some will probably be washed off during preparation but will still not be completely free of Aspergillus material.
This is not a problem for most people and we have no reliable information telling us that anyone is badly effected by eating plant food carrying this much Aspergillus - even those who have increased allergy to this fungus.
Fermented teas are of course completely different. They have been cultured with the mould and will contain many thousands of times more Aspergillus material compared with non-fermented tea.