You’re correct. The risk is there, but it’s not high.

It has been known for centuries that lead is toxic to humans. Chronic exposure to lead, even at low levels, is associated with an elevated risk of cardiovascular and chronic kidney disease in adults and of impaired neurodevelopment and subsequent cognitive and behavioural development in the foetus and young children. Health agencies throughout the world have moved from assuming that there are tolerable levels of exposure to lead to a recognition that valid ‘no-effect’ thresholds cannot currently be defined. Formerly, the most important exposure pathways were occupational exposure, water from lead plumbing, paints, petrol additives and foods. Regulation of products and improved health and safety procedures at work have left dietary lead as the main remaining pathway of exposure in European countries. Ammunition-derived lead is now a significant cause of dietary lead exposure in groups of people who eat wild game meat frequently. These are mostly hunters, shoot employees and their families, but also some people who choose to eat game for ethical, health or other reasons, and their children. Extrapolation from surveys conducted in the UK and a review of studies of game consumption in other countries suggest that approximately 5 million people in the EU may be high-level consumers of lead-shot game meat and that tens of thousands of children in the EU may be consuming game contaminated with ammunition-derived lead frequently enough to cause significant effects on their cognitive development.

https://pmc.ncbi.nlm.nih.gov/articles/PMC6675757/

Apparently, the human stomach dissolves small bullet fragments of metallic lead. That’s probably bad.

Experiments of solubility showed that lead fragments from bullets dissolve in chlo- ric acid of the same concentration as in the stomach of humans

https://www.livsmedelsverket.se/globalassets/publikationsdatabas/rapporter/2014/bly-i-viltkott---del-1-ammunitionsrester-och-kemisk-analys.pdf