Experts say that men between the ages of 25 and 40 have seen a discernible drop in their sperm counts. Lack of exercise, improper lifestyle choices, poor eating habits, stress, pollution in the environment, and addictions are all contributing factors. Environmental pollutants also have an impact.