The European Environment Agency (EEA) listed tiny airborne particles and ozone as posing a "significant threat".
However, the authors said nations had significantly cut emissions of a number of pollutants, including sulphur dioxide, lead and carbon monoxide.
In a separate study, research identified a link between low birth-weight and exposure to air pollution.
EEA executive director Hans Bruyninckx said that EU nations had made considerable progress over recent decades to reduce the visible signs of air pollution, with cities now no longer shrouded in blankets of smog.
However, he added: "Air pollution is causing damage to human health and ecosystems. Large parts of the population do not live in a healthy environment, according to current standards.
"To get on to a sustainable path, Europe will have to be ambitious and go beyond current legislation."
The EEA report showed that data suggested that up to 96% of the EU's urban population was exposed to fine particulate matter concentrations above UN World Health Organization (WHO) guidelines.
Even more, 98%, were subject to ground-level ozone concentrations above WHO recommended levels.
As well has urban outdoor air quality, the report also highlighted that the natural environment was also continuing to suffer.
It said ecosystem were subject to the pressure of air pollution impairing vegetation growth and harming biodiversity."
The EEA also produced country-by-country breakdown of air quality data.
Responding to the report's findings, Environment Commissioner Janez Potocnik said: "Air quality is a central concern for many people.
"Surveys show that a large majority of citizens understand well the impact of air quality on health and are asking public authorities to take action at EU, national and local levels."
He added that he was willing to address those concerns in the Commission's Air Policy Review.
A separate study, also published on Tuesday, concluded that a substantial proportion of the cases of low birth-weight (less than 2.5kg at 37 weeks of gestation) "could be prevented in Europe if urban air pollution was reduced".
The findings, published in the The Lancet Respiratory Medicine journal, said: "The population attributable risk estimated for a reduction in [particulate matter with a diameter of 2.5 microns or less] concentration to 10 micrograms per cubic metre during pregnancy corresponded to a decrease of 22% in cases of low birth-weight at term."
A team of European researchers carried out what they describe as one of the largest studies of its kind, collating data from more than 74,000 births between 1994 and 2011 across 12 European nations.
They explained that babies with low birth-weights were at greater risk of mortality and health problems than infants with higher birth-weights.
"Low birth-weight has been associated with wheezing and asthma in childhood, and with decreased lung function in adults," they observed but added that there was inconsistency in the findings.
"In addition to active and passive smoking, atmospheric pollution exposure is a highly prevalent and controllable risk factor for low birth-weight."
Lead author Dr Marie Pedersen from the Centre for Research in Environmental Epidemiology in Barcelona, Spain, observed: "The widespread exposure of pregnant women worldwide to urban ambient air pollution at similar or even higher concentrations than those assessed in our study provides a clear message to policymakers to improve the quality of the air we all share."
Current EU legislation has sent the annual mean limit on fine particulate matter at 20 micrograms per cubic metre for particles measuring 2.5 microns (PM2.5) or less.
This is twice the concentration outlined in World Health Organization (WHO) guidelines, which recommends an annual mean limit for PM2.5 exposure to 10 micrograms per cubic metre.
These microscopic particles (the diameter of a human hair ranges between 15 and 180 microns) end up in the atmosphere from a range of sources, including road transport emissions, and have been linked to heart and lung disease, cancer and premature death.
Prof Bruyninckx described the Lancet paper's findings as "very concerning", adding that despite progress being made in some areas, more action was needed to tackle air pollution.
"It is the explicit goal of the European Commission to narrow that gap, and in the long run, close that gap because we are... concerned about citizens' health," he told BBC News.
But he explained: "Before you make binding legislation, you want to be on absolutely solid ground scientifically.
"Now, we know that [particulate matter] is having a significant impact so we need to adjust our rules and regulations accordingly."
Commenting on the findings published in The Lancet, Royal College of Obstetricians and Gynaecologists spokesman Dr Patrick O'Brien said the research was "very helpful in providing further evidence on the potential health impacts of air pollution".
But he added: "Exposure to some level of air pollution is unavoidable in day-to-day life and the risk still remains fairly low.
"Other factors, such as smoking, high blood pressure or excessive alcohol consumption, may contribute more to the risk of having a low birth weight baby."