Schmidt's comments follow a report Wednesday by the Washington Post based on documents provided by NSA leaker Edward Snowden that the agency tapped into communication links between the Google and Yahoo data centers.
"The NSA allegedly collected the phone records of 320 million people in order to identify roughly 300 people who might be a risk. It's just bad public policy…and perhaps illegal," he added. "There clearly are cases where evil people exist, but you don't have to violate the privacy of every single citizen of America to find them.
Schmidt said that greater oversight of the NSA is needed and that his company has filed complaints with the administration and Congress.
"The Snowden revelations have assisted us in understanding that it's perfectly possible that there are more revelations to come," he said.