Education Secretary Arne Duncan said there are 50,000 drop outs in New York City — and they're being lost to the streets.

Speaking at Rev. Al Sharpton's National Action Network in New York on Wednesday, Duncan called education the civil rights issue of our time and said there are 103,000 ninth-graders in the city but only 53,000 12th graders.

"Where are they all going?" Duncan asked. "Are they all going early to the NBA? Is Bill Gates coming and saying, 'I wanna hire all you drop outs?'"

Duncan said the city is losing kids to the street, and he outlined what he said was necessary to improve the education system in order to compete with other countries: investing more in early-childhood education, reforming K-12, raising standards and ensuring some form of education beyond high school.

"If you can ride in the front of the bus, but you can't read you’re not truly free," Duncan said.