The Japanese capital won a majority of second-round votes from International Olympic Committee (IOC) members in Buenos Aires to beat Istanbul.
Madrid had earlier been eliminated in a first round of voting.
The announcement was met with jubilant scenes in Japan, as Tokyo prepares to host the event for the first time since 1964.
The Spanish capital went out after a tie with Istanbul in the first round, meaning Tokyo topped the voting, forcing a second decisive ballot.
The result stunned thousands of Spaniards who had gathered in Madrid into silence, as their dreams of winning the Games and a potential economic windfall were shattered.
Many who had been clutching red balloons let them fly into the air and began trudging home in bitter disappointment.
The tie echoes voting four years ago in Copenhagen for the 2016 Games, when strong contenders Chicago flunked at the first hurdle in a contest eventually won by Rio de Janeiro.
IOC president Jacques Rogge, presiding over his final session before retiring on Tuesday after 12 years in charge, announced on Saturday.
Madrid, Istanbul and Tokyo had earlier delivered their final 45-minute presentations for the vote, which one IOC member had said was too close to call.
Turkish Prime Minister Recep Tayyip Erdogan told delegates that by choosing Istanbul, seeking to bring the Games for the first time to a predominantly Muslim country, they would send out a powerful message to the Middle East region.
Erdogan has been at the forefront of moves to punish neighbour Syria for its alleged use of chemical weapons on its people.
"We live at a time when our region and the world wish for peace and at this critical moment we would like to send a strong message of peace to the world from Istanbul," he said.
"I would like to point out that giving the Games to a country where the population is predominantly Muslim would send out a powerful message, to our region especially, which is in desperate need of peace."
He added: "I see the Olympic Rings as being a powerful partner for that, symbolising peace, friendship and partnership."
Tokyo, previous hosts of the world's biggest sporting event in 1964, has been dogged by questions over the safety of the Fukushima nuclear plant, which suffered a meltdown in the wake of the devastating 2011 earthquake and tsunami that left more than 18,000 people dead.
More stories have emerged this week about contaminated water leaking into the Pacific Ocean.
But Japanese Prime Minister Shinzo Abe, who had flown in from the G20 Summit in Russia, reassured IOC members that the situation was under control, while Fukushima operators the Toyko Electric Power Co also said the damage was contained.
"It (the plant) has never done or will do any damage to Tokyo," said Abe.
Madrid looked to have gained a slight, late edge over its two rivals and had shown resilience throughout the race, against fears about the dire state of the recession-hit Spanish economy.
The bid made much of the fact it already has 28 of the 35 venues built -- honouring promises made in the previous failed bids -- and all the infrastructure in place.
Spanish Prime Minister Mariano Rajoy, also fresh from the G20 summit, told the IOC that Madrid offered a "safe, solid and reliable" while heir to the throne, Crown Prince Felipe, said success for the capital could spark wider financial recovery.
"Our organisation and finance are unquestionable, Madrid has and always will keep its promises," said Rajoy.
"It is probably the most reasonable financial Olympic bid in history. Most of the required investment has been made and what little remains is guaranteed by the Government.
"It carries no risk whatsoever to the Olympic Movement but numbers don't lie....
"We are on the road to recovery and our level of surplus has not been seen since the mid 80s."
The IOC meeting is made up of 103 members, of which 97 were eligible to vote in the first round -- members from the bid cities being ineligible.