Japan strongly protested against South Korean President Lee Myung-bak's surprise visit to Dokdo.
Prime Minister Yoshihiko Noda expressed regrets and said the visit is unacceptable.
Tokyo's Foreign Minister Koichiro Gemba recalled the country's ambassador to Seoul, Masatoshi Muto, filed official complaints with South Korean Ambassador to Japan Shin Kak-soo and made a protest call to South Korean Foreign Minister Kim Sung-hwan.
In the meeting with Shin, Gemba said Lee's visit to Dokdo was unacceptable in light of Japan's position, but the South Korean envoy maintained that the trip was part of an inspection visit to a provincial region and that Dokdo is South Korean territory over which Seoul exercises sovereignty.
Gemba also lodged a similar protest in the phone call with Kim, but Kim rejected the appeal.
"Foreign minister Kim made it clear once again that the Dokdo is our territory in historical, geographical and international terms, so then no dispute is possible," a ministry official said, adding Kim also expressed regrets over "undue measures Tokyo took over the matter."
Earlier in the day, Gemba urged South Korea to call off the Dokdo trip, warning that Tokyo would respond firmly to the move and the visit "would definitely have a large impact" on relations between the two countries. Senior diplomatic sources in Tokyo said that the annual finance ministers meeting between the two countries has been postponed over Lee's tour.
Reflecting on the impact of the surprise visit, Ambassador Muto told reporters as he boarded a plane for Tokyo that he was neither optimistic nor pessimistic about the future of bilateral relations, although the diplomat said relations have been improving slowly but steadily.
The envoy added that Japan had been aware of the possibility of the president making a visit but had only recently "confirmed" it, offering no details on the source.
He said Tokyo called on Seoul to reconsider because it was a serious matter.
The recall marks the second time that Japan's top diplomat in the country has been ordered back home. The first took place in 2005, when Tokyo's ambassador went back after publicly claiming that Dokdo belonged to Japan during a news conference in Seoul.
South Korea has recalled its envoy four times so far with the last time being in 2008.
Japanese media reported earlier that South Korea notified Tokyo of the planned visit, but officials in Seoul rejected those reports.
"Why do we have to notify Japan when our president goes to our territory?" an official said.
The environment and culture ministers accompanied Lee on the historic trip, officials said.
South Korea's military tightened security around Dokdo in anticipation of the visit.
"The military increased the number of Air Force combat planes and Navy vessels patrolling (near Dokdo)," a senior official said.
"The military reinforced its forces to prepare in case of an emergency, as Lee is expected to visit Dokdo."
(Yonhap News)
