The Nikkei average erased early gains to close down 0.08 percent, or 9.1 points, at 11,651.02. It had ended Friday at its highest close since July 2.
The broader TOPIX index rose 0.01 percent to 1,166.74.
Analysts said although market sentiment remained solid, investors had become cautious about buying after the Nikkei had risen as high as 11,690.49, near technical resistance at 11,700.
The US stock market is closed on Monday for a holiday, another reason for investors to trade more cautiously.
Trade slowed, with 1.296 billion shares changing hands on the first section, the smallest daily total since December 29. That compared with 1.370 billion on Friday.
Advancers outpaced decliners 818 to 634.
"The market might start jumping even higher in early March when selling by domestic institutional investors ahead of end-March book closing winds down," said Tachibana Securities adviser Masaharu Sakudo.
"Apart from that drug (makers' merger) news, we haven't got many incentives. But investors are seeking the next reason to buy, and relief that foreign investors are continuing to buy stocks is also providing good support to the market."
Sankyo, Japan's second-largest drug maker, rose to 2,460, its highest close since July 2004. Daiichi, the sixth-largest, climbed to 2,565 yen.
The Nihon Keizai Shimbun and other media said on Saturday that Sankyo and Daiichi planned to merge under a holding company with annual sales of more than 900 billion yen ($8.51 billion), trailing only Takeda Pharmaceutical Co domestically.
The reports fanned speculation of mergers among other drug makers, pushing up Eisai Co shares by 2 percent to 3,580 yen. Mitsubishi Pharma gained 1.4 percent to 1,054 yen but Takeda ended unchanged at 5,010 yen.
Advantest, the world's largest maker of chip-testing equipment, fell 1.2 percent to 9,080 yen. The company said it aimed to buy back 6 million of its own shares, or 6.01 percent of the outstanding total, for 54.48 billion yen.
Chip maker Elpida Memory shed 5.9 percent to 4,510 yen on profit-taking following a 19 percent, seven-day winning streak that had sent its shares to a record close of 4,790 on Friday.
Citizen Watch Co gained 0.6 percent to 988 yen. After the closing bell, the precision equipment maker said its nine-month operating profit rose 7.2 percent but it cut its full-year outlook.
Daiwa Securities' strategist Tsuyoshi Nomaguchi said he saw more upside potential for technology stocks than those of shares tied to the domestic economy, and that many investors seemed "overly optimistic" about Japan's economic outlook.
"I don't think Japan's economic outlook is as bright as some say it is, and considering the inventory-adjustment cycle, I see tech stocks as more attractive than domestic stocks," Nomaguchi said.
"Still, market fundamentals are such that the market is not ready to push the Nikkei up to the 12,000 level.
We need to see more good numbers from upcoming economic data to justify further buying."
Key economic figures awaited this week include a key gauge of conditions in service industries and trade data, both due on Wednesday, and consumer prices data on Friday.
UFJ Holdings Inc, the most actively traded issue by turnover, rose 1.2 percent to 586,000 yen after Mitsubishi Tokyo Financial Group (MTFG) offered $29 billion to buy Japan's fourth-largest bank.
Mitsubishi Tokyo's offer to effectively swap 0.62 MTFG share for one share in UFJ values the smaller group at a 3.1 percent premium to its closing share price on Friday, and comes six months after the two lenders agreed in principle to merge.
By contrast, Sumitomo Mitsui Financial Group's unsolicited bid for UFJ in August, which has gone unanswered, now values UFJ at a 23.5 percent premium, or nearly $35 billion.
Shares of MTFG slipped 0.1 percent to 962,000 yen and SMFG put on 0.6 percent to 719,000 yen.