Lubbers, who is based in Geneva, held an impromptu news conference after seeing Annan and said the secretary-general had not asked him to step down.
He said he intended to complete his five-year term, which ends on December 31, despite allegations from women who worked for UNHCR that he had groped them.
But Annan indicated the meeting between the two men had been much tougher than Lubbers acknowledged. His office said discussions had centered Lubbers' "future in the organisation," in light of published reports of sexual harassment.
Still, the reports published recently were known to Annan and Lubbers last summer when the United Nations effectively cleared Lubbers but the issue escalated.
Lubbers' visit had spurred rumours that he would be forced to leave his post after Annan last month appointed Mark Malloch Brown, the high-profile head of the UN Development Program, as his new trouble-shooting chief of staff.
The United Nations is already under fire for the scandal-tainted oil-for-food program and sexual abuse by peacekeepers in the Congo.
Lubbers told reporters that one woman, who had filed charges and then withdrawn them had no real evidence.
"There were two witnesses in the room who very clearly saw that I ushered the lady out of the room with my hand on her back, and that was all," he said.
As a demonstration he patted a male reporter in the small of the back and then said he would greet Nane Annan, the wife of the secretary-general, in the same manner.
"I have become more cautious now with women I don't know," Lubbers said. "But I must admit when I meet Mrs. Annan tomorrow ... I would lead her out of the room the same way (and) I will continue to be the gentleman that I was."
But UN sources said there was no social engagement with the Annans planned on Saturday.
Asked if his position had been made untenable because of the allegations, Lubber replied, "I don't think so."
"If the United Nations gives in to slander, it is against the principle of UNHCR," he said. "We are proud of our record" in caring for more than 17 million refugees around the world.
Lubbers, 65, who was Dutch prime minister from 1982 to 1994, is working for $1 a year. He has donated his salary, estimated at about $300,000, and travel expenses to the United Nations each year.
Last summer the organisation's watchdog, the UN Office of Internal Oversight Services, said allegations against Lubbers by four women in the UNHCR constituted a prime facia case and recommended "appropriate actions."
But Annan rejected the report. His spokesman, Fred Eckhard, said Annan "did not say there was no evidence" but that "he found the allegations unsustainable on a legal basis."
Further details of the OIOS report published in Friday's edition of the London Independent newspaper, includes unnamed women in the UNHCR describing Lubbers' gestures as far from innocent. Some women refused to file complaints, fearing reprisals, the report said.
Lubbers said he had responded to the allegations last year and Annan evidently came "to the conclusion my answer was convincing" and "therefore he closed the case."