The data helps determine various government benefits, including employment insurance.
Officials are supposed to gather data from all firms with 500 or more employees but in Tokyo only about one third of 1,400 such firms were surveyed.
The scandal dates back to 2004 and a total of 53 billion yen ($490 million) will be repaid to 20 million workers, according to the ministry.
"I have received a report from the labour and welfare ministry that they need to provide employment insurance and other payments retroactively," top government spokesman Yoshihide Suga told reporters.
"We will make adjustments to make a necessary budget allocation in the fiscal 2019 budget" for the year starting in April, he said.
He added that the government was now probing dozens of other major data sets.
The labour minister has admitted he received a report about the problem as early as December 20.
The ministry nevertheless went ahead and published data on December 21 and January 9 that it knew had sampling problems, raising questions about the reliability of official statistics in the world's third-biggest economy.
The monthly labour survey has been watched by the government and the Bank of Japan as a clue for their economic policy decisions.