Political analysts said the BJP's defeat would underscore rural dismay with the government and could help unite the opposition led by the Congress party. Modi is personally popular but has been criticised for failing to deliver jobs for young people and better conditions for farmers.
Reacting late on Tuesday to the results, Modi wrote on Twitter: "Victory and defeat are an integral part of life. Today's results will further our resolve to serve people and work even harder for the development of India."
The results came as a shot in the arm for Rahul Gandhi, president of the left-of-centre Congress, who is trying to forge a broad alliance with regional groups and present Modi with his most serious challenge yet in a general election that must be held by May.
Congress has ruled India for most of its post-independence era after 1947 but was decimated by Modi's BJP in national polls in 2014. Since then, it had struggled to make major inroads, even in state polls.
Gandhi, the fourth generation scion of the Nehru-Gandhi dynasty, has sought to build a coalition of regional groups.
On Tuesday, celebrations erupted outside the Congress party headquarters in New Delhi, with supporters dancing, setting off fircrackers and brandishing posters praising Gandhi.
"We are going to provide the states with a vision and a government they can be proud of," Gandhi told reporters.
"There is a feeling among people that the promises made by the prime minister ... have not been fulfilled."
In Chhattisgarh, Congress was ahead in 68 of the 90 seats at stake, with the BJP on 16, according to data from the Election Commission. In Rajasthan, the Congress was leading in 99 of the 199 seats contested, against 73 for the ruling BJP.
CONGRESS CONFIDENT
In Madhya Pradesh, the most important of the five states that have held assembly elections in recent weeks, Congress was leading with 113 seats while the BJP had 110 out of 230.
Regional parties were leading in two smaller states that also voted, Telangana in the south and Mizoram in the northeast.
Congress said it was confident it could form governments in all three big states. The BJP previously ruled all three, for three terms in two of them.
The BJP said the state results would not necessarily dictate what happens in next year's general election.
Investors said the BJP had not fared as badly as feared, and nationally would still likely have an edge over an opposition coalition in the general election.
"A disappointing set of state election results ... suggests that the ruling national party has lost some goodwill," London-based economic research consultancy, Capital Economics, said in a note.
"That said, we maintain our view that the BJP will secure victory in the general election, which would allow PM Modi to get his reform agenda back on track after a lacklustre 2018."
Markets recovered from sharp early losses and ended marginally higher, though the central bank governor's sudden resignation on Monday kept investors nervous.
A lawmaker for the BJP said it had erred in focusing its campaign on partisan themes, such as the building of a Hindu temple at a site disputed by Muslims, instead of offering jobs and growth.
"We forgot the issue of development that Modi took up in 2014," said Sanjay Kakade.