"This is a clear indication that Modi's popularity may now have peaked," said Satish Misra, a political analyst at the Observer Research Foundation. The loss in Bihar will also hamper Modi's push to pass economic reforms, because he needs to win most state elections in the next three years to gain full control of parliament.
Investors are already fretting over the speed of change in Modi's India, and worries over an additional stumbling block will likely knock financial markets on Monday.
In the most significant vote since he won power 18 months ago, Modi's Bharatiya Janata Party (BJP) lost in Bihar after running a campaign that sought to polarise voters along caste and religious lines.
It was the most expensive state election ever fought by the BJP, with more than 90 top party figures addressing 600 rallies over the last six weeks, party officials said.
"The Bihar election was a very important battle for us. We will have to analyse each and every aspect of the result," said Ram Madhav, a BJP general secretary. "There are lessons to be learned."
An anti-Modi alliance led by Chief Minister Nitish Kumar was ahead in 179 seats in the 243-seat regional assembly, an overwhelming majority, tallies compiled by the election commission showed.
Modi tweeted that he had called to congratulate Kumar, whose regional "grand alliance" could now become a template for politicians seeking to prevent Modi's march towards untrammelled power under India's federal system.
The defeat could also dampen the mood as Modi heads to Britain for the first bilateral visit by an Indian leader since 2006. Modi is due to address a crowd next week at London's Wembley stadium.
GOVERN, DON'T CAMPAIGN
Modi's BJP-led alliance was ahead in 58 seats where trends were clear.
Some regional party leaders expressed bitterness over a campaign that thrust Modi into the spotlight - he addressed more than 30 rallies - turning the election into a referendum on his personal leadership.
Analysts said an Indian prime minister has never before invested so much time in a state election.
"The role of the prime minister is to govern the country, and not become the lead campaigner in a state election," one senior BJP state leader said, asking not to be named.
Modi's campaign started with a message of economic development, then, as the race began to tighten, his party shifted to appealing to caste and religious alliances.
The slaughter of cows, an animal revered by the majority Hindu population, became a major topic. Members of Modi's party also expressed concern about the rising Muslim population.
Jyotiraditya Scindia, a leader of the opposition Congress party, said the BJP must end campaigning on issues that fracture the country along religious lines.
"This is a decisive mandate against divisiveness in favour of development," he said.
Bihar is one of its biggest electoral prizes and the most pressing challenges of India prevail there, including widespread poverty, corruption and poor infrastructure. If independent, its 104 million people would be the world's 13th-largest nation, more populous than Germany.
The loss will make it harder for Modi to secure backing for reforms in parliament's upper house where his party is in a minority and seat allocations are dependent on parties' strength in the states. His government has struggled to pass laws, including the biggest overhaul of taxes since independence.
"It raises the likelihood that the opposition will use this mandate to block important bills," said Milan Vaishnav of Carnegie Endowment for International Peace.
This may have been Modi's last chance to win a state election before the spring of 2017. He faces five elections next year in regions where his party has failed to make inroads.