He said Pakistan had an interest "in not just containing these organizations but ultimately eliminating" the groups.
"Quite frankly my view - and I expressed this to the leadership of Pakistan - is we also are concerned about the stability and security of Pakistan's government as well," he told reporters in New Delhi.
"This could lead to a threat to Pakistan's own stability. It is not in anyone's interests that the government of Pakistan be destabilised."
Tillerson's visit - the first to the nuclear-armed nation by a senior official from President Donald Trump's administration - follows months of pressure from Washington on Pakistan over its alleged support for Taliban militants.
It follows an unannounced stop in Kabul Monday, where Tillerson reiterated America's commitment to Afghanistan and warned that Washington has made "very specific requests" of Pakistan over militancy.
Trump has angrily accused Islamabad of harbouring "agents of chaos" who could attack US-led Nato forces in Afghanistan.
Pakistan has repeatedly denied the charge, insisting it maintains contacts only to try to bring the militants to peace talks.
Tillerson did express his appreciation to Pakistan officials for the sacrifices it has made in fighting militancy, and for its help in securing the release of a US-Canadian family held captive by the Taliban for five years.
But America's top diplomat - whose frosty visit to Islamabad lasted just four hours - said the United States would not tolerate extremist safe havens. He thanked India for its support combating extremism.
"In the fight against terrorism the United States will continue to stand shoulder to shoulder with India," he said.
New Delhi has welcomed US overtures for closer ties, saying it shares Washington's optimism about their burgeoning relationship.
Speaking ahead of his visit, Tillerson called for deeper co-operation with India in the face of growing Chinese influence in Asia, and said Washington wanted to promote a "free and open" region led by prosperous democracies.
Trump's top diplomat also said Beijing sometimes acted outside international conventions, citing the South China Sea dispute as an example.
Indian Foreign Minister Sushma Swaraj said that Trump's strategy for the region hinged on Pakistan eradicating militant havens on its territory.
"We believe that President Trump's new policy can succeed only if Pakistan effectively acts against all terror organisations without any discrimination," she told the press conference with Tillerson.
India has historically avoided alliances, preferring to maintain cautious relations with both Washington and Beijing, but Trump has developed a warm relationship with Prime Minister Narendra Modi.
Tillerson, who was to meet Modi later Wednesday, started the day by meeting national security adviser Ajit Doval.
He also laid a wreath at a memorial to Mahatma Gandhi, removing his shoes to approach a pillar marking the spot where Gandhi was shot dead by a Hindu fanatic on January 30, 1948.
Copyright Agence France-Presse, 2017