The move by Uzbek General Abdul Rashid Dostum follows appeals by relatives of the captives at an assembly which adopted a new constitution on Sunday after nearly 25 years of occupation and civil war.
The fighters have been held in the prison since it fell to US-led forces in late 2001.
At the height of the US-led war that led to defeat of the Taleban regime, Dostum's forces captured hundreds of Taleban and al Qaeda fighters during battles in the north.
Dostum's loyalists are accused of allowing hundreds of Taleban fighters to suffocate while being transported to the prison in metal containers. Dostum denies the charge, but says many may have died from wounds sustained during fighting.
He has already released hundreds of Taleban militants from his jail, including many Pakistanis.
Faizullah Zaki, Dostum's spokesman, said on Monday that around 450 Taleban fighters and a similar number of Pakistani prisoners were being held at the jail.