Why was the new hampshire state legislature unable to take control of dartmouth college?

The New Hampshire state legislature was unable to take control of Dartmouth College because it was a privately funded institution. As such, the government had no power over it, because it was created and got its money from individuals, rather than the state itself, meaning that those individuals, and not the state, owned the institution. Private property is protected by the US institution, so no state can break that law. 

In 1769, the charter of Dartmouth College was granted by King George III of Great Britain.

In 1816, the legislature of New Hampshire made an attempt to alter the charter of Dartmouth College so that they could reinstate the deposed president of the College, so that they could appoint new trustees and hand the power in the hands of the governor.

The trustees of the college objected and a decision was made by the Court, which invalidated the act of the New Hampshire legislature and ruled in favor if the college.