The giant castle of Visegrád was built during the 1250-1260's after the Tartar invasion. The castle has two main parts, the lower castle and the citadel.

These two are connected by walls closing down the valley. The walls go all the way down to the Danube. The centre of the lower castle is situated between gates in the walls. The large hexagonal shaped tower was called Water tower in the middle ages, now it is called the tower of Solomon.

The name remained with the tower built in the 13th century due to a mistake as in the 19th century it was thought that the tower was built by the Romans and the King Solomon was held prisoner among the walls of the tower. However, it is now known that the brother of Solomon, László I. held him in the castle of Visegrád. In reality, the tower did not exist in the 13th century and Solomon was held in the castle standing on Sibrik Hills near the castle of Visegrád.

The Gothic living tower of the lower castle was built by Béla IV. as a hunting resort, to secure himself from a new Tartar attack and defend the road from Buda to Esztergom. In 1317 the castle was seized by the flowers of Róbert Károly who relocated his seat to Visegrád and built out a comfortable dwelling place for himself. The chapel for St. John was probably built before 1325 on the top of the tower.

The inner tower walls were probably built around the same time, as well as the terrace south of the tower with the mint. In 1357, the bell foundry of Konrád was also located in this building. Konrád was exempt of tax or his grand bell.

The dwelling tower was used as a store in the 15-16th century. The south part of the inner tower was fortified with a cannon terrace. In October 1540 the warlord of the King Ferdinand, Leonhard Fels seized the castle held by the followers of King John. The south corner of the tower came down during the attack. Ever since then, the tower is nothing but ruins.

The Turkish occupied the castle in 1544. A little Turkish habitat was built where once the town stood. In 1595, the troops of Miklós Pálffy recaptured the castle from the Turkish. However, it was held by Christians only for a decade. In 1605 the Hajdú of János Bocskai seized the castle with the help of the Turkish.

Christian armies ousted the Turkish again in 1684. However, it was captured again by the Turkish in 1685. They burned down the building and left it for good. The castle was not rebuilt again. In the middle of the 19th century, the national romanticism rediscovered the importance of the castle.

The parson József Viktor initiated the rebuilding of the castle. The construction works began in 1871 with the supervision of Imre Henszlmann and János Schulek. They planned to rebuild the Tower of Solomon as a royal hunting castle. Nevertheless, the works were halted due to lack of finances.

In the 1910-20's the tower construction was finished by Kálmán Lux. The first archaeological compilation was placed in the newly reconstructed tower by János Schulek. The reinforced concrete slab and the wooden walls burned down in 1950. The reconstruction works began in 1959 and lasted until 1964 based on the designs of József Szanyi and János Sedlmayr. The new construction was ornamented by works of famed architects.

The tower now hosts exhibitions of the Mathias Rex Museum. A well from the Zsigmond era is located in the yard while the most important statues from the King Mathias era are located on floor one: a late Gothic Lion Well, a Renaissance Well of Muses, pieces of Renaissance altars, among them the Madonna of Visegrád. An archeological exhibition introduces the visitors to the history of Visegrád on the top floor.