In the first phase of its history, the castle stood on a trapezoid-like area measuring 53 x 72 meters.
The castle had crenelated walls and the buildings, standing along the wall, are assumed to have been made of wood. Following the completion of the centre of the castle, the stronghold was further extended.

The commissioner of the building is believed to have been Count Iván Kőszegi in the last decades of the 13th century. Next to the inner side of the northern wall, a building measuring 29 by 11 meters was erected. The residential unit was upstairs, boasting heated bedrooms, while the farm buildings and the kitchen were downstairs. The newly built fifth tower and the battlements facing the other tower were important improvements enhancing the defensibility of the castle.

The next construction phase was completed in the 15th century when the country’s richest and most powerful oligarchs, the Garais, extended the building according to new principles and with an intention to impress.
Based on the stoves, the first construction works commenced as early as the 1420s.
Once the palace on the northern side was completed, the next phase focused on the western part of the building.

The Kőszeg estate belonged to Frederic Habsburg III between 1445 and 1482, and since the land was a pledged property, it often changed hands. From 1586, the pledge was passed on to the Széchy family; then the property rights were purchased by the Esterházys in 1695 who owned the castle up to 1932. With the acquisition of the castle and the surrounding estate, the Esterházy family extended their already vast estates in the western part of Hungary. During this period of time, the castle of Kőszeg functioned as the centre of their estate.

The most notable event recorded in the modern history of the castle was the fire of 1777 which destroyed some of the towers and in the wake of which the northern wing and the courtyard was also completely rebuilt.
From 1932 onwards, the castle was used as barracks for border guards.