Over the centuries of its existence, Lida Castle has repeatedly suffered the alien attacks. In the XIV century, the castle walls repeatedly withstood the onslaught of the Crusaders. They first appeared against the Castle walls in 1384. After a long siege and assault, the enemies captured the castle. The second raid took place in the winter of 1392. They burned down the surrounding buildings, looted the city and captured the Castle. Two years later, the Crusaders attacked Lida again, this time the Castle withstood.
At the beginning of the XV century, the troops of Prince Yuri of Smolensk appeared at the walls of Lida Castle, who came to free his family, captured by Vytautas.(Vitovt). The Smolensk host burned down the city, stormed the Сastle several times, but didn't manage to seize it.
In 1410 by the order of Vytautas (Vitovt) and under his general command, Lida Chorągiew (banner), which participated in the Battle of Grunwald, went on a campaign against the Crusaders from the walls of Lida Castle.
In July 1506, Prince Alexander of Lithuania and King of Poland gathered a militia in Lida for the Battle of Kletsk. During the protracted wars between the Polish-Lithuanian Commonwealth (Rzeczpospolita) and the Muscovite state, the Castle stone walls experienced the strength of the latest siege technology at that time. After a long siege and cannon fire in the summer of 1659, the 30,000 Russian troops of Prince Ivan Khovansky stormed the Castle.
A strong blow to the castle was inflicted by the Swedes, who blew up both towers at the beginning of the XVIII century and the fortification lost its strategic importance. In the summer of 1794, the rebels of T. Kosciusko fought in the ruins of the Castle against the tsarist troops.