Experts are demanding access to the most historic building in Worthing.
And they warned Worthing Borough Council may have to compulsorily purchase Castle Goring if their demands are not met.
Conservation watchdogs are growing increasingly concerned about the condition of the Grade I-listed folly, owned by the Somerset family.
The building, which is partly occupied by a language school, was in 2002 placed on English Heritage's "at risk" register, which warned the building was in immediate danger of further rapid deterioration.
Now Worthing Borough Council wants urgent access to the building to assess its condition.
Landowner Clem Somerset said: "It is the first I have heard of it. Castle Goring is let to a tenant under a full repairing lease. The building is watertight as far as I know.
"There are always problems with the dome in the centre. To get Castle Goring back as it was would be totally beyond any individual's means as far as I am concerned. It would be out of the question."
Clare Mangan, the council's strategic housing and planning manager, said: "Castle Goring is a Grade I-listed building, one of only two in the borough.
"In 2002, Castle Goring was placed in category A of English Heritage's at risk register.
"The condition of the building is described as very bad.
"Over the past three years the council has made various attempts to gain access to the building to ascertain its condition and was successful in viewing the majority of the building in September 2001.
"However, on this occasion, two areas of the building were not accessible and despite numerous attempts to arrange a further visit, there has been no response from the occupier.
"It was clear from the visit in 2001 that there was cause for concern regarding the future of the building, which has been neglected for some time.
"Given the risk to a building of historical and architectural importance, the council must consider whether it is now appropriate for it to adopt a more formal approach to secure entry to the building and ensure its long term future."
She warned that the council may serve a repairs notice on the owners and begin compulsory purchase proceedings.
Twenty-four hours' written notice must be given before the council can enter Castle Goring, when an experienced architect would be present to assess the condition of the structure.
The initial cost would be about £6,000, but spending would escalate considerably if an order was pursued.
Castle Goring was built in 1797 and 1798 by Sir Bysshe Shelley.
The south frontage has a Roman-style facade, while the north frontage is Gothic in design.
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules hereComments are closed on this article