City traders refusing to spend £60 to keep advertising boards outside their shops may face them being taken away.
Brighton and Hove council said it will confiscate any unlicensed A-boards cluttering the streets of Kemp Town.
Highway licensing officer Ian Denyer warned he was considering legal action against the most persistent offenders.
Traders have been divided by the charges, which were introduced last month in St James's Street, Manchester Street and George Street. Twenty agreed to pay while about ten decided to take their boards inside.
Mr Denyer has identified six shops whose displays are in danger of being seized or of prompting legal action.
Public notices were posted in St James's Street on Wednesday outlining the threats.
Traders reclaiming confiscated boards will have to pay the licence fee or sign an agreement not to display the boards again.
Mr Denyer said: "We don't want to end up taking people to court, but there will always be a few who are more intransigent."
Stuart Richardson had a board for seven years outside Richardson's hair salon in St James's Street.
But he has taken it inside in disgust at the charges and is urging fellow traders to do the same.
He said: "The £60 doesn't bother me so much as the principle. I pay enough in business rates already."
The Disabled Action Group and the St James's Street Action Group had raised concerns about the risk from boards to elderly and disabled people.
Similar charges are to be imposed in The Lanes and in Rottingdean High Street.
Sue Addis, of The Lanes Traders Association, welcomed the new arrangements in principle but doubted whether a charge was necessary.
Mrs Addis, owner of the Donatello restaurant, said: "Perhaps there should just be a fine for unlicensed boards.
"But I agree with the scheme. You can barely move for A-boards in The Lanes these days. There needs to be more control."
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules hereComments are closed on this article