Disabled people will be charged up to £15 a day for care.
Brighton and Hove City Council will introduce the charges to save £40,000 a year.
More than 150 people will be affected by the new means-tested Fairer Charging Policy.
Those who work with disabled people were critical of the fees, which they say will cause further difficulty for the most vulnerable members of society.
Parent Sonia Howes, from Seven Dials, Brighton, sends her 23-year-old son Steven to the Avondale Day Centre in Hove to allow her to hold down a part-time job.
She says her income will barely cover the cost of care and she will be better off if she quits work and claims benefits.
She said: "The centre is a total lifeline for us. I do a part-time job and this means I'm in real trouble because I'm trying to support Steven on my own.
"The day centre is vital for me because it means I can work - it keeps me sane. What do they want people like me to do? Quit work and go on benefits so I can afford the charges?"
Andy Bradley, from Brighton and Hove Speak Out, an advocacy project which helps give people with learning disabilities a voice, said he hoped the charges would be reflected in an improved service.
He said: "The general feeling is most people are unhappy and it's an issue of quality as much as anything else.
"At the moment day services are little more than a safe space for people to go. I think people would be more prepared to pay if, for example, the centres were helping people move into employment or voluntary work."
At a meeting of the adult social care and health committee last night, councillors voted five to four in favour of introducing the charges.
Green Party councillor Simon Williams voted against the plans.
He said: "It is quite clear from consultation with families and carers they are strongly opposed to charging.
"Coming so soon after the council's recent budget which inflicted cuts on learning disability services, the timing of this is very poor.
"Whatever the arguments for consistent charging this will hit some of the most frail and vulnerable people in the city. It is heartless."
However, Councillor Gerry Kielty, committee chairman, said the charges would bring people with learning disabilities in line with other day care users.
The scheme will take effect from April 30 and is subject to a six-month review.
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules hereComments are closed on this article