LONDON (AP) — This was the week British Prime Minister Boris Johnson hoped to get a grip on his government after weeks of scandal. By Friday, he was struggling to hang on after a scathing report on lockdown-breaching parties and the departure of several top aides.

Johnson was rocked Thursday by the resignation of his policy chief, Munira Mirza, a trusted adviser who worked with him for more than a decade. Mirza stood by the prime minister amid “partygate” revelations that Johnson and his staff broke the rules they had imposed on the country. But she said Johnson’s “scurrilous accusation” this week that an opposition leader had failed to stop a notorious pedophile was the final straw.

“This was not the normal cut-and-thrust of politics; it was an inappropriate and partisan reference to a horrendous case of child sex abuse,” Mirza wrote in a resignation letter, which was published by The Spectator magazine.

After Mirza quit, Johnson’s office announced the departure of three more top staffers: chief of staff Dan Rosenfield, communications director Jack Doyle and principal private secretary Martin Reynolds. Elena Narozanski, who worked in Murza’s policy unit, resigned Friday.

Johnson tried to reassure staff that he remained in control by quoting the wise monkey Rafiki from “The Lion King,” telling a meeting: ”Change is good” even if it is difficult.

Conservative lawmakers loyal to Johnson depicted the departures as part of a planned overhaul to restore order to his 10 Downing Street office.

“The prime minister was absolutely clear on Monday that there would be changes at the top of No. 10 and that is what he has delivered,” Energy Minister Greg Hands said. “This is the prime minister taking charge.”

Others weren’t so sure. The prime minister’s grip on power has been shaken by public anger at revelations that his staff held “bring your own booze” office parties, birthday celebrations and “wine time Fridays” at times in 2020 and 2021 while millions in Britain were barred from meeting with friends and family because of COVID-19 restrictions.

A total of 16 parties have been investigated by a senior civil servant, Sue Gray, with a dozen of them also under investigation by the Metropolitan Police.

On Monday, Gray released an interim report looking at the four parties police are not probing. She found that “failures of leadership and judgment” allowed events to occur that “should not have been allowed to take place” and described a Downing Street operation marked by excessive drinking and dysfunctional dynamics.

Johnson apologized and pledged to fix the problems in his office, though he didn’t admit personal wrongdoing.

Rosenfield, Doyle and especially Reynolds — who sent 100 government staff an invitation to a BYOB garden party in May 2020 — were always likely to be ousted as part of Johnson’s post-“partygate” shakeup.

But Mirza’s departure was a major blow. In her resignation letter, she said Johnson had not heeded her advice to apologize for accusing Labour Party leader Keir Starmer in the House of Commons on Monday of “failing to prosecute Jimmy Savile” when Starmer was the U.K.’s director of public prosecutions. Savile was a long-time presenter of youth television shows who was exposed after his death in 2011 as a sexual predator who had abused hundreds of children.

Starmer said the accusation was “a ridiculous slur peddled by right-wing trolls.” A 2013 report found that Starmer hadn’t been involved in decisions about whether Savile should be prosecuted.

Some Conservatives also recoiled at the use of Savile in a political attack. In her resignation letter, Mirza said Johnson had let himself down “by making a scurrilous accusation against the leader of the opposition.”

The Downing Street exodus is sending further shockwaves through Conservative lawmakers as they mull whether to seek a no-confidence vote in the leader who won them a big parliamentary majority just over two years ago. Under party rules, such a vote is triggered if 15% of party lawmakers — currently 54 people — write letters calling for one. If Johnson lost such a vote, he would be replaced as party leader and prime minister.

Only about a dozen Conservative legislators have publicly called for Johnson to quit, though the number who have written letters may be higher.

Aaron Bell, a lawmaker who asked in Parliament on Monday whether Johnson thought he was a fool for following social distancing rules at his grandmother’s funeral in 2020, added his name on Friday to those calling for a no-confidence vote. He wrote on social media that “the breach of trust that the events in No. 10 Downing Street represent, and the manner in which they have been handled, makes his position untenable.”

Many others are biding their time, waiting to see whether police censure the prime minister and what Gray will say in her final report, due once the criminal investigation is over.

Huw Merriman, a moderate Conservative lawmaker, said the prime minister had to shape up or ship out.

“My constituents are upset,” he told the BBC. “I feel like we’ve lost face and public trust with them. We’ve got to gain that back.”