"With the moral authority of the presidency, you have to call that stuff out", Scaramucci said.
Scaramucci, who held the position as Trump's top spokesman for 11 days, gave ABC his first interview since he was sacked at the end of July by Gen. John Kelly, the new White House chief of staff.
A White House spokesperson clarified on Sunday the recent statement by US President Donald Trump regarding the violence in Charlottesville, Virginia, and said that it included condemnation of white supremacists, media reported. "I think he would have needed to have been much harsher".
Scaramucci's interview Sunday came two weeks after his firing.
The former White House communications director's remarks refer to a controversial statement Trump gave Saturday condemning violence "on many sides" - rather than violence from white supremacists and neo-Nazis gathered at the rally.
He also called on Trump to distance himself from top adviser Steve Bannon, the divisive former head of Breitbart News who Scaramucci argued was pulling Trump too far to the right and was "not serving the president's interest".
They included his calling then-Chief of Staff Reince Priebus a "fucking paranoid schizophrenic" and saying of White House strategist Steve Bannon: "I'm not Steve Bannon".
His firing capped a tumultuous period within the Trump White House.
The ex-aide was speaking on the ABC News show This Week when he was asked by host George Stephanopoulos about the statement the United States president gave yesterday in which he condemned the violence but not neo-nazis specifically.
"He has to move away from that sort of "Bannon-bart" nonsense", Scaramucci said, apparently combining Bannon's name with Breitbart, the website that Bannon ran before joining Trump's campaign past year.
Trump's response was criticized by politicians on both sides of the aisle for failing to name white supremacist or alt-right groups involved in the protests.
Scaramucci is scheduled also to appear Monday night on CBS' "The Late Show with Stephen Colbert".