I had an interesting day on Monday after releasing a thread of eight tweets on the toppling of statues around the world. I had pondered hard whether to release the thread, suspecting it would raise the ire of the more intolerant of our activists. I was not disappointed.

The thread provoked furious debate on my account, which I engaged for a considerable time. There were some thoughtful engagements and critiques, but also the now common labelling, coupled with ideological and macho political posturing. What struck me most about many of these responses is that they did not appreciate or internalise that I was not proposing the retention of the statues of the more reprehensible historical figures; I was instead proposing their reimagining to honour the victims.

It made no difference to many of the critical responses though. In their view there was no difference between retaining the statue and reimagining it. Nuanced interpretation was lost in the almost visceral emotional responses.

The irony is that my position was not diametrically opposed to theirs. I held the view that the rage against statues of racist and colonialist figures was understandable and had to be addressed. But I also noted that the attacks on statues had gone beyond slave traders to politicians and other historical figures, and I raised the almost trite point that historical figures need to be understood and judged in their context.

I immediately qualified this by suggesting that some acts are so vile that they must transcend historical divides. On the basis of these first principles I suggested that three options are available with regard to statues and monuments: leave them as is, reimagine them to honour victims, or remove them. I recommended that the decision among these choices should be the result of public deliberation, not the random act of selected stakeholders or populist mobilisation.

My own preference among the options was for reimagining statues. This is because I am concerned about historical memory and the danger of humanity repeating past atrocities. Removing statues to museums does not resolve this given that most people will not go there. I proposed reimagining statues in honour of victims so as to consolidate a popular consciousness of the barbaric acts of these individuals and ensure that their ignominy remains forever etched in the minds of future generations.

At one level  the response to my argument is understandable given the hurt experienced by victims and their descendants as a result of slavery, colonialism and racism. But on another level one must be concerned at the intolerance of the responses; the message was you are either with us or against us. No dissent, however minor, was to be brooked within the community of those opposed to the statues. This almost enforced homogeneity just cannot be an enabling attitude within democracies.

However, there is an even bigger concern in this debate. I am struck by the silence or absence of prominent public leaders and intellectuals reflecting on the issue. The only people commenting on the statues are conservative politicians, the most prominent being Donald Trump and Boris Johnson. And in their typical reflex response the status quo must be retained and the statues left alone. The net effect is a polarised engagement between protesters intent on destroying statues and other symbols of oppression, and conservative politicians who want to keep them. There are of course myriad options public intellectuals could and should be exploring, yet are not.

Let me demonstrate this absence of principled engagement by highlighting my debate with Mmusi Maimane on twitter. Maimane waded into the debate sometime on Monday evening, arguing in particular against my view that statues and monuments should be reimagined and calling the entire argument “faulty”. He correctly held that these statues are “symbols of pain and trauma” but that they should be taken down and removed to museums. He also referred to the German case, where statues of Hitler were removed.

I responded by indicating that there were memorials to victims in Germany, but Maimane held that they were “a de novo creation”. I again responded that “reimagining a statue can fundamentally transform its meaning ... but this is something we can respectfully disagree on”. I called on him to respond to the other parts of my recommendation, namely that decisions on statues should flow out of public deliberations and that in most cases historical figures need to be understood in their historical context. But try as I might Maimane would not comment on these issues.

The important question is why is this the case? These are, after all, sensible recommendations in line with civil liberties and how decision-making should occur in democracies. Why would the former leader of a liberal political party, and the founder of a civic movement, find it difficult to comment on the need for public deliberation and participation, especially after voluntarily wading into the debate? The only answer that makes sense is that he may be concerned about a backlash to his answers that might compromise his political and civic plans.

If this is correct and Maimane was the only example of this, it would not be a systemic challenge. But I fear his concerns and reaction are typical of a wider one among liberal and progressive leaders and thinkers, which could compromise our democracies. Reimagining and redefining our society so all of us feel a part of it requires critical engagement and nuanced debate, in which liberal and public intellectuals must participate. I recognise that it can be difficult, with the labelling, intolerance and sometimes even racism that ensues, but this societal engagement on public expressions is too important to leave solely to angry activists and conservative politicians.

Building inclusive societies is a result of thoughtful and collective measured deliberation. It will not result from the sole actions of the loudest and most muscular. Otherwise we risk a conservative backlash and authoritarianism, which history has taught us is never in the interests of the poor, marginalised and excluded.

• Prof Habib, vice-chancellor and principal of Wits University, is director-elect of the School of Oriental and African Studies at the University of London.