This is exactly how I have described the limitations of black box models to others.

A good scientific model should do two things:

1) Provide accurate predictions of outcomes given certain inputs, and
2) Enable greater understanding of how a system works.

Simpler machine learning models, like logistic regression or decision trees, can sometimes do both, at least for simpler phenomenon. The models are explainable and their decisions are interpretable. For those reasons among others, applied machine learning researchers still use these simpler approaches wherever they can be made to work.

But in our haste to increase accuracy for more complex phenomenon, we've created models that merely provide semi-accurate predictions at the expense of explainability and interpretability. Like the ptolemaic model of the solar system, these models mostly work well in predicting outcomes within the narrow areas in which they've been trained. But they do absolutely nothing to enable understanding of the underlying phenomenon. Or worse, they mislead us into fundamentally wrong understandings. And because their training is overfit onto the limits of their training data, their accuracy falls apart unpredictably when used for tasks outside the distribution of their training. Computational linguists and other experts that might celebrate these models instead lament the benighted ignorance left in their wake.

Or how it was more eloquently stated in the great philosophical film Billy Madison:

"Mr. Madison, what you've just said is one of the most insanely idiotic things I have ever heard. At no point in your rambling, incoherent response were you even close to anything that could be considered a rational thought. Everyone in this room is now dumber for having listened to it. I award you no points, and may God have mercy on your soul."

The ptolemaic mental model of the solar system was used as a pretext for centuries of religious warfare.

Entrenched interests fought wars to keep it according to James Burke's 1st episode "The Day the Universe Changed".

Why? Because it fed a narrative of an unchanging "natural order".

Will today's mental model of AI feed an equally self-serving set of narratives?

The tell-tales are lots of special casing in the code (the epicycles there) and the response "it works and that is all that counts" when you try to discuss what you see in the code.

Ah, and strange variable names.