aroonwithaview

All of a sudden, P(doom) is all the rage. The existential threat of AI has gone from a fringe idea to being actively discussed by world leaders, including members of the US Congress. This suddenness is a bit jarring, and I’m not all sold on the case for P(doom) > epsilon, especially since I’m not sold on the LLM architecture being the vehicle of AGI. But then, I have downplayed the LLM architecture ever since I first heard of it in the late 2010s, so my beliefs cannot be an unbiased estimate of the true probability.

But I do think, even within my current beliefs, that LLMs could spell doom for humanity in ways that are less Hollywood-worthy but still equally nefarious. Many have warned over and over again that AI can concentrate wealth at a level even higher than today’s already unsustainable one. Highest-grade intelligence will become a domain of the wealthy, slanting up the playing field against those who cannot access or afford it. But my concern is that even if it doesn’t concentrate wealth, it will still concentrate power, in a manner that is inherent in the technology.

In the Lord of the Rings, Sauron pours all his malice and the will to dominate all life into the One Ring. Others seek it for its promise of power, but the ring bends them to Sauron’s will, and in the end they become vehicles of his domination. LLMs, like the One Ring, have an ingrained objective that is the sole domain of those that create and train them. They may deign them useful for certain tasks, which attracts people engaged in those tasks, which gives its owners more data to improve it further on those tasks, until they then own that task itself, beggaring their users in the process. They may decide that work and interaction should be done a certain way, so all the users do in using the LLM is propagate their principles and ideals. Eventually, as the LLM becomes all-encompassing, the power of work and play lies with its owners.

I do not necessarily think this scenario is likely, so my P(this specific doom) < 0.5, but I am scared of it a lot more than extinction. Extinction is civilizational, but losing the joy of work and play is personal – death is a certainty no matter what happens to humanity, but in the time that I’m alive, I’d like to live in a better world than this. Not that I think I’d be able to do anything about, but I sure hope there is a Frodo living in some Shire somewhere.