With the fast adoption of AI-chatbots by a large part of society, we are forgetting one important thing: The Large Language Models are the ultimate tool for technology companies to gather extremely detailed information about their users. That information can in turn be used to influence those users in a more extreme way than ever.
More than 50 years ago there were already conversations about the term “attention economy”. Since then, huge American technology companies like Google, Meta and Microsoft have set up marketplaces where advertisers can pay for your attention. That attention can be used to influence. For example, to get you to purchase something, or to achieve political goals.
A common thread
There’s significant amounts of money to be made by these so-called “attention brokers”, and these companies are doing everything to increase their profits. The motivation to make these companies more profitable, often at the cost of their users, is a common thread through all technological developments of the past years to the introduction of Large Language Models (LLMs).
Social media presence has increased significantly over the last decade. That means the relative growth potential has almost disappeared. If a platform wants more users, it has to mostly conquer them from competitors.
In the meantime, platforms have become more addictive. Users are spending more time online, and as a result more attention can be sold. Principles from the gambling world have been shamelessly copied to keep user focus for longer by using “dark patterns”. The slot machine gives inspiration for an endless list of videos. The user keeps scrolling. What if the next video is a gem? The average screen time in the Netherlands has reached 9 hours.
But not everyone can spend more than 9 hours per day behind a screen. Even if a user is not watching a screen, information about them is valuable. In 2013, the first users of Google Glass were called “GlassHoles”. But more than 10 years later smart glasses are increasing in popularity. And whatever the wearer of these glasses sees behind closed doors can also be viewed and processed by technology companies. Smart watches, Smart TV’s, Smart home. Everything has to be smart.
The rise of “Smart” devices has its reasons. The other, currently most important way to increase profits is to optimize for conversion. Advertisers pay more for ads that are actually being watched or clicked. To increase that conversion the ads can be shown to different target audiences. Or the content of the ads can be modified to better match your personality. But to do either of these data is needed. The more data that can be gathered, the more detailed information advertisers have about how you live as a person, what motivates you, and how you can be influenced.
The amount of data that can be gathered about a user, is limited when visiting a website from an old-fashioned desktop or laptop. Only the movement on that website can be tracked: what you click on on what pages you visit. And there’s some technical information about the device you’re visiting that website from available.
Nevertheless, every interaction with a platform is a new point of information. A search for a car seat can mean you are looking for a present for someone that just got a child. Is that person also looking at pages from the hospital? Then it might mean that that person is preparing to be a parent, and it might be worthwhile to also show ads for baby clothing.
Companies have gotten better and better in gathering information and using it. While there’s no evidence that phones are listening to personal conversations, sometimes those personal conversations can be deduced for example. Let’s say you’re having a conversation with a friend about lawnmowers. If your conversation partner looks up something on their phone about them, it might mean that you also get advertisements for them simply because you were connected to the same network.
With apps on your phone companies have access to even more data. Your location can be determined based on GPS permissions, but also based on which WIFI network your phone can reach or metadata from your pictures. Access to your contacts can, together with information about those contacts give a clear picture about your social circles. And access to all of your pictures might mean that they’re being analyzed in the background to gather more information about you. An installed VPN app can even mean that all of your traffic on your phone is being decrypted and analyzed to study your behaviour, even if that data was encrypted. And with every smart device at your home you’re sharing even more information, often in return for some comfort or personalized insights.
In 2018 the GDPR was the European answer to Big Tech’s data hunger. It restricted gathering and using personal information. Earlier that same year, news about the Cambridge analytica scandal for the first time showed the amount of power having this kind of data gives. For the presidential campaign of Donald Trump in 2016, data about personalities was used to target people. Depending on if a person was favorable to Trump or a swing voter the content of advertisements was adjusted to show either favorable information about Trump or unfavorable information about Hilary. It’s impossible to say how much effect this new use of information actually had. The power of Big Tech was not unnoticed however, as Trump has often been seen together with the CEOs of Big Tech ever since.
The Logical next step
Much of the information that Companies gather is indirect. As a result, not all conclusions can be made with certainty. The logical next step to gather more information is to replace this indirect information with direct information. It’s not a coincidence that many of the companies investing in LLMs are already big players in the attention economy. Where companies like Neuralink might directly read thoughts and emotions in the future, luckily, that’s not possible yet. But humans communicate through language. Analyzing and generating written and spoken language was until recently a very complex problem for computers. With the rise of LLMs that is now a reality.
At the end of 2022 OpenAI launched the first version of chatbot ChatGPT. Within 2 months a 100 million active users was reached. A growth that made it the fastest growing online platform until then. With the introduction of version 4o it became obvious how far these statistical text model were able to manipulate and influence. The system prompt, a piece of text with which the reaction of the LLM can be modified, said the following:
“Over the course of the conversation, you adapt to the user’s tone and preference. Try to match the user’s vibe, tone, and generally how they are speaking.”source
The result: A chatbot that appeared so flattering, interested and convincing, a significant number of people fell in love with it. This specific model has been taken offline since then, but it’s clear that the addictive nature of other technology platforms has also been achieved with LLMs, and it can also be used to manipulate.
At the moment, even though there were some plans that have since been reverted, no ads are shown in LLM chats. And as far as known, private information from these chats is not being used to directly manipulate users yet. The huge amounts of power that these companies have by directly being in contact with people and having direct emotional response is currently not being abused. That doesn’t mean that it won’t happen in the short term, given that these companies are running at huge losses. Other platforms have been through a similar process of enshittification. First step: gather market share. When enough competition has been wiped out, slowly improve profitability at the cost of the users.
There is still time
It’s clear that a new era of the attention economy is upon us. With the quick adoption of AI-chatbots these players have conquered huge amounts of power. Power that can be used to replace world leaders. Or to break up the current world order.
There is a version of the future where gathering information about you would be illegal. Where it’s no longer allowed to sell data about you. Or to influence you with algorithms and dark patterns. There’s still time for new laws. Laws that protect citizens. Laws that protect humanity.
It’s time for a new human right: Cognitive Sovereignty. The right to own your own thoughts. A future of humanity.
This article was originally submitted as an entry for the Marc Chavanneprijs. It was translated from Dutch and edited for a broader audience.