Reddit signed a deal with Google in February allowing the search giant to use its content for an estimated $60 million a year. This agreement between the two companies gives Google access to Reddit’s archives of vast data readily available on millions of different topics.
Now, the news is coming in that Reddit has moved to block search engines not named Google from crawling its site. They did it through an update to its robots.txt file. This blocks all crawlers from accessing Reddit’s content.
This means that Microsoft’s Bing has stopped crawling the website from July 1. Now, the website refuses access to all non-paying search engines.
After the deal, Google has been referring a heap of traffic to Reddit’s pages. This deal has now opened a new door for Reddit to set a new precedent on data access.
However, Reddit is saying that this is not related to the deal with Google.
“This is not at all related to our recent partnership with Google. We have been in discussions with multiple search engines. We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.”
X and Reddit are two of the vastly popular platforms that have years of user-generated data. Many AI platforms are scraping their data to train their AI systems on human-created data.
Both of these platforms have increased their price for API access. They want to ensure that AI platforms are not profiting off of their insights. This gives them more control over which AI projects are allowed to use for their initiatives.
In the future, it’ll be interesting to see how this trend shapes up.
