TikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAI
ByteDance looks like it’s eager to make up for lost time when it comes to scraping the web for data needed to train its generative AI models.
The China-based parent company of video app TikTok released its own web crawler or scraper bot, dubbed Bytespider, sometime in April, according to research from Kasada, a company that specializes in bot management for companies with online data. The existence of the bot was also confirmed by Dark Visitors, which monitors scraper bots.
ByteDance’s bot has quickly become one of the most, if not the single most, aggressive scrapers on the internet, the research shows. It’s scraping data at a rate that’s many multiples of other major companies, such as (Google, Meta, Amazon, OpenAI, and Anthropic, which use their own scraper bots to help create and improve their large language or multimodal models, known as LLMs or LMMs.
Sam Crowther, the CEO of Kasada, said since Bytespider showed up, it’s been scraping data at about 25 times the rate of GPTbot, which scrapes data for OpenAI’s ChatGPT platform and underlying models, for instance. Bytespider has been scraping at 3,000 times the rate of ClaudeBot, from Anthropic, which operates the Claude platform.
As the months have gone by, Bytespider has become even more aggressive, according to Kasada. Data shows huge spikes in scraping activity from Bytespider over each of the last six weeks.
Representatives of TikTok and ByteDance did not respond to emails seeking comment.
ByteDance’s aggressive scraping comes despite the possibility of TikTok being banned in the U.S in the coming months. President Joe Biden has signed legislation that requires ByteDance to sell TikTok, due to national security concerns, or shut it down.
The Bytespider bot, much like those of OpenAI and Anthropic, does not respect robots.txt, the research shows. Robots.txt is a line of code that publishers can put into a website that, while not legally binding in any way, is supposed to signal to scraper bots that they cannot take that website’s data.
Web scraping goes back decades, mainly by search engines to gather links to web pages. But the rise of generative AI tools has added a new dimension and made the practice a prime source of lawsuits and controversy. People and organizations whose work has been scraped argue their copyright is being infringed in the process. All of the models that underly generative AI tools were trained on massive amounts of online data, effectively everything available on the web, particularly written information. Tech companies use scraper bots to essentially copy it all for all for free and put it into their datasets.
“It’s like they’re trying desperately to catch up,” Crowther said of the aggressive scraping being done by Bytespider. Just last year, ByteDance was reportedly so far behind in the generative AI race that it was using OpenAI to help build ByteDance’s own LLM, which is against OpenAI’s terms of service. Earlier this year, ByteDance released a chat-based LLM called Duabo, but work on that model would have been completed prior to the accumulation of more recent training data scraped by Bytespider.
It’s “clear” that ByteDance is at work on a new LLM, according to one person familiar with the company. As for what ByteDance plans to do with a new LLM, a person familiar with the company’s ambitions said one goal has to do with the search function for TikTok.
Last week, TikTok released an update to its current search function focused on keywords for ads, basically allowing advertisers to search in real time for words that are trending on TikTok. It allows marketers to build an ad with relevant keywords that would ostensibly help the ad show up on the screens of more users.
A new AI model with data on more recent internet trends and topics could expand and improve TikTok’s search environment further, according to the person familiar with the company’s ambitions.
“Given the audience and the amount of use, TikTok with a search environment that is a completely biddable space with keywords and topics, that would be very interesting to a lot of people spending a ton of money with Google right now,” the person said.
Are you a TikTok or ByteDance employee or someone with insight or a tip to share? Contact Kali Hays securely through Signal at +1-949-280-0267 or at kali.hays@fortune.com.
Data Sheet: Stay on top of the business of tech with thoughtful analysis on the industry's biggest names.
Sign up here.
