Ads.txt - What it is and why it can’t protect you from invalid traffic and click fraud

Table of Contents

Ad fraud is everywhere. According to recent reports, $68 billion will be lost to digital ad fraud in 2022 alone.

 

While some companies are still not convinced that they are putting large amounts of money into the pockets of fraudsters, others are working on a solution to limit the fraud problem. One of these solutions was already presented by the IAB in 2017: the ads.txt file.

 

In this article we will look at what ads.txt is, how it works and why it can’t solve the biggest problems of digital ad fraud.

What is ads.txt?

Launched in May 2017 by the IAB Tech Lab, the Authorized Digital Sellers project aims to tackle various types of ad fraud, most notably domain spoofing and illegal inventory arbitrage.

 

Ads.txt is a simple text file that contains information about which companies are allowed to sell digital inventory on a particular domain. As it can be created and modified only by the webmaster of a domain, the information of the file is considered valid and authentic.

 

The file is also intended to increase transparency in the increasingly non-transparent ecosystem of programmatic advertising. This is done by giving publishers a flexible way to publicly declare the companies they allow to sell their digital inventory on their behalf, which makes the inventory supply chain transparent and protects buyers from fraudsters selling inventory that does not belong to them.

What problem does ads.txt solve?

Ads.txt tackles mainly two things:

 

  1. Domain spoofing, which is the process of faking URLs in the ad tag either by using a malware that has been installed on a victim’s device or using malicious on-site scripts. The result: Advertising exchanges and other programmatic platforms are fooled into thinking that a user has called up a legitimate URL, while the ad is displayed on another, untrustworthy website and most often only seen and clicked by bots.
  2. Inventory arbitrage, which is the process of buying impressions from a publisher, repackaging and reselling them at a higher price by a third party.

 

Fraudsters set up fake websites that either have no content at all or whose content has been scraped from large, trusted publishers like the New York Times, and send fake bot traffic to them to increase the ad views and even click on the ads. They then change the URL in the ad tag to make it look like the ad delivery actually took place on the New York Times domain. The fraudsters are paid by the advertising platform to serve ads on their fake website, and advertisers are made to believe that their ad was displayed on the New York Times website.

 

To be clear, ads.txt does not solve bot-related fraud. It is primarily aimed at reducing domain spoofing and fake ad requests.

How does ads.txt work?

Publishers create a file called “ads.txt” and host it under their root domain. The file itself contains information about all the authorized digital sellers like supply-side platforms (SSPs), ad exchanges and ad networks, that the publishers work with and which may manage ad inventory.

 

To clarify what this means exactly, let’s take a look at the ads.txt from nytimes.com:

Screenshot of the ads.txt file from New York Times

Each line represents a partner who is authorized to manage the ad inventory on the domain nytimes.com and consists of up to 4 fields. As an example, line 20 says the following:

 

google.com, pub-1793726897772453, DIRECT, f08c47fec0942fa0

 

google.com → Domain name of the advertising system (required)
This field indicates the domain of the platform the publisher uses to sell their inventory. We can see that the New York Times only uses a handful of partners, including Google, Amazon, Yahoo and AppNexus.

 

pub-1793726897772453 → Publisher Account ID (required)
The Publisher Account ID is the publisher’s account ID for the particular partner – in this case Google – and is used to validate the authenticity of the inventory during RTB bids.

 

DIRECT / RESELLER → Relationship Type (required)
This field can have either the values “DIRECT” or “RESELLER”. It indicates whether a publisher is working directly with the vendor to sell its ad inventory, or has appointed another company (such as an ad network or digital advertising agency) to manage the ad inventory on its behalf.

 

f08c47fec0942fa0 → TAG ID (optional)
This optional field lists the TAG ID that identifies the partner within the Trustworthy Accountability Group (TAG) certification authority. The TAG ID is a globally unique and case-insensitive 16-character hexadecimal string that you can request from your partner.

 

# comment → comment (optional)
Some publishers also include a hashtag followed by a comment. The information after the hashtag is not used by any crawler and is intended only for the publisher and his webmaster to keep track of when the file becomes too large. To see an example, you can have a look at businessinsider.com/ads.txt. They have added the ad formats that partners manage as comments on some lines.

 

After a publisher publishes the list of partners that are allowed to sell ad inventory on its domain, demand-side platforms (DSPs) crawl the domain and also the ads.txt file. Each time an OpenRTB bid request is made to the DSP by an exchange, the DSP checks the domain and Publisher ID against the information in ads.txt. If the information matches, the bid is placed. If the information does not match, no bid is made.

Overview of how ads.txt works

Image source

 

For further information on the technical specification or some examples and use cases, see the official ads.txt specification from IAB TechLab or their overview of all ads.txt related resources.

 

How to create an ads.txt file?

Creating the ads.txt file is simple and straightforward. All you need is a text editor that can create a .txt file and information from your partners and networks.

Step 1: Gather all the required information

Before you create the actual file, you need to collect information from all your vendors, ad exchanges, and ad networks. This includes the domain name, your publisher ID for that partner, and whether it is a direct or reseller partnership with that partner.

 

If you want to be very specific, you can also request the TAG ID from each partner. To verify your partners, you can use the TAG Registry to look up the TAG IDs they provide. The TAG Registry can confirm in real time if a provided TAG ID is associated with a company name or if a company is included in the TAG Registry.

Step 2: Create a .txt file and name it ads.txt

After you have gathered all the information, simply open the text editor of your choice and paste the information in the desired format. Remember: each field is separated by a comma and each partner must be on a separate line. Make sure to save the file as a .txt document and name it “ads.txt”.

Example of ads.txt entry with explanation of the syntax of each field

Some tips for free text editors include:

Step 3: Verify your ads.txt

Before uploading your file to your web server, you should first verify the content, readability, and syntax of the file. You can do this manually, but it is much easier if you use one of the following tools:

Both tools will check your file for several potential errors like spelling mistakes, invalid domain names or wrong canonical domain names.

Step 4: Upload the ads.txt file to your root domain

The last step includes uploading the file to your root domain. Please note: The ads.txt works only on the root domain, not on any subdomain or folder.

 

You can verify that the file is in the correct location by simply appending /ads.txt to your domain and the correct information from the file will appear. So for example: yourdomain.com/ads.txt

Benefits of ads.txt for publishers and advertisers

The biggest advantage for publishers who include an ads.txt on their domain is that illegal and fraudulent inventory purchases and fake ad requests are mostly avoided.

 

But advertisers also benefit from ads.txt, as they can verify merchants and prevent wasting budget on fake ad requests.

 

The entire ad ecosystem also benefits from increased transparency. Ads.txt makes publicly available the information about which companies are authorized to sell ad inventory on which domains.

Publisher adoption of ads.txt

Adoption of ads.txt was very slow when it was launched in 2017. It wasn’t until Google announced that it would support the new standard and that DoubleClick Ad Exchange and AdSense would filter out unauthorized inventory from their auctions that most publishers moved to creating an ads.txt file for their domain.

 

To date, just over 45% of the 1,000 largest domains have adopted the standard.

Ads.txt for mobile apps

Overview of how app-ads.txt works

Image source

 

While the initial release of ads.txt in 2017 was aimed at ads on websites, mobile ad fraud continued to grow. As a result, the IAB TechLab published app-ads.txt in 2019, an update to the existing specification that can also be used to fight ad fraud in mobile apps.

 

The new standard was also quickly picked up by Google and AdMob and now has an even higher adoption rate than ads.txt for websites. 68% of the top 1,000 Google Play apps have an app-ads.txt, but only 42% of the top 1,000 apps in Apple’s App Store.

 

The app-ads.txt deserves its own detailed article on our website. Until then, you can find the latest specification and instructions for implementation in apps on the official IAB website.

Vulnerabilities of ads.txt

Now that we have looked at the intentions and benefits of the ads.txt file, we need to take a look at potential vulnerabilities. First of all: There are some gaps in the system that could be exploited by fraudsters and the biggest error factor is, as so often, human error.

Lack of attention and maintenance from publishers

The ads.txt file is not a solution that you simply create and forget. Publishers need to actively keep track of listings, maintain it and also review it periodically.

 

When more and more parties are authorized to sell ad inventory on behalf of the publisher, the ads.txt file gets bigger and bigger. This not only makes it very difficult to read and maintain, but also no longer accurately reflects who is authorized to sell the actual inventory.

 

Every time a new partner is signed, they have to add that particular information to the file. The other way is the same: every time a partnership ends, the information must be removed from the file. Otherwise, publishers run the risk that long-expired partnerships still have access to their advertising inventory – at least according to the ads.txt entry. This situation can be very quickly exploited by malicious partners, which brings us back to the issue of faking ad request.

 

Take the New York Times example above. It currently has 21 authorized partners (as of June 2021) who are allowed to manage the advertising inventory. A direct relationship has been established with all partners, so no one can resell inventory without further ado.

 

If you look at ESPN.com, on the other hand, you are almost blown away. A whopping 460 partners are listed there, and more than half of them are resellers.

404Bot – Keep an eye on your resellers

At the beginning of 2019, a new scam made the rounds: 404Bot. Fraudsters leveraged the functionality of ads.txt by opening accounts with ad networks that were listed as approved resellers with major publishers.

 

It is estimated that due to the aforementioned lack of overview within a large ads.txt file, the scammers were able to steal between $15 and $80 million over a period of several months.

 

The scheme was pretty simple:

  1. Fraudsters created accounts with advertising networks listed as resellers with major publications.
  2. Then they sent bot traffic to fake but real looking URLs on the publishers’ website that didn’t exist and returned a 404 status code – hence the name 404Bot. HTTP status code 404 indicates that the browser was able to communicate with a particular server, but the server could not find the requested file.
  3. However, there was ad code embedded on these fake URLs that was executed in the usual way.
  4. The scammers were then paid by the ad network for successfully selling ad inventory from premium publishers.

The big advantage of this approach was also that the URLs appear legitimate in all reports. If someone wanted to visit the URLs, they were only shown the notice that the page was not found. However, it could also have been that the article had been deleted again by the publisher in the meantime. The perfect cover for the fraud.

 

As a publisher, it is essential that you keep an eye on the content of your ads.txt file and avoid resellers as much as possible. All domains that were affected by the 404Bot had one thing in common: Their ads.txt files were huge and contained a lot of reselling partners. The more resellers are listed in it, the higher the probability that you are affected by fraud.

Social engineering to be included in ads.txt

With the launch of ads.txt, publishers received more and more requests from fraudsters pretending to be digital marketing agencies who wanted to secure an official place as a reseller in the list. The scam looked something like this:

 

Scammers contacted major publishers to be included in their ads.txt. In doing so, they pretended to have acted as resellers in the past and to already have sold publishers’ ad inventory. One of these providers made it to over 60 lists in a short period of time because publishers did not verify the requests properly, but gave out many free passes, especially at the beginning, out of fear of losing revenue.

 

You can probably guess how the story ends. Once the scammers got on the list, they were able to sell ad space on premium publishers for a high price and use bot traffic to drive up profits.

 

In this case, it was not directly a vulnerability in ads.txt, but social engineering and thus humans as the weakest link.

Ads.txt can’t protect you from invalid traffic and click fraud

In this article we have covered everything worth knowing about ads.txt: What it is, how it works and also what the vulnerabilities are. While the current state is a step in the right direction to reduce ad fraud – in this case, especially domain spoofing – it’s important to keep one thing in mind: Ads.txt does not protect your ad spend from invalid traffic and click fraud!

 

Even if your ad is displayed on a website in a legitimate way, it does not guarantee that real people will see it or even click on it. Ads.txt makes sure that your ad is only displayed by authorized partners, not that real people interact with it.

 

As an advertiser, you will therefore continue to see invalid bot traffic and fake clicks on your ads, which will drain your advertising budget, even if your ads are served directly to publisher websites.

 

Very worrying in this context is a report at the end of 2019, which came to the conclusion that websites that use ads.txt have about 10% more invalid traffic (IVT) than websites that have not implemented ads.txt.

 

To solve this problem, you need ad fraud detection software that can detect fake bots and prevent them from clicking your ads and wasting your advertising budget.

 

Sign up for a free 14-day trial with fraud0 today and see the results for yourself – no strings attached and no credit card required.

More Articles
How much of your marketing is wasted on fake traffic?
1%, 4%, 36%?
Full access. No credit card required
_Newsletter
Stay on top of things and subscribe to our newsletter.

Try fraud0 for 7 days

No credit card required. Full access
or access quickly