We Tested 1,000 Labelled IP Addresses. Here Are the Five We Got Wrong
Our homepage says 995 out of 1,000. This is everything behind that number: where the test set came from, what counted as right, the five addresses we got wrong, and what happened when we ran it all again three weeks later.
Until the end of October our homepage carried a different number: 14 out of 16. It came from a comparison we ran in May on sixteen addresses we had picked ourselves, against two paid services. A reviewer who looked the site over this autumn said what we should have said to ourselves, which is that sixteen hand-picked addresses prove nothing about anything.
So we built a set of a thousand. The rule we set was that every label had to come from somewhere other than our own data, because a test where you grade your own guesses with your own guesses will always pass. Then we ran the live IP score against all thousand on 28 October and wrote down what came back. The result was 995 correct. This post is the long version, and it includes several things that make the number less impressive than it looks on a homepage.
One dot per address. Five of them are red.
- Tor exit nodes 271
- Cloud servers 250
- Home connections 247
- Website front-ends 201
- Resolvers and root servers 31
- Miss 5
Each dot is one address, grouped by what it really is. Hover or tap any dot to see what it was and how it scored.
Show the numbers
| Category | Tested | Correct | Median score | Range | Low | Medium | High | Critical |
|---|---|---|---|---|---|---|---|---|
| Tor exit nodes | 271 | 271 | 98 | 75–100 | 0 | 0 | 34 | 237 |
| Cloud servers | 250 | 250 | 37 | 15–80 | 1 | 248 | 0 | 1 |
| Home connections | 247 | 247 | 0 | 0–13 | 247 | 0 | 0 | 0 |
| Website front-ends | 201 | 197 | 35 | 0–84 | 79 | 120 | 1 | 1 |
| Resolvers and root servers | 31 | 30 | 0 | 0–25 | 30 | 1 | 0 | 0 |
Where the labels came from
An IP reputation service makes claims about what an address is: a Tor exit, a server in a rack, somebody’s home router. To test those claims you need addresses where the answer is already known, and known for a reason that has nothing to do with the service being tested. That turned out to be the whole job. Writing the script that calls the API took ten minutes.
We ended up with five sources, each of which is the best authority we could find for one kind of address. The Tor Project publishes its exits. Cloud providers publish their customer ranges. A website’s address is whatever DNS says it is. Home connections were the awkward one, because nobody publishes a list of living rooms. We took random addresses inside the networks of large consumer ISPs and kept the ones whose reverse DNS name looked like a customer line, the pool-, dyn and cable sort of names ISPs give their subscribers.
What each address had to be called for us to count it right
Tor exit nodes 271
- Label comes from
- The Tor Project’s own published list of exits, sampled at random the same morning.
- Correct when
- We flag it as a Tor exit.
Cloud servers 250
- Label comes from
- The address ranges five large cloud providers publish for their customers. Fifty from each: a random range, then a random address inside it.
- Correct when
- We class it as hosting: by network, by range, or by a server-style hostname.
Home connections 247
- Label comes from
- Random addresses inside the networks of 26 large consumer ISPs on four continents, kept only if reverse DNS looked like a customer line. No more than 14 per ISP.
- Correct when
- We class it as an ISP or mobile line and raise no Tor, VPN, proxy or hosting flag.
Website front-ends 201
- Label comes from
- The addresses that 131 well-known sites resolve to, up to two each.
- Correct when
- We see infrastructure, and nothing abusive: no threat listing, no crawler flag, not in the high or critical band.
Resolvers and root servers 31
- Label comes from
- Public DNS resolvers and the thirteen root servers, typed in by hand.
- Correct when
- Low risk.
The counts are untidy, and the reason is not flattering. We aimed for 250 in each of four groups. The home-line filter left us three short, and the last two groups together came to 232. The script then topped the set up to exactly 1,000 with 21 extra Tor exits, which is why there are 271 of them. Tor exits are the easiest thing in this field to detect. Padding a test with the easy category is not something we would have chosen on purpose, and you should weigh the headline number with that in mind.
Two of the marking rules are also on the generous side. A cloud address passes if any one of three signals says “hosting”. One address in a published cloud range sits inside a carrier network that we class as an ISP; we scored it 15 and attached a “residential ISP” note to it, and it still passed because the range list caught it. A stricter marker would call that a sixth miss. We kept the rules exactly as they were when the test ran, since rewriting the rules after seeing the results is how benchmarks reach 100%.
What came back
Switch the first figure to By score and the five groups separate.
Tor. All 271 were flagged, with scores between 75 and 100. 237 of them were also sitting on threat lists, typically 5 different ones at once, and those landed in the critical band. The other 34 had no abuse record anywhere and scored in the high band on the strength of being an exit alone. The exits were spread over 86 networks, and not all of them are hosting companies. 8 sit in networks we class as consumer ISPs, two small American telephone companies among them.
Cloud servers. 248 of 250 scored in the medium band, most of them at exactly 37. That is the intended answer for a clean server: it is not a person, and it has done nothing wrong. One address broke away from the pack and scored 80. We assumed a bug. It was an open proxy. For addresses in hosting ranges the scorer tries a few common proxy ports and checks whether something answers in a proxy protocol, and on this randomly chosen address, something did.
Home connections. All 247 came back as ISP lines with no anonymizer flag. 173 scored a flat zero and none went above 13. That clean result hides two things, and both are worth more than the result.
Signals that fired on ordinary households
Share of the home connections in the test that tripped each check.
static.example-isp.net matches the same pattern a hosting company would use. Without a rule that holds clean ISP lines in the low band, 74 households would have drifted toward “medium” for no reason of their own.The first is in the figure above. 74 of the home lines tripped an infrastructure check, mostly because ISPs name static customer lines the way hosting companies name servers. Each of those would have been a small false positive. They stayed low because the scorer caps a clean line on a known consumer network at 15, a rule we described in an earlier post. The test did not reward the heuristics here. It rewarded the rule that overrules them.
The second is that the first time we ran this set, the home lines did not all pass. 228 of 247 did. The scorer decides what kind of network an address belongs to from a table of network types, and when we went looking for the missing nineteen we found the table held hosting networks and nothing else. The rows for ISPs, businesses, universities and governments, which a monthly import is supposed to supply, were not there. Recognition of consumer ISPs had been running on a fallback that matches words in the operator’s name. We ran the import by hand, about sixteen thousand networks appeared, and the home lines went to 247 out of 247.
The 995 is an after-number. Before the test found our bug, it would have been 976.
We did not stop to ask why the rows had been missing, which turned out to matter. Using a benchmark this way seems fair to us, since finding that sort of thing is what it is for. But a figure measured after you fixed what the test exposed is a different object from a figure measured cold, and the homepage does not have room to say so.
Websites and resolvers. The front-ends of well-known sites mostly scored either low (79) or medium (120), depending on whether they live in cloud ranges. This is the group that produced four of the five misses.
The five
What each one was, what we said, and why
soundcloud.comA CDN edge address serving a music site
An edge address like this carries traffic for thousands of customers. It was on a threat list, in a cloud range, with no reverse DNS, and all three counted against it.
telegram.orgA front-end address of a messaging service
On a threat list, with no reverse DNS and no trusted-owner match to soften it.
zoom.usA front-end address of a video-call service
Threat listing, datacenter range and no reverse DNS again. It stopped two points short of the high band.
archive.orgThe web front-end of an internet archive
The archive also runs a well-known crawler under the same domain, and our crawler check fired on the front-end's hostname. This address serves pages. It does not fetch them.
b.root-servers.netOne of the thirteen DNS root servers
It changed address in 2023, and the block it moved into is one our data files under hosting. So it got the standard server penalty.
Three of the five are the same mistake. A threat list records that abuse arrived from an address. It does not record who was behind it, and on shared infrastructure those are very different questions. A content network’s edge address carries traffic for thousands of customers, and when one of them misbehaves the address gets reported. Our scorer saw a listing, saw a datacenter range, saw no reverse DNS, and added them up as if it were looking at a rented server with a bad history.
The fourth is a category error of our own making. The archive’s web front-end shares a network and a domain with its crawler, and our crawler check fired on the hostname. A front-end that serves pages is not a bot. The fifth is one of the thirteen DNS root servers, which changed address in 2023; the block it moved into is one our data files under hosting, so it picked up the standard server penalty and landed on the first rung of the medium band.
None of these five would hurt a real user, because websites do not sign up for accounts. They matter anyway. They show where the method bends: an address that many parties share, or one whose role changed while the paperwork stayed the same.
What a thousand addresses cannot tell you
Here is what the set leaves out. If you are deciding whether to trust a score, ours or anyone’s, this list matters more than the 995.
There are no commercial VPN exits in it, because we could not find a label for them that was independent of the lists we already use. There are no residential proxies, the infected or rented home connections that are the hardest thing in the field: they look exactly like the households they sit in, and no public list of them exists. There are no mobile carriers and no IPv6. The home lines were chosen from the biggest ISPs in the world and filtered by a tidy hostname, which makes them the easiest home lines there are; a small regional provider with no reverse DNS is a harder question and we did not ask it.
And we marked our own homework. We wrote the scorer, built the set, wrote the marking rules and ran the test. The labels are independent. The marker is not.
On a set of 1,000 IPv4 addresses whose category is known from outside sources, the service put 995 in the right category on 28 October 2026. That is a statement about categories on easy-to-label addresses. It says nothing about fraud on your traffic, and nothing about the hard categories listed above.
How long an answer takes
Every query in the run was cold: nothing cached, each address seen for the first time. The median was 424 ms, but that hides a spread of nearly thirty to one between groups.
Played back at a little over half speed
Cloud addresses are slow because that is where the scorer does the most work. An address in a hosting range gets the proxy-port probe described above, and 144 of the 250 had no reverse DNS record, so that lookup ran to its timeout as well. Two seconds is a long time to wait in a signup flow. It is also the check that found the open proxy. Repeat queries for the same address inside a minute come from cache and skip all of it.
Three weeks later
A benchmark you ran once is a photograph. While writing this post, on 17 November, we ran the same thousand addresses again without changing anything in the set.
The first thing that had changed was the truth. Of the 271 addresses that were Tor exits on 28 October, 253 still are. The other 18 have left the Tor Project’s list. Twenty days, and 6.6% of the labels in our easiest category had expired.
18 of them are not exits any more
The scorer had mostly kept up. All 253 that are still exits were still flagged. Of the 18 that retired, we had already stopped calling 14 of them exits, and 5 of those now carry a “Tor relay, non-exit” note instead: the machine is still part of the network but no longer lets traffic out to the internet. The remaining 4 we still flag as exits. Going by the Tor Project’s list, those four are stale on our side.
This is where the marking scheme falls over. Mark the re-run against the October labels and it scores 982, thirteen lower than before, and almost all of that drop comes from addresses where changing our answer was the right thing to do. Remove the 18 expired labels and it is 978 out of 982, but then the four stale exits vanish from the count along with them. Neither figure can be compared with 995. A labelled set starts going off the day it is built, and the Tor part of ours has a shelf life of weeks.
| Category | 28 Oct | 17 Nov, old labels |
|---|---|---|
| Tor exit nodes | 271 / 271 | 257 / 271 |
| Cloud servers | 250 / 250 | 250 / 250 |
| Home connections | 247 / 247 | 247 / 247 |
| Website front-ends | 197 / 201 | 198 / 201 |
| Resolvers and root servers | 30 / 31 | 30 / 31 |
| Total | 995 / 1,000 | 982 / 1,000 |
The misses moved as well, and we had touched nothing. Two of the original five fixed themselves: the messaging and music addresses dropped off the threat lists that had caught them and now score 12 and 37. A new one took their place. A front-end address of a large social network picked up a listing during those three weeks and went from 37 to 67. The video-call address, the archive and the root server returned exactly the scores they had in October. Across all thousand addresses, 841 scores were identical to the point and 44 had moved by ten points or more.
So the shared-infrastructure problem is not five particular addresses we could add to an exception list. It is a steady trickle. On any given day a handful of famous front-ends are on somebody’s list, and which handful changes.
The bug came back
There is one more thing the re-run turned up, and it happened before we pressed go. When we opened the network-type table to prepare, the ISP rows were gone again.
The cause was dull. Two monthly jobs write to that table. One adds the ISP, business and university rows. The other rebuilds the whole table from its own sources and swaps it in, and it knows nothing about the rows the first job added. The first ran on the 2nd of the month and the second on the 5th. For three days a month the table was complete. On 5 November the rebuild wiped it, and for the twelve days until we looked, consumer ISP recognition was back on the name-matching fallback, the same state the October run had caught.
We restored the rows, moved the import to run after the rebuild, and added a daily check that re-imports and raises an alert if the table ever loses its ISP rows. Then we ran the numbers above. That makes the 247 out of 247 for home lines an after-number for the second time. We do not know what it would have been an hour earlier, because we fixed it first. In October the same condition cost nineteen.
The benchmark found this bug in October. It did not keep it fixed, because nobody ran it again until there was a blog post to write.
Run your own
The useful version of this test is one we cannot run, because it needs your traffic. Take a hundred addresses you know something about: customers who have paid you for years, accounts you banned for card fraud, your office, your own phone on mobile data. Put each through the score endpoint, or paste them in one at a time, and see whether the answers sort your good from your bad. A hundred addresses with known outcomes from your own business will tell you more than our thousand.
If you do, and we get one of yours badly wrong, send it to us. Five misses is a short list and we would like a longer one.
Test set, results and scripts for both runs are archived and the run can be repeated against the live service. Our earlier 16-address comparison from May 2026, which this test replaced, scored 14 correct against 10–11 for two paid services; we no longer quote it.