ENOSUCHBLOG

Programming, philosophy, pedaling.

• Home
• Tags
• Series
• Favorites
• Archive
• Main Site

Scanning the .nyc gTLD

Nov 11, 2018     Tags: data    

Disclaimer

All of the techniques and data described in this post were used and collected for research purposes only.

Only publicly available information is presented below and in the linked dataset.

Did you know:

• …that NYC has its own gTLD?
• …that only New Yorkers can register for a domain under it?
• …that the Mayor’s Office of Data and Analytics provides a daily CSV dump containing every .nyc domain?

Overview

As of 11/3, there are 71,762 active domains¹ under the .nyc gTLD².

The first .nyc domain registered was nic.nyc, unsurprisingly.

Live domains

I was originally going to run a WHOIS lookup for each domain, but they (the Registry Operators for .nyc) started limiting WHOIS results back in May. Oh well.

Instead, I ran DNS queries to see how many domains were currently pointing to servers. Of the 71,762 domains in the dataset, 65,009 have DNS records pointing to one or more IPs or CNAMEs^{3 4 5}.

The vast majority of sites have just one IP, while a sizable minority have four (consistently CDN IPs):

Between those 65,009 domains there are 8550 unique IPs⁶, less than I would have thought.

The top IPs are, unsurprisingly, large CDNs and hosting providers^{7 8}:

Reader: I’d never heard of Confluence-Networks before and couldn’t find any real information about them online, so I’d appreciate any insight you might have.

Scans

I decided to run nmap on the remaining 65,009 hosts, checking only for a few common open ports. This ended up taking about four days in “polite” (-T2) mode:

1
2
3
while read domain; do
  nmap -v -p 21,22,80,443 -T2 --append-output -oG nmap.txt
done < livedomains.txt

(nmap -iL <file> does about the same thing as this loop on newer versions of nmap, but my older version didn’t support domains via that flag.)

This resulted in a big old text file in nmap’s “greppable” format (protip: do not use this format), from which I extracted port information⁹. Here’s the breakdown for open ports:

So, about one out every ten domains on .nyc (a relatively new gTLD, mind you) is still running an FTP service in 2018. Pretty much every domain is serving HTTP, but only about one quarter are doing so over a not-completely-insecure channel.

Since the number of SSH and FTP hosts was nontrivial, I decided to fingerprint them^{10 11}. Of the 8021 SSH services discovered during the initial scan, 7982 (99.51%) responded to the fingerprint scan. For FTP, it was 6434 out of 6487 (99.18%).

SSH fingerprinting results (independent columns):

Some interesting outliers there: I had no idea that ProFTPD had SFTP support, or that people actually used SSH Communications Security’s proprietary SSH server. Serv-U appears to be another proprietary SSH offering.

26 hosts are running OpenSSH 4.3, which had a remote DoS (and potential ACE) all the way back in 2006. Only 765 hosts are running OpenSSH >= 7.4, the first version to fix CVE-2016-10708, a trivial DoS.

And for FTP (independent columns):

Note: “Unknown” above reflects that nmap couldn’t identify both the product name and its version — 3508 of those “Unknown” FTPDs are actually Pure-FTPd, making it by far the single most popular FTPd on .nyc.

Some interesting outliers: WarFTPd is a very old Windows FTP server, and I can’t even find an original source for Gene6.

Several hundred hosts are running versions of ProFTPD that may be vulnerable to CVE-2011-4130. Several dozen are running older versions of vsftpd that may be vulnerable to CVE-2011-0762.

Websites

To cap things off, I wanted to see how many websites were following security best practices.

To do that, I ran twa on 1000 randomly sampled domains¹² that the nmap scan indicated had both HTTP and HTTPS available¹³. For simplicity’s sake, I only ran twa on the base domain, not www or any other common subdomains.

1
2
3
4
5
6
while read domain; do
  TWA_TIMEOUT=2 twa -c "${domain}" | tee -a twa.csv
done < randomhttps.txt

# remove the CSV headers
sed -i '/status,domain/d' twa.csv

Some interesting statistics:

70 (7%) of the websites scanned sent a Server header containing version information (e.g., nginx/1.14.0)¹⁴:

260 (26%) of the websites sent one or more cookies missing either (or both) the 'secure' or 'httpOnly' flags¹⁵. The worst offender sent 18 unsecured cookies¹⁶!

235 websites redirected HTTP requests to HTTPS using a 301 (or other permanent redirect)¹⁷, while 25 used a 302 (or other temporary redirect)¹⁸. 549 websites were serving HTTPS (confirming the nmap scan), but didn’t redirect their HTTP traffic¹⁹. Another 191 websites redirected their HTTP traffic to another HTTP endpoint²⁰. Thus, without intervention from a browser extension like HTTPS Everywhere, the average user will wind up using plain old HTTP on the average .nyc domain. Not great for 2018.

106 (10.6%) of the websites were listening on one or more non-production ports, possibly indicating either a development version of the site or some kind of backend service²¹.

There are a bunch of other interesting datapoints in twa.csv, but I’ll leave it up to you to dig through and interpret them.

Wrap-up

This blog post took longer than I thought it would — I started writing it on the 3rd of November, but many of the scans I used didn’t complete until the 9th. As a result, I limited the amount of analysis that I did on the data. It would be interesting to see someone more statistically inclined than myself take it on.

I’ve published an archive containing some of the data and tiny scripts I used for this blog post. I have left out files that contain scan information for any particular domain or domains, including my copy of the nmap scan. I’d like to share the full data with people interested in legitimate research — if that applies to you, please contact me directly and we’ll work out some kind of agreement about usage²².

Where applicable, the footnotes below refers to a specific file or invocations of one of the scripts in that gist. Note that most of the scripts are ad-hoc, and do things via stdin/stdout — you should modify them if you need to do anything more complicated.