Web page quality statistics by Nikita The Spider
Join the DZone community and get the full member experience.Join For Free
Statistics about Web page quality – like the frequency of certain validation error messages and the popularity of XHTML versus HTML doctypes – are pretty esoteric stuff, and likely not of great interest to most Webmasters. But they are of interest to the people who build the tools that build the Web, like the fine folks behind the W3C Validator.
If you're one of those people, or you're just interested in Web site quality, then you might be interested in this statistical overview of things that Nikita examines about each page she validates. (If you're unfamiliar with Nikita, you can see a sample of what Nikita reports about a site or even take Nikita for a free test drive.)
The data for each topic below are summarized as a table and, where appropriate, a pie chart. The data are an aggregate of the sites that Nikita has seen; no particular site is singled out. You can jump directly to:
- A description of my sampling methods
- Validation message statistics
- Encoding statistics
- Encoding source statistics
- Doctype statistics
- Media type statistics
- The conclusion
In March 2008 Philip wrote a small program to review the data generated by Nikita's most recent crawls. The program ignored duplicate crawls of the same site and all sites where Nikita saw fewer than 30 pages. This left 360 crawls of unique sites. From each of these, the program selected 30 pages at random for a corpus of 10800 pages. (10800 =p
There are a total of 540726 validation messages in the sample containing almost 5400 unique messages. Philip decided (somewhat arbitrarily) to show only the 25 most frequent messages in the table below. These 25 represent just over half (55%) of the sample.
Note that the message 'required attribute "ALT" not specified' appears twice, once with the attribute in uppercase and once in lowercase (an HTML/XHTML difference). If Philip was to combine these two entries, it would be the most common error message by a large margin. Even without combining them, it's still the most frequent.
|Instances||Portion of Total||Message|
A page's encoding is sometimes also called its "character set" or "charset" for short.
In the pie chart and table below you can see that the vast majority of encoding declarations are either UTF-8 or ISO-8859-1. About half of the pages in the sample (52%) declare an encoding of UTF-8 and four in ten (41%) declare ISO-8859-1. The third most popular is Windows-1252 (an ISO-8859-1 superset) with a meager 2% of the total.
When no encoding is specified, Nikita usually defaults to ISO-8859-1 per HTTP rules.
In the table below, the total number of encodings found is greater than the number of pages in the sample because many pages declare multiple encodings. (Nikita will warn you if she finds multiple encodings declared for a page on your site.)
|Instances||Portion of Total||Encoding|
|* Philip counted the encodings "UTF-8" and "UTF8" as the same. For the record, the former is the overwhelmingly more popular expression with 7,423 occurrences compared to 23 for the latter.|
Web pages can declare their encoding in four different places: in the HTTP Content-Type header, inside the HTML in the META HTTP-EQUIV Content-Type tag, in the file's BOM, or in the XML declaration. (You might want to read about how Nikita divines a Web page's encoding from this jumble of information.)
In the pie chart and table below you can see that almost ⅔ of the encodings are specified via a META tag and most of the rest were declared via HTTP.
In the table below, the number of encoding sources is greater than the number of pages in the sample because many pages declare their encoding in multiple places. (Nikita will warn you if she finds multiple encodings declared for a page on your site.)
|Instances||Portion of Total||Source|
|9021||63.61%||META HTTP-equiv tag|
|4299||30.32%||HTTP response header|
|737||5.20%||Fallback to default|
|123||0.87%||The file's BOM (byte order mark)|
|1||0.01%||The XML declaration|
There are 41 distinct doctypes in the sample. For purposes of discussing them here, Philip considered doctypes the same if they used the FPI (Formal Public Identifier). For example, these two doctypes were considered equivalent:
- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
- <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
Philip chose not to display doctypes that represented < 1% of the sample.
In the pie chart and table below, you can see that XHTML doctypes represent a little more than ⅔ of the sample, with HTML (of course) making up the remainder. Interestingly, transitional doctypes (both HTML and XHTML) dominate the field.
In the table below, the number of doctypes sums to less than the number of pages because Philip ignored doctypes that represented < 1% of the total.
|Instances||Portion of Total||Source|
|4123||43.93%||XHTML 1.0 Transitional|
|1994||21.24%||XHTML 1.0 Strict|
|1655||17.63%||HTML 4.01 Transitional|
|533||5.68%||HTML 4.01 Strict|
|249||2.65%||HTML 4.0 Transitional|
A media type is also sometimes called a "content type". nearly all of the pages in this sample (> 99.6%) use the text/html media type except for the one-third of one percent that use application/xhtml+xml.
Nikita's view of the world might be slightly skewed away from the application/xhtml+xml for a couple of reasons. First of all, Nikita sends an Accept header of */* and many servers capable of sending application/xhtml+xml might do so only if they find that exact string in the Accept header sent by the client.
Second, Nikita doesn't attempt to masquerade as a browser via her user agent string; it is simply "Nikita the Spider" and doesn't contain words like "Mozilla", "Opera", "WebKit", "KHTML", etc. that might convince servers that shes's capable of handling XHTML. It's likely that cautious servers choose to send text/html to Nikita.
The overwhelming dominance of text/html makes the pie chart and table below superfluous but they're here for completeness.
Unlike some of the other properties discussed in this article, there's a simple 1:1 relationship between pages and media types. Each page has exactly one which is specified in the HTTP Content-Type header. (It's possible to specify something else in an HTML META HTTP-EQUIV tag, but this is non-standard and of no practical use. Nikita doesn't look for media types specified in the page contents.)
|Instances||Portion of Total||Source|
The statistics above give an overview of what Nikita has seen on the sites she's crawled recently.
The most frequent validation messages are probably not a great surprise to anyone who has hand-coded HTML. It's unfortunately easy to forget the alt attribute on img tags, and the second-most common error ("reference not terminated by REFC delimiter") is a sure sign of unescaped ampersands (&, ASCII 38). Validators are excellent tools for catching these sorts of mistakes.
The frequent use of META tags to specify encodings speaks to a common inability to set encodings via HTTP headers, or an unawareness of that ability.
The most surprising observation to me is the widespread use of the transitional doctypes. At the time this data was collected in 2008, the transitional doctypes were already more than eight years old which is about half the lifetime of the Web itself. That's quite the lengthy transition...
Philip believes his sample is a sufficiently random representation of what Nikita sees, but what Nikita sees doesn't represent the Web as a whole. For one thing, Nikita has mostly been promoted in English-speaking venues. Also, it's reasonable to assume that those who ask Nikita to crawl their site are more aware of Web standards than the average Web site author. To quote How To Lie With Statistics, "The result of a sampling study is no better than the sample it is based on."
This article is copyrighted to Philip Semanchuk under an attribution, non-commercial, share-alike Creative Commons License.
Published at DZone with permission of Schalk Neethling. See the original article here.
Opinions expressed by DZone contributors are their own.