Twitter Clients

Danny Sullivan has a flawed piece today titled “Is Twitter Sending You 500% To 1600% More Traffic Than You Might Think?” The sensational headline is no doubt good for traffic and his basic information and method is good, but the numbers are way off because he has a miniscule sample size.

Google Analytics, and other analytics tools, misreport Twitter traffic because many Twitter users use desktop clients that don’t show any referrer data. When these visitors open a link from their Twitter client, they get reported as direct traffic.

Amongst the analytics community there is a debate over how exactly how much traffic gets misreported. Tweet Stats shows 52% (at the time of writing) of visits coming from the web while TwitStats shows just 61% (at the time of writing).

Danny Sullivan’s Numbers

Danny does some testing to see how much direct traffic in Google Analytics actually comes from Twitter.

His method is to tweet a link to a post tagged with tracking parameters. He then compares his properly-reported tagged links against his Google Analytics referrer data and his Bit.ly stats against his log-files to get his numbers.

His final numbers?

Google records 9 visits tagged with the tracking parameters, but just 2 of those come from Twitter (as in the web site) thus under-reporting by 450% (he rounds it to 500%, I’m cool with that).

Bit.ly records 58 clicks, but there are only 32 tagged visits in his logs. The unaccounted Bit.ly clicks are caused by url-lengthener plugins that request the Bit.ly URL so that someone can see where the shortened URL goes before clicking but don’t follow the redirect. (Bit.ly deserves props for being very up-front about the gaps in their data.)

Log-files record 32 visits tagged with tracking parameters, but just 2 of those come from Twitter (according to Google Analytics) thus under-reporting by 1600%.

Google, in response to Danny’s questions about the differences, says that they are working on some issues caused by mobile devices.

Reconciling GA’s and Bit.ly’s Numbers

I give Google’s numbers more credit than Bit.ly’s.

I get notified every time there is a 404 on one of my sites. I’ve spent a lot of time investigating, and blocking, strange requests. Digging through log files and discovering all sorts of strangeness has given me a great appreciation for how much crap that Google Analytics blocks out.

There are a ton of bots out there, both malicious and innocent, that can quickly inflate your states. GA catches most of them. As a result, I’m inclined to believe the GA numbers whenever there is a big discrepancy.

Even with under-reporting caused by issues with mobile devices, I believe Google’s numbers are closer to the actual numbers than Bit.ly’s.

The Numbers in Greater Context

Danny’s 500% is in line with TwitStats’s numbers, and what I would have guessed because I can’t imagine not using a desktop client. But I’ve got some pretty good data, from a much larger sample than Danny’s, that aligns closely with Tweet Stats’s 52% web visits.

Soon after launching this site, I had a moderately influential Twitterer tweet my post on Involvement Device CAPTCHAs, thanks Naomi.

At the time I was getting very little traffic. Out of the traffic to that post on that particlur day (shown below), only 2 pageviews entered from elsewhere on the site and there was no search traffic to that page so the data is relatively unpolluted.

Google Analytics Twitter Stats

The BeTwittered, Netvibes, and TwitterGadget traffic are all functionally Twitter traffic (bonus points to anyone who writes a filter to group all of this traffic together).

I got 45 Twitter visits with referral information (41 actual visits, plus 4 functionally Twitter visits) and no more than 51 direct visits that actually came from Twitter. Because I was just starting up, I was getting practically no direct traffic and I’d already filtered out my own visits so nearly all, if not all, of these visits actually came from Twitter.

47% of my Twitter visits come from the web and show up in my Google Analytics stats. My Twitter stats are under-reported by about 200%.

It is impossible for this particular post to be under-reported by the 500% that Danny reports simply because all of my direct visits are already presumed (and likely) to be actually from Twitter.

Explaining the Differences

Why is there such a big difference between my numbers and Danny’s?

Here’s a hint. He’s only working with 2 Twitter visits in Google Analytics. If he had just one more visit reported in GA, his headline would be 300% to 1000% instead of 500% to 1600%. Compared with my 45 Twitter visits, his sample size is just too small.

Even though my experiment was accidental while Danny did a better job of setting up the experiment by tagging links and comparing tagged values, my larger data set on a page with less noise makes my data better.

The 47% of visits coming from the web on my post is also a lot closer to the TwitStat and Tweet Stats numbers which use an even better sample size.

Photo Credit: The Next Web

Is Google Analytics Really Under-Reporting Twitter Traffic by 500%? is a post from: Damon Gudaitis

Google is starting to add the search ranking directly to the referring URL which means the convoluted search ranking filters I wrote about earlier will soon be obsolete. That’s good news because now we’re going to have much better information than before and it’s going to be easier to get.

I quickly took the sample query string from the Clickz article and created a set of filters that should give you the exact search ranking in Google Analytics.

http://www.google.com/url?sa=t&source=web&ct=res&cd=7&url=http%3A%2F%2Fwww.example.com%2Fmypage.htm&ei=0SjdSa-1N5O8M_qW8dQN&rct=j&q=flowers&usg=AFQjCNHJXSUh7Vw7oubPaO3tZOzz-F-u_w&sig2=X8uCFh6IoPtnwmvGMULQfw

I can’t test this out because I’m still getting the old query string, but I’m confident this will work.

Setup Instructions

Create a Profile for the Filters

If you need instructions explaining how to do this, have a look at my original clean SERP report post. I called my new profile Google Search Ranking.

Filter Organic Traffic

Create a Custom > Include filter. Choose Campaign Medium for the Filter Field and enter organic for the Filter Pattern. More detailed instructions are again in my original clean SERP report post.

Filter Old Google Organic URLs

New Google Organic Filter

Google is rolling out the new URL format slowly. That means you’re going to get a lot of matches with no data if you don’t filter out the old URLs. As an added bonus, this filter also banishes Yahoo and Live as well, so we won’t be needing to create a separate filter to get rid of them.

Create a Custom > Include filter. Select Referral for the Filter Field and enter google\.[a-z]+/url in the Filter Pattern box.

The /url part of the Filter Pattern identifies search queries using the new string.

Output Google Ranking

Google Ranking Filter

1. Create a Custom > Advanced filter.
2. In the Field A -> Extract A area, select Campaign Term and enter (.*).
3. In the Field B -> Extract B area, select Referral and enter (\?|&)cd=([^&]*).
4. In the Output to -> Constructor area, select User Defined and enter $A1 | Rank: $B2.

Assign Filter Order

Make sure the filters are in the order you created them. You can do this in the Profile Settings page by clicking on the Assign Filter Order link just above the list of filters applied to this profile.

Viewing the Search Ranking Report

You can see your data in the Visitors > User Defined report.

Remember, though, it takes about a day before you start seeing data and there’s a good chance you won’t get any visitors using the new query string for awhile so don’t panic. Yet.

Google Analytics Search Ranking Filters for New URLs is a post from: Damon Gudaitis

I just started testing some of the features for a WordPress Google Analytics plugin that I’m working on when I discovered event tracking hadn’t been enabled for my account.

While trying to figure out if there was any way to get in to the event tracking beta, I discovered that you can access the Event Tracking report by typing in analytics/reporting/events as the URL.

This doesn’t work if you have more than one account as it defaults to the first account listed alphabetically. If, like me, you have analytics accounts for several different sites under your Google account, then you need to add get the id parameter from the query string to view the correct account.

To access Google Analytics Event Tracking reports in a multiple account setting:

1. Go to the Dashboard page for the desired account.
2. Find and copy the 7 numbers that make up the id in the query string (id=0000000).
3. Replace the 0′s in the following url with your id https://www.google.com/analytics/reporting/events?id=0000000.

Done.

Event Tracking

[btbuckets#direct_and_twitter]

Access Google Analytics Event Tracking Before It’s Enabled is a post from: Damon Gudaitis

A few people have asked me about viewing Google, Yahoo, and MSN separately in the Google Analytics search ranking report.

At first I recommended creating 3 separate profiles, 1 each for Google, Yahoo, and MSN. But using the Google Analytics custom segments makes a lot more sense because it’s easier to switch between the 3 when viewing the SERP report and it’s easier to set up.

Segmenting the Search Ranking Report

1. In the Organic Search Ranking report, in the Settings section of the left menu, select Advanced Segments.
2. In the Manage Advanced Segments page, select Create new custom segment at the top right of the screen.
3. From the list of Dimensions on the left, open the Traffic Sources dimensions and drag Source to the dimension or metric box.
4. From the Condition box, select Matches regular expression. Matches regular expression is better than Matches exactly because Matches regular expression lets you enter case insensitive Values.
5. In Value enter google or yahoo. For MSN create two statements using Add “or” statement one matching msn, the other matching live. I’m surprised you have to do this, but it seems the pipe (|) character doesn’t work in the custom segment regular expressions. Maybe it’s because the custom segments already have an or function built in.
   

   MSN Settings

6. In the Name segment box, give your segment a name. Google Traffic seems appropriate.
7. Repeat for Yahoo and MSN.

Viewing the Segments

1. Open your User Defined report.
2. In the Advanced Segments box at the top right of the page, click to open the Advanced Segments dialog.
   

   Open Advanced Segments

3. In the Custom Segments area, select your Google, Yahoo, and MSN segments and click Apply.
   

   Select Custom Segments

Your organic traffic will be graphed by search engine and your keywords will be segmented by search engine.

Unfortunately, keywords on different pages will appear multiple times in the report. Anyone got any experience writing Greasemonkey scripts so we can see keywords grouped together with the pages showing after each search engine?

Enjoy.

Segmenting the Clean SERP Report in Google Analytics is a post from: Damon Gudaitis

I just updated my original Clean Ranking Report in Google Analytics post fixing and improving a number of things.

I still can’t figure out how I messed up in the first place, but I figure it must have been because I was running about 6 different test profiles trying different ways of cleaning up the report. I must have gotten them mixed up somehow because I know I had it working.

Once I had worked out how to clean up the ranking report, I erased all the unnecessary profiles and filters, so I didn’t have any way of retracing my work.

Changes

I changed the Search Ranking filter’s Field A->Extract A to use Campaign Term instead of Referral. The search terms from the Campaign Term field look a lot prettier than those from the Referral field.

I was really surprised this worked because Campaign Term is also used to track PPC bid terms. If you are using broad match in your PPC campaign then your bid term won’t necessarily match the search term.

Of course none of this is set with organic traffic, but it doesn’t seem like an intuitive place to put this information.

I was having all sorts of trouble getting Google Analytics to match parenthesis. Yes, I know how use the escape character. No, it didn’t work.

So instead of beating myself over the head, trying to get everything to work with the parenthesis intact, I changed the Search Ranking filter’s Output To->Constructor to get rid of the parenthesis. This makes formatting later a lot easier and, if you really want to see the page number in parenthesis, you can probably add them back in with your Search and Replace filters.

The Search and Replace filters are completely revamped. Not only do they work now, but I also got rid of some unnecessary characters. I also used Google-friendly end of string anchors in my regular expressions which might not have any benefit to you, but they match more efficiently saving processing power.

Clean Ranking Report Reloaded is a post from: Damon Gudaitis

Joost de Valk has a post up explaining how to track Google search ranking in Google Analytics. The SEO ranking hack (Dutch) is by Andre Scholten, he guest posted on Yoast for English speakers.

I added a comment to the post showing how to use this technique to also track Yahoo and MSN rankings as well.

But the ranking report diplays page 1 results as unset, (page: ), page 2 results as (page: 10) or (page: 11). SERPs from subsequent pages are similarly awkward.

If you don’t mind writing a few extra filters, you can display search rankings more intuitively. Here’s how.

Basic Instructions

If you are reasonably experienced with Google Analytics, you should be able to get by with the basic instructions. If you are new to Google Analytics, there are very thorough detailed instructions further down the post.

1. Create a profile for your search rankings filters.
2. Create filters that include only organic traffic and only traffic from Google, Yahoo, MSN, and Live.
3. Create a Custom > Advanced filter to extract search engine rankings using the following settings:
   1. Field A-> Extract A from Campaign Term: (.*)
   2. Field B-> Extract B from Referral: (\?|&)(start|first|b)=([^&]*)
   3. Output To-> Constructor: $A1 | page: $B3.
4. Create a Custom > Search and Replace filter to format page 1 results on the User Defined field using the following settings:
   1. Search String: page:$
   2. Replace String: page: 1.
5. Create Custom > Search and Replace filters to format page 2+ results on the User Defined field using the following settings:
   1. Search String: page: 1[0-1]$ for page 2, page: 2[0-1]$ for page 3 etc…
   2. Replace String: page: 2 for page 2, page: 3 for page 3 etc…
6. Check that the Search and Replace filters are in numerical order.

As you can probably imagine, step 5 can get time-consuming if you are dealing with a lot of sites. I tried number of different arithmetic ways of formatting the results that would have made set up simpler, but none of them worked. Right now, it seems you can’t perform mathematical manipulation of numbers in Google Analytics.

I couldn’t figure out any easier way of displaying the SERPS but if you want to experiment yourself, the Replace String field from the Custom > Search and Replace filter requires a valid regular expression and seems more likely to allow basic arithmetic.

If you plan on using this hack, please be sure to write a whiny blog post about how Google Analytics should let you:

1. export profiles to other accounts,
2. allow filters to perform basic arithmetic,
3. make this hack standard on all accounts.

Maybe if enough of us whine, someone will listen.

Detailed Instructions

Careful step-by-step instructions for people who aren’t very familiar with Google Analytics.

Create Profile for Search Ranking Filters

You need to create a separate profile for the search ranking filters because  applying the filters will exclude everyone except for organic search visitors from Google, Yahoo, and MSN.

1. On the Overview page, select Add Website Profile.
2. On the Create New Website Profile page, select Add a Profile for an existing domain.
3. In the Select Domain box, choose the domain you want to track.
4. In the Profile Name box, enter Organic Search Rankings (or some similar name).
5. Enter Country and Time zone as appropriate.

Filter Organic Traffic

Include only organic traffic.

1. On the Overview page, select Filter Manager.
2. On the Filter Manager page, select Add Filter.
3. On the Create New Filter page in the Filter Name field, enter Organic Traffic (or other similarly descriptive name).
4. From the Filter Type box, select Custom Filter and choose Include.
5. From the Filter Field box, choose Campaign Medium. I’m really surprised that Campaign Medium works here because all the other campaign options refer to PPC and other links that you can tag with traffic parameters.
6. In the Filter Pattern box, enter organic.
7. In the Apply Filter to Website Profiles area, select Organic Search Ranking and click Add to apply the filter to the profile.
8. Click Save Changes.

Restrict to Google, Yahoo, and MSN

Include only the organic visitors from Google, Yahoo, and MSN.

1. On the Filter Manager page, select Add Filter.
2. On the Create New Filter page in the Filter Name field, enter Search Engine Visitors.
3. From the Filter Type box, select Custom Filter and choose Include.
4. In the Filter Field box, choose Referral.
5. In the Filter Pattern box, enter (google|yahoo|msn|live)\.[a-z]+. This regular expression means
   1. (google|yahoo|msn|live) match one of google, yahoo, msn, or live
   2. \. match a literal period (.)
   3. [a-z]+ match 1 or more lower-case letters (for the TLD).
6. In the Apply Filter to Website Profiles area, select Organic Search Ranking and click Add to apply the filter to the profile.
7. Click Save Changes.

Extract Search Ranking from URLs

Extract the term and results page parameters from the referring search engine’s query string and output it to the User Defined report.

Google uses q to identify the term and start to identify the results page in the query string.

• Page 1:  http://www.google.com/search?client=opera&rls=en&q=seo&sourceid=opera&ie=utf-8&oe=utf-8
• Page 2: http://www.google.com/search?hl=en&client=opera&rls=en&hs=QB0&q=seo&start=10&sa=N
• Page 3: http://www.google.com/search?hl=en&client=opera&rls=en&hs=tB0&q=seo&start=20&sa=N

Yahoo uses p to identify the term and b to identify the results page in the query string.

• Page 1:  http://ca.search.yahoo.com/search?p=seo&fr=yfp-t-501&toggle=1&cop=&ei=UTF-8
• Page 2:  http://ca.search.yahoo.com/search?p=seo&ei=UTF-8&fr=yfp-t-501&xargs=12KPjg1ihSroGmmvmnEOOIMLrcmUsOkZ7Fo5h7DOV5CtdY6hNdE%2DIfXpP0xZg6WO8T7xvSy7HBreVFdJGu277WVk0qfeK%5FGr%5FM6c6Vk9AoNdukWnl3tOJ%5F0%2Dmdm6d%2DDA%2E%2E&pstart=6&b=11
• Page 3: http://ca.search.yahoo.com/search?p=seo&ei=UTF-8&fr=yfp-t-501&xargs=0&pstart=1&b=21

MSN uses q to identify the term and first to identify the results page in the query string.

• Page 1: http://search.msn.com/results.aspx?q=seo&FORM=MSNH11
• Page 2: http://search.msn.com/results.aspx?q=seo&first=11&FORM=PERE
• Page 3: http://search.msn.com/results.aspx?q=seo&first=21&FORM=PERE2

1. On the Filter Manager page, select Add Filter.
2. On the Create New Filter page in the Filter Name field, enter Search Ranking.
3. From the Filter Type box, select Custom Filter and choose Advanced.
4. In the Field A->Extract A box, select Campaign Term and enter (.*).
5. In the Field B->Extract B box, select Referral and enter (\?|&)(start|first|b)=([^&]*).
   1. (\?|&) match either a literal question mark(?) or an ampersand (&)
   2. (start|first|b) match one of start, first or b; Google, Yahoo, and MSN use these to identify starting point in search results
   3. = match an equals sign (=)
   4. ([^&]*) match everything until an ampersand (&).
6. In the Output To->Constructor box, select User Defined and enter $A1 | page: $B3.
   1. $A1 get the values from the 1st set of parenthesis in the Field A-> Extract A box
   2. | page: $B3 write a pipe character and page followed by the value from the 3rd set of parenthesis in the Field B-> Extract B box.
7. In the Apply Filter to Website Profiles area, select Organic Search Ranking and click Add to apply the filter to the profile.
8. Click Save Changes.

Format Page 1 Results

The query string parameter identifying the results page parameter for Google, Yahoo, and MSN are all unset on the first page, so you need a special rule to format page 1 results.

1. On the Filter Manager page, select Add Filter.
2. On the Create New Filter page in the Filter Name field, enter Page 1.
3. From the Filter Type box, select Custom Filter and choose Search and Replace.
4. In the Filter Field box, select User Defined.
5. In the Search String box, enter page:$.
   1. The dollar sign means match page: where it appears at the end of the string. Without it, the filter would match every occurance of page:.
6. In the Replace String box, enter page: 1.
7. In the Apply Filter area, select Organic Search Ranking and click Add to apply the filter to the profile.
8. Click Save Changes.

Format Page 2+ Results

Match results pages starting at result #10 or #11 and change to page 2.

Google starts counting results at 0, so page 2 results start at #10. Yahoo and MSN start at 1, so page 2 results start at #11.

Off topic: you can hand edit the query strings to start anywhere you like letting you put result number 5487 first. So don’t trust screenshots showing top results in Google & imitators.

1. For every search results page you want to track, on the Filter Manager page, select Add Filter.
2. On the Create New Filter page in the Filter Name field, enter Page 2 (or 3 for page 3, 4 for page 4 etc…).
3. From the Filter Type box, select Custom Filter and choose Search and Replace.
4. In the Filter Field box, choose User Defined.
5. In the Search String box, enter page: 1[0-1]$ (or page: 2[0-1]$ for page 3, page: 3[0-1]$ for page 4 etc…).
   1. page: 1[0-1] match page 10 or 11
   2. $ where it occurs at the end of a line.
6. In the Replace String box, enter page 2 (or page 3 etc…).
7. In the Apply Filter area, select Organic Search Ranking and click Add to apply the filter to the profile.
8. Click Save Changes.

Check Filter Order

Filters in the wrong order can mess up your results. Of particular concern are results from pages 10 or 11, 20 or 21, 30 or 31 etc…

Put your page: x rules in numerical order. If your page 10 or 11 rules comes before your page 2 rule, then page 10 or 11 results will be converted to (page: 10) or (page: 11) which will then match page:10 for Google or page: 11 for Yahoo and MSN. These will then be converted to page 2 when they pass through the Page 2 Search and Replace filter messing up your results.

Otherwise, filters are probably fine in any order because they act on different fields, in ways that are not mutually exclusive, or on fields that aren’t set until they pass through a filter. But I can confirm that the following order works:

1. Organic Traffic
2. Search Engine Visitors
3. Search Ranking
4. Page 1
5. Page 2
6. Page 3 etc…

To set filter order:

1. From the Overview page, click Edit for the Organic Search Ranking profile.
2. On the Profile Settings: Organic Search Ranking page, scroll down to the Filters Applied to Profile section and click Assign Filter Order at the top right of the section.
3. Adjust filter order as necessary.
4. Click Save Changes to finish.

View the Results

It takes about a day for Google to update your profile with the new filters.

To view the report:

1. In the Overview page, select the Organic Search Ranking profile.
2. In the left sidebar, click Visitors to open the menu for all visitors pages.
3. At the bottom of the list of visitors pages, click User Defined to see the results.

When you are done with this, you might want to check out my post on segmenting the ranking report by search engine.

Clean Rankings Report in Google Analytics is a post from: Damon Gudaitis