Robots.txt

The robots.txt file tells search engines which page not to bother crawling on your website (not that all of them will listen to it).  Less widely used now it still has the potential to easily block off an entire website to ALL the search engines with just one mis-guided line of code!  If you are going to create a robots.txt file then it’s best to steal one off another website that is doing things right: more on Robots.txt files

Relative links

Spotted on many early versions of ecommerce sites, relative links have the potential to turn into broken links in certain places.  An link such as <a href=”../contact/”>Contact</a> may work on URLs such as [example.com/categories] but break when added to sub-pages such as [example.com/categories/products].

A big SEO killer is also relative links with URL re-writes on systems such as Magento.  Re-writes can loop if a relative URL path is added to the end of an existing URL giving crazy results like this below that search engines crawl for hours on end until they give up:

http://www.example.com/ links to http://www.example.com/first-category
http://www.example.com/ links to http://www.example.com/first-category/second-category
http://www.example.com/first-category/second-category links to http://www.example.com/first-category/second-category/first-category
Etc. etc. until the spider runs of out allotted time.  This not only creates masses of duplicate content it also wastes time which could be spent crawling your unique pages.

Canonical tags

Canonical tags tell Google that the page they are looking at is actually a similar copy of another page elsewhere on the site.  Whilst very useful for hiding parameters and major URL changes for on-site SEO they can be very harmful if implemented incorrectly.  A canonical tag left in the header section of a scripted website would be copied across the entire site unless hidden with some IF statements.  This would tell Google that every page on your website was actually a copy of the page linked to in the canonical tag thus hiding all but that one page away in the SERPs!  Google does a “good job” of dealing with duplicate content on websites now so if you’re unsure how to make a dynamic canonical tag then leave them well alone.

Keyword spam

People amazingly still spam web pages with big long lists of keywords, locations and postcodes to try and quickly get those local or long tail searches.  This breaks Google’s Webmaster Guidelines which could be very harmful for SEO; it also creates ugly pages with sometimes monstrous amounts of meta information.  We’ve even seen one website paste every town, city and postcode in the UK and place them all on the ‘Contact Us’ page! (there’s a crazy amount of towns in the UK it seems!)

Conclusion

Play things safe and use webmaster tools to see if any issues are present on your on-site SEO.  If your number of indexed pages are a lot lower than the number in your XML sitemap then hunt around for common SEO goofs that might be holding things back.  If in doubt look for examples on well established websites of how they do things or use Wikipedia or Google to spot the correct solution.

Say you were building a brand new search engine, what are the advantages and disadvantages of checking and ranking for grammar and spelling?

Advantages of ranking well written content highly

- If grammar and spelling are correct the content is more likely to be of good quality
- No software would be required to correct spellings or grammar, it would be much quicker to process
- All keywords would be clear and a generic theme can be established with the content quickly
- Spam from cheap, foreign content writers would be harder to pass through
- Users of your search engine would always get clearly written content and make them happier

Disadvantages of ranking well written content highly

- People are discriminated against if they are learning English or are bad at it
- Brilliant content may be lost due to a mistake on the keyboard or mistake with a spelling correction
- Many people when typing online use abbreviations, txt speak and ‘kooky‘ made up words every time they type
- Every company with a blog will be wasting many man hours on spell and grammar checks instead of working
- People would be at risk when updating a homepage’s content as a bad sentence or two could drop their rankings
- Checking grammar takes up a lot of processing power, crawling the internet for grammar would be very costly

Looking at the advantages and disadvantages of spelling and grammar checks seems to suggest that filtering out sites with bad English wouldn’t be a smart move despite some of the improvements it may bring.  Blogs would take twice as long to write, people’s opinions would be lost and every website would be treading on egg shells to avoid any punishments with bad English.

I’m sure the major search engines such as Google, Yahoo and Bing have some sort of English checker and it will be a small ranking factor.  It would have to be a very simple algorithm to reduce the time spent crawling each page and this is why NASA scientists are now being employed by search engines, the whole thing is a headache!

What do you think about search engine rankings in regards to spelling and grammer?

Whereas it would be nice to rank for the biggest keyword out there is it what you should really be focusing on? When should you quit and target smaller keywords?

Cost of SEO

Let’s say for example you sold “Used Cars” on your website and you plan to be number 1 for the massive keyword “Cars”.  There are hundreds of other websites targeting the keyword “Cars”, some with literally millions of backlinks, millions of pages and a domain that has been around for over 10 years.  If SEO were a race they would have started way, way before you even contemplated competing, and guess what they are getting faster and faster at running!

It’s not impossible to win these search results but it will be very costly with the amount of link generation, content writing and product integrations you’ll need to beat the competition.  If there are 10 major corporations battling out for a keyword then you may have to beat teams of SEO workers just to reach page one!

Traffic Type

You could be number one for a massive keyword and find that 80% of your traffic are people that aren’t looking for your product.  Let’s say you got a website upto number 1 for the keyword “house”, you’d find people hitting your site looking to buy house, rent houses, let houses, house music or even the medical TV show ‘House’.  That would gain your website a lot of traffic that hasn’t found what they were looking for, the bounce rate of the site would be through the roof and it would be hard to maintain the position if the most popular subject on the keyword “house” wasn’t for your specific niche.

Window Shoppers

Anyone who has run a successful Adwords campaign will tell you that the top converting traffic comes from specific searches in Google.  Say you sold trainers, you’d find a much higher sales percentage from keywords such as “new balance trainers” than you would do with just “trainers” alone.

When people want to buy something they may shop around with a search for a generic keyword for example “trainers”.  After seeing large ranges of trainers online they may decide for a certain brand or colour; this is when they use a search engine again to find more specific trainers that they want and check prices, styles etc.

Being number one for a generic search term may not be as fruitful as you think but people who are not computer savvy, richer people or lazy people may stick on the first site they see and make a purchase so it will still make money.

What to Focus on with Keyword Optimisation

First of all quit aiming for the biggest most exciting keyword and start focusing on what people really search for when looking for your product.  If you sell wooden toys then stop optimising for “toys” and make sure you get to number one for “wooden toys” firstly!  Money made from winning business from small keywords will fuel ambitions for larger search terms and allow you to finally aim for the top.  Just aiming for the top straight away will end in tears and has make many, many business owners lose out over the past 10 years.

If you have passed the W3C validation then well done!  If you have failed then perhaps you don’t need to worry as much as you think… keep reading.

Let’s look at the number of validation errors for the top 10 results in Bing for the keyword “slippers”, we have no experience in “slipper” websites so it will be an un-biased keyword.  Also included is Google’s top 10 results for “lava lamp” and Yahoo’s top 10 results for “ice cream”:

Number of validation errors for top 10 ranking websites

There are no significant results from this small sample, if W3C validation were a ranking factor then it may only be a very small one.  This applies to all the major search engines, Matt Cutts from Google said in one of his Youtube videos that validation was a slippery slope for providing the best search results from content.

If validation were a big ranking factor then many small websites would be at a dis-advantage as they are usually created by people with little knowledge of HTML.  If you were searching for “the perfect way to sew on a button” for example then you may not find the golden advice displayed on sewing blog for example.

Also the world’s programmers would have a constant worry that their site didn’t validate, imagine the amount of man hours wasted worldwide every day if validation were a ranking factor!

The job of the web browser is to spot missing tags and try and re-build the website from what’s left intelligently.  Most coding errors don’t show up when viewed in advanced web browsers such as FireFox, Chrome and Internet Explorer.

Too much badly written code can confuse web spiders though, this is why it’s best to veer away from multi-table design sites in HTML as one missing </tr> tag can ruin an entire site.  Using content managed systems such as WordPress or Joomla means that it’s near impossible to break the site layout which is why we recommend it for any client.

So the overall line is that clean HTML code does little (if nothing) to effect your search engine position; if your website looks alright in all the major browsers then it’s probably alright and you can concentrate on optimising it whilst adding valuable content.

Do any of your main keywords have alternative spellings? If you sold “fridges” online for example then “refrigerators” would be an example.

Missing out on alternative spellings of your main keyword could lower your traffic levels.  On-site SEO should include the main variations of keywords; Google seems to count the multiples of words as the same word so for example “tent” would be counted equally as “tents”.  Major differences such as “camp” and “camping” will not be considered as the same keyword as the words could have different meanings, “camp” could be used to describe a personality trait for example.

If you sold “fridges” online then you may miss out on all the people who type in the word “refrigerator” into the search engines if that word never appears on your site.  Regular blogging usually solves the problem with different spellings, if you keep writing about your website’s main subject then you’ll eventually hit on all sorts of keywords that you may have missed with 4-5 static sales pages.

The main page of your website is usually the one that has the most Page Rank and therefor appears in Google’s search results more often.  Make sure your main keywords appear at least 2-3 times on the main page with all the different variations of spellings to make sure you get every bit of traffic possible!

Unfortunately many spell checks and coding practices prefer American English instead of UK English.  If you run an international site then it may be best to use both American and English spellings of words, for example “optimized” and “optimised” are two different spellings of the same word.  If your site is UK only then please stick to the English spellings, not only does it look more professional to readers, Google’s language centers may add a little more SEO power on the site for UK searches.

Twitter comprises of short messages that are under 150 characters long, any big gossip, news or viral gets ‘tweeted’ by thousands and quickly spreads across the globe.  Twitter is mostly used for self-promotion due to the fact that you can stick advertisements and links directly into the feeds of people that follow you.

Twitter can be used as an SEO tool, but beware, some SEO companies are using Twitter alone for SEO which isn’t right.  For starters you can only get NoFollow links from Twitter which don’t pass hardly any SEO power to the sites linked to.

When you create a post on your blog then tweet about it, you’d be surprised how many people search twitter for the latest tips or information and may find your blog.  Here at Real Web SEO we tweet every blog we make – take a look at our twitter profile here.  Choosing the best text to add next to you blog link is crucial, a boring title could never get clicked on whereas an exciting title like for example “10 ways to instantly make you life more exciting” may get hundreds of clicks!

So don’t use Twitter for the search engines, use it to drive traffic to your unique content.  It’s a social tool and if you have a creative skill then you could reach out to people worldwide with your talent.

Following people in your niche is the way forward too, about 30% of the people you follow will follow you back and they will see whenever you announce anything within their feed.  If you convince people in your niche to follow you then they may be truly interested in your latest project or news, what a wonderful PR tool Twitter can be!

The website TweetMeMe show you how many people have tweeted with a link to your website, it has a great plugin for WordPress and quickly shows you which posts you created have got picked up by the general public.  Real Web SEO’s record is currently 18 re-tweets, and our average post receives around 4 natural links from external sources.

The “robots.txt” file is a small text file on a website’s main root that tells search engine spiders how to interact with the site.  Our “robots.txt” file allows any search engine spider (from Google, Bing, AOL, Yahoo etc.) to crawl every webpage of our site and use the pages in the search engine results.  Also the file importantly tells spiders where to find a website’s XML Sitemap; this is a list of all the URL’s on a site and makes sure no page is missed out when added to search engine indexes.  The XML Sitemap also notifies search engines of any new webpages that have been created, if you have a WordPress blog then make sure you have an XML Sitemap generator plugin installed on your blog for this purpose.

Another feature of robots.txt is to tell search engine spiders which pages to ignore, or at least skip out in the search engine results.  This is the same as adding the “NOINDEX” attribute in a meta tag.  Examples of using this could be on duplicate pages, pages with sensitive information such as names/addresses or sales pages full of repeated keywords.

Unfortunately it’s rather easy to set robots.txt to not index the entire website!
Just changing the line of code “Disallow:” to “Disallow: /” is all it takes to bring a whole website out of Google and all the other search engines.  This effect takes about 1-2 weeks but it has devastated many sites in the past, sometimes even deliberately.

WordPress blogs also have a habit of setting the NOINDEX attribute on sometimes when installed.  Check that WordPress isn’t blocking your website from search engines by going into the settings pictured:

If you don’t know what to do with a robots.txt file then it’s best to leave it altogether, Googlebot and other search engine spiders will happily go through the whole site if they don’t find it.  You cannot ensure that no web spider will crawl every single page of your website even when it’s completely blocked by robots.txt, if you have information that you do not wish to be seen then use passwords or don’t publish it at all!

Without really going hard on SEO it’s impossible to achieve number 1 rankings without an amazing (and usually expensive) PR or viral effort.  You can be too scared of Google’s changes and never reach the extremely lucrative top positions so what’s the best option?

Here at Real Web SEO we do a mixture of SEO techniques, some slightly risky, some safe and back it all up with great unique content on a website’s blog.  If Google shift the goal posts and say that current SEO techniques are bad for it’s rankings you can remove that risk by doing safe techniques alongside the current work such as participating in social networks.

Again it’s the analogy of putting all your eggs into one basket, if you just do one SEO technique that currently works well you could find yourself in a sticky situation if it all goes to pot!

Synonyms are words that relate to other words; these related words can sometimes be inter-changeable.  Google loves related content but can spot an obviously keyword stuffed webpage which keeps mentioning the same word again and again and again in an un-natural fashion.

You can use synonyms to make sure a webpage is targeted around your main keyword without flagging up the page for any spam reports.

Firstly open up a Thesaurus and enter your main keyword, you should hopefully get a bunch of synonyms this way.

For example:

Let’s say my main keyword is “waterproof“.  Synonyms would include “air tight”, “impermeable”, “impervious”, “rainproof”, “sealed”, “tight” and  “watertight”.  So instead of repeatedly mentioning “waterproof trousers” for instance I could say “rainproof trousers”, “sealed trousers” or even “rainproof trousers” to mix up the keywords a little whilst keeping things relevant!

So increasing your vocabulary can help a lot with content writing for SEO.  Writing content can be a chore but the results speak for themselves; you can also gain more traffic through the variations of your main keyword which is another little bonus too.

Forums can also be a curse on a website when members start chatting about un-related subjects.  Too much chitter-chatter on the latest news or gossip starts mixing up the subject matter on a forum, if the forum’s main subject was “fishing” and all people talked about was the latest World Cup news then the forum is neither fully fishing related or fully football related and this will damage SEO.

If you’re whole website is based on one subject then it’ll rank well for that subject, if it’s based on two or more unrelated subjects then it’ll rank badly for each one

So how can you get around this potential nightmare, you cannot ban your forum users from talking rubbish!?

The answer is to put your forum on a subdomain such as http://forums.moneysavingexpert.com instead of within a folder such as http://www.driftworks.com/forum/.  A subdomain (the part before the domain name) is considered as a completely different website to search engines so it’s a perfect place to put un-related content.  Subdomains are used on large sites for different languages, mail servers and apps, make sure to use one on your forum but NOT on your blog, that should be all about your main niche and will help your SEO effort no end.