Six steps you can take to help search engines find and index your content

You’re trying to get your website noticed and search engines are an excellent channel for visibility. In this article, we’ll look at some basic things you can do to make sure your content is being indexed by search engines. The more of the items you can implement below, the more noticeable your site will be to the search engines. 

1. Check your “Robots.TXT” file
One little line of code in a simple txt file can be very costly if it’s blocking your site from search engines. A robots.txt file allows you to tell search engines to crawl or not crawl certain directories or files in your site.

For example, this simple line of code disallows all crawlers (a.k.a. robots):

User-agent: *
Disallow: /
To find your robots.txt file simply type in your domain followed by /robots.txt. For example, here is the robots.txt file for the W3C: www.w3.org/robots.txt. For more details on what can be in this file and how search engines treat it visit robotstxt.org.

2. Make sure your content is indexable
Although search engines have come a long way over the years in terms of indexing all kinds of content on the web, there are still some types of content that may not be fully indexed or not indexed at all. If you are seeing missing content when you look at the search engine’s cache of your page, you may want to check if the content is presented in one of the tough-to-index ways below.

This is also true for people using screen readers due to a disability. A screen reader “sees” the page much in the same way a search engine crawler does – by crawling content and deciphering the elements.

JavaScript
Some JavaScript is crawled by search engines today and more will likely be in the future, but JavaScript (including its related scriping technique AJAX) can still present an issue.  Because most content in JavaScript is usually not indexed, things like navigation, on-page apps, and any other content presented by using JavaScript may not be seen and therefore cannot contribute to the context of the page (or sometimes cannot be followed, in the case of links) for search engines.

Flash
With better indexing capabilities coming about recently, Flash sites are becoming more prevalent in SERPS (Search Engine Result Pages), although a site built entirely in Flash is still probably not the best idea if you care about search engine traffic.

Today search engines primarily attempt to index links and text from Flash files. While this is better than it used to be, 100 percent of content still may not be indexed depending on how your Flash site is created. Navigation through “pages” in a Flash file is all contained within a single swf file that lives on one URL, eliminating separate topical content for separate pages.  This can be problematic when you’re up against competitors with much more targeted topical and sub-topical content living on distinct URLs (with links to each of those specific URLs providing even more context).

To minimize indexing difficulties, try to use Flash in smaller pieces. Make sure each topical page of your site has its own unique URL first, then put Flash elements on each page if you like.  Beware though—the more of your content you put in Flash, the less content and context you may be providing to the search engines. 

Image Text
It’s rare to see the entire content of a page posted as a .jpg or other image these days, but it still happens. And when it does, a crawler goes through the code and just sees an image instead of seeing all of the pictures, content, and link text on the page. Search engines simply cannot read any textual content you present in an image, whether it is the entire content of the page or just titles or headers.  You will see the images displayed when you check the cached version of a page.  This is because it is displaying the actual image that is cached, not reading the text content within it.

3. Strenghten your link structure
Links to and from your pages are very important for the “findability” of your pages.  If a page has no links connecting it with any other indexed pages on the web, it may not be found by search engines, since they follow links to discover new content. 

Internal links
Make sure you have a sensible linking structure in place on your site that is crawlable, links to top level as well as deeper level pages, and links to content relevant to the page the links are on.

Crawlable links are links that can be seen by search engines, meaning they’re not in JavaScript or in unindexable links within a Flash file. Also link to different pages within your site, not just from the home page, but all pages. Deeper pages in a site tend to be tougher to find and index, since they are linked to less often, or from more obscure pages in a site. Try to include links to pages most relevant to the content of each page, to give the search engines better context, and to provide a good mix of deeper links. 

You can also include a sitemap page on the site (similarly named xml sitemap files are discussed later).  Provide the sitemap link from your home page and/or from a header or footer on all pages.

External links
If you provide worthwhile content, your site and the pages within it will attract links naturally.  These links from external sites help search engines find and classify your site, especially if your site is newly published. To kick-start your visibility,you can add your site to  trusted directories like The Open Directory Project and Yahoo! Directory.  If it is relevant, you can also add your site to online local listings pages like Yelp, Yahoo! Local or CitySearch. 

Promote your website in your advertising campaigns, add it to your business card, and provide any other means for visibility that you can. If people find your site interesting and useful they will link to it.

To see what your inlinks looks like, go to https://siteexplorer.search.yahoo.com/ and type in your URL. Click on the Inlinks button.  Use the dropdowns to look at links to one page or the entire site, or to look at links from all pages, all pages except that subdomain, or all pages except that domain.

4. Create a  sitemap XML file
The major search engines we’re addressing here all support xml sitemap files. These sitemap files are different from the onsite sitemap pages previously described. They are xml files that contain a list of the URLs on your site along with a small amount of information about the URLs that is placed on your server and crawled by search engines. This allows you to tell search engines about your URLs, even if they haven’t crawled them naturally by following links on the Web.

Visit sitemaps.org for more information, or see Yahoo!, Google, and Bing’s support of sitemaps.

5. Verify your “nofollow” and “noindex” tags
Noindex and nofollow tags can be used to block search engines from crawling specific links or content. 

Noindex
The noindex meta tag tells search engines not to index a page.  It looks like this:

<meta name=”robots” content=”noindex” />

To check for noindex tags on any of your pages, right click on the page in the browser and choose “View Source”.  Search for noindex on the page.

For more information on the search engines’ support of noindex, see these Yahoo!, Google and Bing pages mentioning it.

Nofollow
Nofollow tags can be found in a robots meta tag at the page level, or within the <a> tag at the link level.

Nofollow at the page level tells search engine robots not to follow any of the links in the body of the page that the nofollow meta tag is on.  It looks like this:

<meta name=”robots” content=”nofollow” />

Nofollow at the link level tells search engine robots not to follow that particular link that the nofollow attribute is applied to.  It looks like this:

<a href=”http://www.example.com/” rel=”nofollow”>link text</a>

To check for nofollows on any page, you can look at the source code of the page by right clicking on the page and choosing “View Source.” Then do a search for the word nofollow in the source code.

For more information on nofollows, see this Wikipedia article, or see Yahoo!,  Google and Bing’s  support of nofollow.

6. Specify your site’s language
You can also help search engines by specifying what language your site is written in. This is a simple meta tag that looks like this:

<meta http-equiv=”content-language” content=”en”>

See all ISO codes at the Library of Congress site for more information.

To check for language meta tags on any page, you can look at the source code of the page by right clicking on the page and choosing “View Source.” Then do a search for the word language (or content-language) in the source code.

Still having problems?
If you’ve tried everything above and believe you still have indexing issues, browse the webmaster guidelines below for more information, troubleshooting, and contact information for the search engines. 

Search Engine Guidelines for webmasters

• Yahoo! Search Content Quality Guidelines
• Google Webmaster Guidelines
• Bing guidelines for successful indexing

For more on getting your site noticed, refer to Laura Lippay’s previous post “Is Your Site Invisible?”

— Laura Lippay, Director of Technical Marketing

(Image by Kapungo via Flicker, CC 2.0)

The Bing Webmaster Center team has been very busy lately, working on very cool stuff that we can't wait to share with you (patience, Grasshopper - all will be revealed in time). But the blog waits for no one (well, that's the intent, anyway). From time to time, we gather up enough interesting tidbits of Q&A that we want to share with all of our blog readers. Now it's that time again. So let's get to it.

Q: I'm not able to gain access to Webmaster Center with the authentication code used in a <meta> tag. Can you help?

A: The Webmaster Center online Help topic Authenticate your website recommends using a <meta> tag formed as follows:

<meta name="msvalidate.01" content="0123456789ABCDEF0123456789ABCDEF" />

However, some users attempt to combine the flow of authentication codes for multiple sites in the <meta> tags. If you must use the <meta> tag method of authentication (as opposed to the XML file authentication method as described in the Help topic), we recommend placing your Bing Webmaster Center authentication code last so that it is not followed by a space. In addition, Webmaster Center does look for the proper XHTML-based closing of the <meta> tag - the " />", so be sure to use this closing in your code.

This issue is discussed further in the Webmaster Center forum topic Site Verification Error for Bing Webmasters Tools.

Q: Why do I have to register as a user in the Webmaster Center blog just to post a comment?

A: We were getting a few non-registered visitors who were posting way too much spam in the blog comments. We needed to block that junk from being posted, so we implemented a new rule that requires folks to register before they can leave comments. Since we can control spam from registered user accounts, we felt this was the best course for minimizing the disruption of irrelevant comments. We hope this is not a hardship on anyone!

Q: I've posted two random blog comments requesting inclusion of my site into Bing News Service? Why haven't you added my site?

A: Let's redirect those requests to the right place. To request that Bing add your news site to our list of news sources, we ask that you send the request via email to the Bing News Service team. Please be sure to identify yourself, your URL, what types of news you provide, your audience, and any other determining factors such as awards won, etc.

Q: I have a very complicated or specific question to ask about my site and the Bing index. Can you answer it here?

A: Blog comments are best used for furthering the conversation about the associated blog article. Specialized service requests or specific questions about Bing products and services requiring detailed, individualized answers are always better left in the Bing Webmaster Center forums as a starting place. We have a forums administrator on staff who, along with the regular VIP contributors there, can offer helpful advice and insight to your questions. There are some amazing folks participating over there!

Q: How do I get my company listed in the Bing local listings?

A: Use the Bing Local Listing Center form. You may need to sign in to your Webmaster Center account or create a new sign-in account to access this form.

Q: How can I ensure that my local business contact information (address and phone number) from my website get into the Bing index?

A: One common problem we see with this is that some sites rely solely upon an image containing text to convey this information. This is not good practice for SEO. If you want to be sure MSNBot (or any other search engine bot) can to read such information, please add it to your website as text (the image is OK as long as the text version also exists)!

Q: Your recent posts on web spam has brought up a question: how do I report web spam that I find in search engine results pages to Bing?

A: To report web spam sites, we recommend that you go to the Bing Support web form to file the complaint. In the Problem list, select Content Removal Request. In the resulting list box, select Other. In the comments text box, include specific and detailed information in your report. Complete the rest of the form and then click Submit.

A member of the Bing web spam team will review the report and investigate the matter. If the report is accurate, appropriate action will be taken. Note that if the report is malicious and false, no action will be taken against the accused site.

Q: My website offers tax-related services. As a result, I use the word "tax" numerous times in my content. Could Bing consider my site to be web spam due to the appearance of keyword stuffing? When do I cross the line from acceptable to web spam?

A: The key here always comes back to how the content appears to the human reader. Is it logical? Is it readable? Does it make sense? In this particular case, the repeated use of the word "tax" in content regarding tax services offered is reasonably expected and thus is fine. In fact, including a solid set of explanatory content that defines these keyword phrases only strengthens the case for reasonably repeating this word. If the use of this repeated word makes contextual sense to the reader and is not a clumsy attempt to stuff the word in where it's not necessary or helpful, and you have a good amount of supporting content to accompany it, you'll be fine. Our crawler sees this usage and understands it is legitimate. Just write your content for the reader's comprehension and the crawler will not penalize you for keyword stuffing.

The important thing to remember is that true web spam often involves multiple issue violations. As such, it typically takes more than one violation to trigger web spam consequences - having a slightly above average number of keywords won't automatically torpedo your site. Just as you need to do several things well to improve your ranking (build good content, build valuable inbound links, target several keywords, etc.), you need to do several things wrong to really hurt your ranking. That said, if it's obvious that you are trying to abuse the system, even with just one egregious issue, then penalties will ensue.

Lastly, we don't define any borderline between acceptable and non-acceptable web spam. If you think what you've done might be considered web spam because you know you're trying to game the system, then take a different approach to optimizing your pages. I'll repeat my mantra: write content for the human reader, not the crawler. Develop good, unique content that is readable, understandable, and valuable. If you do this without involving any black-hat, SEO-style trickery in an effort to artificially boost your ranking, then you'll never have to worry about this being an issue.

Q: Regarding backlinks in forum comments and link-level web spam, is it only a problem when the page linked to is not relevant to the conversation in the forum, or is this a problem for all backlinks?

A: It always comes down to whether the effort is intended to legitimately benefit the human reader or benefit the owner of the link. If the link in a blog comment is relevant to the content in both the blog article and the blog comment and as an extension to that content, is of value and interest to the reader, then it is not a problem. In fact, this is a fine idea (whether or not the rel="nofollow" attribute is automatically applied by the blog to user-generated links). However, if the link in the blog comment is not relevant to either the blog article or the blog comment's content, is not of relevant, legitimate interest to the reader, and instead is only beneficial to the link owner, then that is web spam. It's pretty straight-forward.

Also consider how the blog comment link is formed, as in whether it is a single link inline to the comment's content or is it a bazooka blast consisting of multiple, irrelevant links following a short, generic message that could be applicable to anything (or nothing). If your goal is to tell the reader about some information relevant to the post and that info is found within good content on your site, that's great. Add those links! Even if rel="nofollow" is employed by the blog in all UGC-based links, the potential for driving live traffic to your site is good, and if the content there is worthwhile, that will improve public awareness of that content and ultimately be a good link building strategy. But if the comment is merely an excuse for blatant advertising links, it is web spam. Note the difference in intent. If you do right by the reader, you'll be fine.

If you have any questions, comments, or suggestions, feel free to post them in our SEM forum. Later...

-- Rick DeJarnette, Bing Webmaster Center

Here is a recap of what happened in the search forums today, through the eyes of the Search Engine Roundtable and other search forums on the web....

Yesterday, I posted Video Marketing Case Studies with David Meerman Scott and Me. At the end of the post, I said, "If you have any video marketing case studies that you'd like to share with us, just mention them." Silly me.

Click to read the rest of this post...

I have been enrolled in the Market Motive social media online course for close to two months now.  Each week I have been given a social media topic and have been responsible for viewing a certain number of films which go into detail about the topic being taught.  There have been a wide range of social media topics such as a week dedicated to Twitter and another dedicated to Youtube.

Personally, going into the course I thought I was pretty familiar with the social media areas I was anticipating studying. I was excited to learn more of course, but didn’t have the highest of hopes. I honestly thought it would be nice to have a recap on the social areas I work with daily and after a few months of reiteration, I would obtain a certification to show off my knowledge.

I have to say my original thoughts were narrow minded. I have been continuously surprised at the insight I am gaining each week.  Until now, I have been responsible for watching videos each week and partaking in conference calls with the faculty.(The course takes a turn now and the responsibilities change but I will write about all of that in a few days). The videos have each had such detail. They are unique to the faculty members who make the videos and specialize in the areas they are teaching.

Video Topics	A small amount of the material covered
Blogging for Business	Permalinks, RSS feeds, key word longtail , pitching check list
Video & Youtube	Popular categories, channel, market share
Viral Marketing	Brand, conversion, campaign styles, viral themes
Social News Networks	Do’s and don’ts across various networks, strategies to becoming more successful on each
Twitter	Appropriate business decisions, tools, connecting on and offline, tracking
Flickr	Titles, tags, descriptions, search engines and Flickr
Facebook	How to utilize Facebook for your business with Facebook connect and ads
LinkedIn	Advertising on LinkedIn, groups
Online Press Releases	Wire Services, measurement, Keyword positioning

 

Market Motive resources that have been provided to me and that have impressed me throughout each week’s video viewings:

1) Examples of useful books to buy pertaining to the areas being taught that week which are based on social media current events and practices.

2) Informative commentary (humorous when applicable and serious when appropriate)

• Current
• Authoritative
• Timely

3) Graphs

4) Tools that the faculty use and recommend

• Seo plug ins
• Twitter tools
• Better headlines
• Tracking your Brand

5) Viral Marketing Brainstorming Checklist

6) Pitching check list

Brief Video information for each week:

1)  Length of each video is provided

2)  2-7 videos necessary to watch each week

3)  Average 25 minutes per video

• Viewings do not have a specific deadline but rather anticipated due dates which are flexible. They are listed as an ideal due date but it is up the student to get things done in a timely manner. Weekly meetings with the faculty help ensure that students are on track and are not missing any points.

4)  One Quiz to follow all videos combined for the week

• The quiz each week is just for practice. A grade is assigned but does not count toward the overall grade. They are meant as preparation for the final exam and reiterate the material in the videos. The quizzes help make sure the students have understood what is being portrayed in the videos.

Check out the SEO Tools guide at Search Engine Journal.

I Admit It – These Videos Taught Me a Few Social Media Tricks

Users of the mobile version of Google Product Search can now see if items are in stock at a local store from select retailers. These retailers include: Best Buy, Sears, Williams-Sonoma, Pottery Barn, and West Elm.

Product listings from these retailers will have blue dots that can be tapped to see an "in stock nearby" link, which takes you to a seller's page where it will either say "in stock" or "limited availability". In addition, these pages will show you how far away the store is from your location (assuming you have My Location enabled or have manually specified your location).

The feature is available for iPhone, Palm WebOS, and Android users in the U.S. Users can hit the "more" link, and select "Shopping" or find the "Shopping results" section in Google's Universal Search results when searching from Google.com.

Google offers a form for retailers who are interested in participating in this program. Google says to make sure you have your Local Business Center data up to date and to ensure your Product Search data is in "great shape."

In the range of thigs discussed at Google’s Search Evolution event in December Engineering VP Vic Gundotra demonstrated mobile shopping with real-time inventory information.  Now Google has announced that product is live. Google is working with a limited number of retailers for the time being but is inviting others to apply to become part of the [...]

*** Read the full post by clicking on the headline above or, in Facebook, by clicking on the "View Original Post" link below. ***

What sites send Bing the most traffic? Here’s a surprise. According to Hitwise, Google is Bing’s fourth largest referring source. Below are the top five sites that drive traffic to Bing, or “upstream” to Bing, as Hitwise calls it: MSN is by far the top source, the last site visited by 42.7% of Bing users before they [...]

*** Read the full post by clicking on the headline above or, in Facebook, by clicking on the "View Original Post" link below. ***