There might be many reasons why your site does not appear in search engine results. One reason of that is to keep same content in multiple URL’s.

There are several reasons why search engines don’t like duplicate content. One is that to keep quality in results. They don’t like to show same pages on their search results. Another reason that they don’t want to use theirs resources to index pages those has similar content.

WordPress is a very powerful product, but it is not very search engine friendly. If you are a WordPress user, there will be somethings to do in order to avoid duplicate content problem. When you write an article, your content goes to be published in your index file and single post page as well as category and archive pages.

What you can do to get rid of this problem is to tell to search engines what URL’s to be indexed and what are not with noindex, follow and index,follow meta tags or to use robot.txt file. If you are not very good in creating robots.txt files, you can instead use this code to tell to Googlebot not to index everything.

<?php if(is_home() || is_single() || is_page()){
echo ‘<meta name=”robots” content=”index,follow”>’;
} else {
echo ‘<meta name=”robots” content=”noindex,follow”>’;
} ?>

Just simply copy and paste this code into your header.php template file.

Oleg has very detailed and great post about this issue. Worth to check it out.