How to prevent google spider

Hi guys,

Does anybody how i can stop google from indexing profile pages?

I am sticking echo '<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">'; in profiles.php but it is either standing at the very top giving warnings or at the very bottom after html.

Do you know what file is it that does the <meta tags for pages??

thanks

Admin > Settings > Meta Tags  ..... 

right but that is description and keywords. besides it applies to all pages. i simply want to exclude profiles.php from search engines listings...

robots.txt

http://www.robotstxt.org/

Block everything (robots.txt):

User-agent: *
Disallow: /

Plus try here:

http://www.google.com/support/webmasters/bin/answer.py?answer=61062&cbid=-1ssgm74wvjdpy&src=cb&lev=answer

Also see recommended article links on that same page.

Thanks guys for replies. Correct me if i am wrong but:

 i have considered the robot.txt method before. However what google spider follows are links such as "www.sitename.com/nickname" which are listed in the new members block in the front page. Therefore i dont think google bot will notice that the file is profiles.php.

Excluding profile images wont exclude the the link from search engines. Excluding search.php and browse.php wont make a change because search engine bots dont fill in forms.

Getting google to remove links one by one will take forever although there are only several members' profiles listed as site grows.

I dont want to block the whole site either.

Hmmm.  Not sure I would agree with Sammie's poor advice comment.  I think she is pissed at me and just owes me a stinger.

Those who are hooked on phonics will notice I specifically state that the robots.txt file will Block All bot traffic, and if you actually follow the links (which were also part of my comments), it will give you several examples that will not block ALL bot scanning!

Feel free to make your own judgements on whether or not that is poor advice.

How do I use a robots.txt file to control access to my site? .....

.

I'm still researching this problem..

but for a little light relief, if you have FF3 type "about:robots" in your address bar.. (no quotes)

enjoy

Well, short of actually placing each member profile into the robots.txt file manually, there doesn't seem to be any way to block spiders from indexing the profile pages.

"<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">" is the correct tag to place in the header but I don't know how to make Dolphin place it into the head of the profile pages. Ideally it needs to be added alongside the site meta tags (description/keywords) at the same time they are but I'm not sure if it's profiles.php that does that or BxDolProfiles.php (or some other??).

Are the meta tags placed in the head by some other script, and if so, is it possible to add another rule where the above tag is added if it's a profile page that's being created?

Create a file robots.txt at the site root. with following contents.

User-agent: *
Disallow: /

This will keep all bots away.

If you use google bot for user agent, google won't crawl your site.

Someone from Boonex care to answer this problem as no-one seems to have posted a real solution here...

I mean to stop the robots from listing the profile pages?

Thx Dan V

Topic is 2.5 years old, but searching the forums hasn't resulted in an answer to the question: How to prevent Google from indexing profiles?

• Will placing profile.php in robots.txt do the trick?
• Or does string below have to be placed in the header:
  <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
  • If so, how is this done?

Don't have New Members block on home page so perhaps helps limit exposure, but there are other pages with Browse Member blocks on them, for example http://boonex.us/browse.php.

Thanks!