Weird entries in my robot.txt

Rigo

Rigo

Active Member
Reaction score
141
Location
Australia
Hi folks,
Don't know how these entries got into my robot.txt
Though I don't understand much of these, some of these appear suspicious to me. Note my site www.cmss.com.au - is simply for my IT support business, no advertising, no connection with anyone else. Anything on the site entered by me or the developer at the time of setting it up. Should I disallow them all instead?
Thanks for any guidance/explanation .

User-agent: *
Allow: /_/aihub_pod/
Allow: /_/boq/
Allow: /_/boq-play/
Allow: /_/compare-banking/
Allow: /_/compare-insurance/
Allow: /_/fonts/
Allow: /_/mss/boq-geo/
Allow: /_/pantheon/
Allow: /_/play/
Allow: /_/pog/
Allow: /_/cloudsourcetools/
Allow: /_/devopsconsole/
Allow: /_/transparencyreport/
Allow: /_/gstore/
Allow: /_/ain/_/js/
Allow: /ac/
Allow: /actions/news/rss/
Allow: /ads/learn/
Allow: /ads/local/
Allow: /ads/gpa/vidgen/
Allow: /ads/research/
Allow: /ads/ubaq/
Allow: /android/market/
Allow: /android/market_images/
Allow: /audio_news/
Allow: /bfe/apps/website/
Allow: /compare/
Allow: /chrome/crlset/
Allow: /cloud-launcher/sitemap/
Allow: /cloudsourcetools/images/
Allow: /commercesearch/images/
Allow: /communities/sitemap/
Allow: /culturalinstitute/images/
Allow: /culturalinstitute/sitemaps/
Allow: /devopsconsole/images/
Allow: /devrel-devsite/
Allow: /docs/forms/social/
Allow: /docs/socialmarkup/
Allow: /dictionary/static/sitemaps/
Allow: /droidguard/
Allow: /earth/social/
Allow: /enterprise-partner-search/sitemaps/
Allow: /external_hosted/
Allow: /flights/
Allow: /foodordering/
Allow: /frommers/
Allow: /gb/images/
Allow: /glass/images/commerce/
Allow: /gsx/
Allow: /gtp/stir/
Allow: /GoogleInternetAuthority/
Allow: /healthricherkp/illustrations/
Allow: /healthricherkp/cropped/
Allow: /history/static/
Allow: /identity/accountsettingsactivityweb/
Allow: /identity/accountsettingssyaf/
Allow: /images
Allow: /inproduct_help/chatsupport/
Allow: /local/donut/
Allow: /mapspro/
Allow: /moderator/static/ux/
Allow: /og/
Allow: /onebox/media/sports/photos/soccer/wc2018/
Allow: /patent-search/frontend/
Allow: /pay/images/merchant_logos/
Allow: /pay/images/program_logos/
Allow: /paygo/
Allow: /play/store/web/
Allow: /podcasts/test-podcast/rss/
Allow: /politics/e/sharing/
Allow: /prettyearth/assets/
Allow: /psb/udc/
Allow: /s2/sitemaps/
Allow: /search-console/scfe/
Allow: /sites/
Allow: /sitemaps/
Allow: /social/photosui/images/
Allow: /social/photosui/sitemap/
Allow: /support/content/
Allow: /travel-vacations/
Allow: /touringbird/
Allow: /trends/
Allow: /trends_tpt/
Allow: /trustedstores/images/
Allow: /trustedstores/js/
Allow: /tv/thumb/
Allow: /ui/
Allow: /vr/*
Allow: /youtube/fonts/
Allow: /youtube/img/promos/
Allow: /icons/
Allow: /js/
Allow: /ss/
Allow: /*.js
Allow: /*.css
Allow: /*.svg

Disallow: /
 
I see it's a WordPress site. Do you have any anti-virus/anti malware plug-ins uninstalled? WordFence, Bullet Proof Security, Sucuri - that sort of thing? Some of those write entries in robots.txt, as well as your .htaccess file.
 
seashore said:
That's spam.
But you should contact your webdev before deciding exactly which entries to remove.
Click to expand...

Yeah, that what I thought when I saw these.
Last time I had a look at my robot.txt about a couple of months ago, there was nothing listed in it. Wonder how these entries got in? Are these open to anyone to edit?
 
Mick said:
Do you have any anti-virus/anti malware plug-ins uninstalled?
Click to expand...

Yeah, it's Wordpress. Haven't uninstalled any security plugin lately. Using WP Cerber right now as suggested a little while ago when I was looking for some protection. Pretty good one, the stuff it's been blocking, wonder how my site hadn't been breached beforehand .
I've just removed a lazy load plugin that actually made my Pagespeed insights results worse A3 Lazy Load. Hadn't had a look at my robot.txt file for a while but the last time I did, there was nothing in it. Don't know whether that lazy load plugin would've added these entries?
Another worry would be if anyone could just edit this one externally?
 
Now the plot thickens ☹
When I go to: www.cmss.com.au/robots.txt this what I get:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Yeah I've changed the login URL as advised by WP Cerber.
Search Console gives me a scrollable list of entries, where did those come from?

I do have yoast seo premium, and when I check through it, the plugin says I've got no robots.txt ¯\_(ツ)_/¯
 
Last edited:
robots.txt should have a permission set of 644, meaning that it is not 'publicly' writable. But any plug-in you install (or that somehow gets installed) will be able to write to it because 'you' are its owner.

FWIW, robots.txt would normally have an entry along these lines - these may vary a bit from site to site but this is a pretty standard example:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-content/uploads/
Disallow: /wp-content/files
 
Mick said:
robots.txt should have a permission set of 644, meaning that it is not 'publicly' writable. But any plug-in you install (or that somehow gets installed) will be able to write to it because 'you' are its owner.

FWIW, robots.txt would normally have an entry along these lines - these may vary a bit from site to site but this is a pretty standard example:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-content/uploads/
Disallow: /wp-content/files
Click to expand...

Thanks Mick,
Adjusted as suggested,
Great help.
I think I ended up working out these odds entries were connected to a link I had setup to display my pics library from a service referral site on my front page.
Another problem that had also created was messing up my mobile friendliness with the display no longer rendering correctly ~ another warning from GSC.
Thought I was only displaying pics but never trust anyone I found out.
Thanks folks for the help
 
You must log in or register to reply here.
Back
Top