Evan Martin ([info]evan) wrote in [info]lj_dev,
@ 2005-09-14 10:54:00
Previous Entry  Add to memories!  Tell a Friend!  Next Entry
feeds don't block robots
Now that blog search is getting some new press, I expect a lot more users to be unhappy about bug 2232.

(Post a new comment)


[info]thehumangame
2005-09-14 05:57 pm UTC (link)
I dunno... my journal doesn't seem to be indexed, when my friends' are, and I have bot blocking enabled and they don't. I wonder if blog search pays attention to the robots.txt on the journal itself...

(Reply to this) (Thread)

[info]thehumangame
2005-09-14 05:58 pm UTC (link)
Meh, scratch that. It is indexed, just sporadically so. I guess this service isn't really ready yet.

(Reply to this) (Parent)

[info]marksmith
2005-09-14 06:04 pm UTC (link)
People can use the syn console function that turns off full content.

If it's public, it will be indexed, by someone who respects the rules (there are really no "no bots" headers for RSS/ATOM, as far as I know). And people who don't respect the rules will index anyway.

(Reply to this) (Thread)

[info]flying_squirrel
2005-09-14 09:13 pm UTC (link)
Is this "syn function" undocumented, or am I looking in the wrong place?

In other words, wha?

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]ruakh, 2005-09-14 10:42 pm UTC (Expand)
(no subject) - [info]kate_nepveu, 2005-09-15 01:03 am UTC (Expand)
(no subject) - [info]flying_squirrel, 2005-09-15 03:53 am UTC (Expand)
(no subject) - [info]kate_nepveu, 2005-09-15 11:00 am UTC (Expand)
(no subject) - [info]ruakh, 2005-09-15 04:40 am UTC (Expand)
(no subject) - [info]njyoder, 2005-09-16 09:18 pm UTC (Expand)

[info]feignedapathy
2005-09-14 06:07 pm UTC (link)
Yeah, noticed this. Whoops.

(Reply to this)


[info]xfyre
2005-09-14 06:12 pm UTC (link)
And they really are, in Russian segment also.

(Reply to this)


[info]omnifarious
2005-09-14 06:18 pm UTC (link)

Allowing people to turn off syndication is good. Allowing people to block bots is good. Pretending that they meant to turn off syndication when they ask to block bots is very bad.

Some sort of no-robot extension needs to be added to RSS and Atom in order to properly fix this bug. And I'm totally against any fix that pretends that bot blocking means no syndication. That's not a fix, that's purposely adding a bug.

Now there is a fix I would accept, though it would be kind of silly and poor. That would be to check the User-Agent field for known bots. LJ would then refuse to hand them the feed by returning the appropriate 'access denied' HTTP header.

(Reply to this) (Thread)

[info]anildash
2005-09-15 04:52 am UTC (link)
"Some sort of no-robot extension needs to be added to RSS and Atom in order to properly fix this bug."

...and then deployed on feeds other than just LJ's. And reconciled between Atom and RSS. And then search engines need to be updated to honor the spec. And all the other user agents have to honor the spec, too.

This is nightmare, boil-the-ocean type stuff. Of course, I'm biased because I'd have to be spending time each day evangelizing it. :)

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]omnifarious, 2005-09-15 05:05 am UTC (Expand)
(no subject) - [info]jamesd, 2005-09-15 05:13 pm UTC (Expand)

[info]thehumangame
2005-09-14 06:25 pm UTC (link)
After a bit of investigation, I found that Google Blogsearch implies that it will respect robots.txt.

I guess it just needs to be turned on for feeds?

(Reply to this) (Thread)
/robots.txt
[info]pne
2005-09-14 06:42 pm UTC (link)
Remember that there can only be one /robots.txt URL per hostname -- so for each user for whom you wanted to block http://www.livejournal.com/users/examplusername/data/rss, you'd need one line in the global LiveJournal robots.txt. Clearly not feasible given the hundreds of thousands of active accounts.

(Reply to this) (Parent)(Thread)(Expand)
Re: /robots.txt - [info]thehumangame, 2005-09-14 06:50 pm UTC (Expand)
Re: /robots.txt - [info]mart, 2005-09-14 06:59 pm UTC (Expand)

[info]mendel
2005-09-15 02:42 am UTC (link)
Apples and oranges -- the problem isn't a question of permitting or forbidding them to read particular files. Google is (presumably) pulling data from the all-posts feed, not from individual journals. One data source, one filename, one big on-off switch.

I think that level of control is going to have to come from this end, letting people include or exclude themselves from the all-posts feed. (The console feature discussed above may in fact do that, but I want to be indexed so I am not sure.)

(Reply to this) (Parent)

[info]chris21718
2005-09-14 06:29 pm UTC (link)
There is a post indexed in Google's new service that I have marked as private, although it was public for a very short time. It's important to keep in mind that any post that has ever been made public can be indexed forever, regardless of the security settings put on the post in the future.

(Reply to this)


[info]wtf
2005-09-14 06:33 pm UTC (link)
Indeed. All my public entries (most of which are complete and utter nonsense) are listed, despite my having the 'block robots' function enabled. This makes me, well...less than happy.

(Reply to this)


[info]purly
2005-09-14 06:44 pm UTC (link)
not cool.

(Reply to this)


[info]mart
2005-09-14 06:56 pm UTC (link)

This is a little off-topic, but why to the Blog Search result links take you to a page that then does a refresh to the real page? Some inspection shows that the Location: header is there but it's being served as HTTP status code 200 rather than 301 or 302 so my browser is ignoring it. If it's sent with 301 or 302 (or one of the other, similar 3xx response codes) it'll be much more useful to user-agents which aren't browsers as well as being less lame in those that are.

(Reply to this)


[info]octal
2005-09-14 07:09 pm UTC (link)
Count me among the users who are really annoyed at this.

friends-only seems like the most viable solution.

Blocking RSS entirely seems excessive just to keep out spiders. I like rss, and am not a spider.

Some kind of meta tag semantic in rss, or making rss accessible only via username.livejournal.com/rss which could then be protected by username.livejournal.com/ROBOTS.TXT, seems like the only solution which is really decent in general though.

(Reply to this) (Thread)

[info]mart
2005-09-14 07:21 pm UTC (link)

Event that solution isn't brilliant, because not all LiveJournal-based sites actually uses the user vanity domain functionality. It would effectively disable the machine-readable data completely on those sites which don't.

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]octal, 2005-09-14 07:46 pm UTC (Expand)
(no subject) - [info]heydusty, 2005-09-14 09:38 pm UTC (Expand)
(no subject) - [info]anildash, 2005-09-15 04:54 am UTC (Expand)
(no subject) - [info]omnifarious, 2005-09-15 05:10 am UTC (Expand)
(no subject) - [info]jamesd, 2005-09-15 05:32 am UTC (Expand)
(no subject) - [info]evan, 2005-09-15 06:27 am UTC (Expand)
(no subject) - [info]octal, 2005-09-15 08:07 pm UTC (Expand)
(no subject) - [info]jamesd, 2005-09-16 12:14 am UTC (Expand)

[info]ladysorka
2005-09-14 07:11 pm UTC (link)
I have both bot block enabled and my feed set to title only, and it still managed to grab a few of my entries - though, notably, nothing after last May, which I believe was when the synlevel option was added to the admin console.

(Reply to this)


[info]rahaeli
2005-09-14 08:55 pm UTC (link)
It's really, really slimy of Google to do this before there is an acceptable, agreed-upon convention for RSS bot-blocking. Please feel free to let whomever made this decision know that I've got a lot of very unhappy people over here who'd like to talk to him or her. :/

(Reply to this) (Thread)(Expand)

[info]evan
2005-09-14 09:03 pm UTC (link)
technorati.com, icerocket.com, feedster.com

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]octal, 2005-09-14 09:07 pm UTC (Expand)
(no subject) - [info]mart, 2005-09-14 09:11 pm UTC (Expand)
(no subject) - [info]octal, 2005-09-14 09:14 pm UTC (Expand)
(no subject) - [info]midendian, 2005-09-15 06:01 am UTC (Expand)
(no subject) - [info]ruakh, 2005-09-14 10:51 pm UTC (Expand)
(no subject) - [info]flying_squirrel, 2005-09-14 10:57 pm UTC (Expand)
(no subject) - [info]anildash, 2005-09-15 04:57 am UTC (Expand)
(no subject) - [info]emarkienna, 2005-09-15 12:01 am UTC (Expand)
(no subject) - [info]crschmidt, 2005-09-15 02:37 am UTC (Expand)
(no subject) - [info]emarkienna, 2005-09-15 03:05 am UTC (Expand)
(no subject) - [info]jamesd, 2005-09-15 05:07 pm UTC (Expand)
(no subject) - [info]anildash, 2005-09-15 04:56 am UTC (Expand)
(no subject) - [info]jamesd, 2005-09-15 05:34 am UTC (Expand)

[info]abates
2005-09-15 12:22 am UTC (link)
This may require detecting hits on RSS feeds from GoogleBot and giving it a blank feed. hmmm.

(Reply to this)


[info]jamesd
2005-09-15 05:29 am UTC (link)
Personally, when Brad implemented the bulk feed of all new posts and circumvented even the near-useless (but better than nothing!) title only RSS/Atom option, I decided that LJ just wasn't going to get it and that I'd have to go about setting up a community for people who want DMCA takedown notices sent to all the major search engines, including blog aggregator sites which provide search. Only reason I haven't done it yet is having better and more pressing things to do with my time. It's a crude and nasty hack but without a technical solution, it's the best
there is.

It's already established that it's not acceptable to ignore robots.txt directives, so search engines using RSS or Atom as a techniical means to circumvent robots directives in the HTML version are clearly acting improperly.

I still don't like the law as a hack, though, even if it is the best there is today.

(Reply to this) (Thread)

[info]rho
2005-09-15 12:47 pm UTC (link)
I thought that "set latest_optout yes" in the admin console worked to remove oneself from the latest posts bulk feed?

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]jamesd, 2005-09-15 05:05 pm UTC (Expand)
The Rules
[info]rabababa
2005-09-15 09:53 pm UTC (link)
Bots should follow the rules.
http://en.wikipedia.org/wiki/Three_Laws_of_Robotics

(Reply to this) (Thread)
Re: The Rules
[info]bitxdeadweight
2005-09-16 12:20 am UTC (link)
Just watch out for Asimov's Literary Doomsday Device

(Reply to this) (Parent)

[info]macaholic
2005-09-15 10:30 pm UTC (link)
Yes...what is going to be done to not only block but remove our comment from google when we had previously set our accounts in a manner to block spiders and such?

(Reply to this)


[info]secretbutterfly
2005-09-16 12:09 am UTC (link)
A lot of us werent really aware of it until people on our friendslsit mentioned it. I am very pissed off about thisand sent a complaint to google. I don't know how the coding for LJ works. Is there any way for them to create an option where we can designate a post as "Livejournal only"? Because I mean sometimes you jsut want to share something and not friendslock it, but you don't want the entire free world to see it, you know?

(Reply to this)

Create an Account
Forgot your login or password?
Login w/ OpenID
English • Español • Deutsch • Русский…