Evan Martin ([info]evan) wrote in [info]lj_dev,
@ 2003-08-25 16:59:00
Previous Entry  Add to memories!  Tell a Friend!  Next Entry
AOL blocking LiveJournal image loading
Here’s the scoop:

Find a random image on an AOL member’s page.
Retrieve it with a Referer of something other than LiveJournal.com:
% curl -s -i -H "Referer: http://www.livejournalx.com" 'http://hometown.aol.com/dvyfacs/myhomepage/old+friends+1.jpg' | head -1
HTTP/1.1 200 OK
Yay, we got the image!

Now try it with LiveJournal:
% curl -s -i -H "Referer: http://www.livejournal.com" 'http://hometown.aol.com/dvyfacs/myhomepage/old+friends+1.jpg' | head -1
HTTP/1.1 404 Not Found

Hey, where’d it go?



...and what can we do about it? :(

I guess it’s their jurisdiction, but it’s a pretty low thing to do, especially because it appears to their users like it’s our fault. More interesting is that other Referers (for example, using evan.livejournal.com) doesn’t trigger it.

Page 1 of 2
<<[1] [2] >>

(Post a new comment)


[info]dottey
2003-08-25 05:05 pm UTC (link)
I would hope it is just because many other websites have blocked such offsite linking, but it is mostly obvious this has more to do with the new AOL Journals feature they've just announced.

*sigh*

(Reply to this) (Thread)

[info]redfarmer
2003-08-25 05:50 pm UTC (link)
There have been several support requests saying they could load images on uJournal but not LiveJournal. I think it has more to do with a specific grudge against our site rather than against all journaling sites. Besides, if they were to block all journaling sites, the journal sites (LJ, uJournal, Blogger, etc.) could file a class action law suit for AOL attempting to create a monopoly (like they don't already).

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]krellis, 2003-08-25 07:41 pm UTC (Expand)
(no subject) - [info]amokk, 2003-08-25 09:05 pm UTC (Expand)
(no subject) - [info]akeller182, 2003-08-26 02:28 pm UTC (Expand)

[info]leora
2003-08-25 05:05 pm UTC (link)
It's worse than that. Most of their low level tech Support people don't seem to be aware that they blocked LiveJournal. So, they're often telling AOL users it is our fault. Their upper level tech Support does know though.

Fun, fun, fun.

And now we have a request from an AOL user that suddenly they stopped getting LJ emails. They say AOL did just add some new spam filters, so that may relate.

It almost makes you think that they don't like us...

(Reply to this) (Thread)(Expand)

[info]burr86
2003-08-25 05:17 pm UTC (link)
Well, we didn't really like THEM, so ... :p

(Reply to this) (Parent)(Thread)(Expand)
AOL - [info]pne, 2003-08-26 09:34 am UTC (Expand)
(no subject) - [info]bery, 2003-08-29 05:53 pm UTC (Expand)
(no subject) - [info]leora, 2003-08-29 06:01 pm UTC (Expand)
(no subject) - [info]acpizza, 2003-10-27 10:39 am UTC (Expand)

[info]mr_dark
2003-08-25 05:19 pm UTC (link)
That begs a very interesting question:

What effect will it have on LJ if AOL decides to basically blackball us from their systems? More importantly, what recourse do we have if they decide to do so? I mean, it isn't illegal as far as I know to simply disallow e-mail, image linking, etc. from a certain domain as an ISP.

They've certainly tried to block alternative chat clients like Trillian more than once.

(Reply to this) (Thread)(Expand)

[info]leora
2003-08-25 05:24 pm UTC (link)
My guess. Some AOLers will leave LJ and some will leave AOL.

But they've pulled this before with other sites, and in my experience, it's not all that harmful to the other sites.

LiveJournal is big enough and strong enough that it could survive with no AOLers. Plus, it's not trying to make as much money as possible by any means.

So, if some AOLers leave, where's the big harm? LiveJournal will continue and still be popular.

Personally, if I were an AOLer (which I'd never be, I admit), I'd be furious at them for cutting off some of my services without telling me. But that's up to individual AOLers to decide. And for them to decide if they want to take action against their ISP for it.

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]mart, 2003-08-25 05:39 pm UTC (Expand)
(no subject) - [info]evan, 2003-08-25 05:45 pm UTC (Expand)
(no subject) - [info]ibrad, 2003-08-30 01:19 pm UTC (Expand)
(no subject) - [info]volantwish, 2003-08-25 11:00 pm UTC (Expand)
(no subject) - [info]leora, 2003-08-25 11:08 pm UTC (Expand)
(no subject) - [info]volantwish, 2003-08-25 11:33 pm UTC (Expand)
(no subject) - [info]akeller182, 2003-08-26 02:31 pm UTC (Expand)
(no subject) - [info]leora, 2003-08-26 02:44 pm UTC (Expand)
(no subject) - [info]akeller182, 2003-08-26 03:04 pm UTC (Expand)
(no subject) - [info]leora, 2003-08-26 03:10 pm UTC (Expand)
(no subject) - [info]k1001001, 2003-08-30 01:28 pm UTC (Expand)
(no subject) - [info]ziemkowski, 2003-08-25 07:11 pm UTC (Expand)
(no subject) - [info]seattlesparks, 2003-08-26 03:20 am UTC (Expand)
(no subject) - [info]deahamlet, 2003-08-27 12:28 am UTC (Expand)
(no subject) - [info]k1001001, 2003-08-30 01:33 pm UTC (Expand)

[info]lisanys
2003-08-25 05:38 pm UTC (link)
With Opera, I have referrer logging turned off (always have and will).

The image you pasted loaded just fine.

I said:
GET /dvyfacs/myhomepage/old+friends+1.jpg HTTP/1.1
User-Agent: Opera/7.20 (X11; Linux i686; U)  [en]
Host: hometown.aol.com
Accept: text/html, application/xml;q=0.9, application/xhtml+xml;q=0.9, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1
Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0
Cache-Control: no-cache
Connection: Keep-Alive, TE
TE: deflate, gzip, chunked, identity, trailers

hometown.aol.com said:
HTTP/1.1 200 OK
Date: Tue, 26 Aug 2003 00:37:16 GMT
Server: Apache/1.3.26 (Unix) Resin/2.0.5
Keep-Alive: timeout=10, max=700
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: image/jpeg

(Reply to this) (Thread)(Expand)

[info]jtek
2003-08-25 05:41 pm UTC (link)
Same here. Works great when people post images from GeoCities (who block ALL offsite linking) on their journals.

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]k1001001, 2003-08-30 01:35 pm UTC (Expand)
(no subject) - [info]uncaring, 2003-08-25 05:50 pm UTC (Expand)
(no subject) - [info]jackola, 2003-08-26 01:16 pm UTC (Expand)
(no subject) - [info]lisanys, 2003-08-26 01:51 pm UTC (Expand)

[info]gwferguson
2003-08-25 05:52 pm UTC (link)
Well, that explains a peculiar problem I've been having: I'm unable to link my own images or websites on CompuServe, which is now owned by AOL, to LiveJournal.

Poopie-bears to CompuServe!

(Reply to this) (Thread)

[info]oncolgist
2003-08-27 11:48 am UTC (link)
and as of last night, i wasn't getting any of my comment replies to my compuserve address, so i've had to start using my hotmail one, which i'd rather not do as my cs address i use for all "internet" things such as lj and other mailing lists...

(Reply to this) (Parent)

[info]vfc
2003-08-25 05:56 pm UTC (link)
Does anyone think it might have something to do with the fact that AOL is starting its own journaling service?

(Reply to this) (Thread)(Expand)

[info]flashfire
2003-08-25 06:08 pm UTC (link)
Yes, I think that thought's probably crossed the minds of a few people.

(Reply to this) (Parent)
(no subject) - [info]skyknyt, 2003-08-26 12:06 pm UTC (Expand)
(no subject) - [info]insaint, 2003-08-26 09:51 pm UTC (Expand)
(no subject) - [info]skyknyt, 2003-08-26 09:53 pm UTC (Expand)
(no subject) - [info]akeller182, 2003-08-26 02:33 pm UTC (Expand)
(no subject) - [info]arturus_magi, 2003-08-27 09:51 pm UTC (Expand)
(no subject) - [info]thatsmytrunks, 2003-08-30 09:31 am UTC (Expand)

[info]mvivei8828
2003-08-25 06:17 pm UTC (link)
aol is sucha crock of shit. their service is slow, and unreliable and the email filter is a piece all on its own. im glad that i switched to msn

(Reply to this) (Thread)(Expand)

[info]insaint
2003-08-26 05:54 am UTC (link)
The lesser of two evils...

(Reply to this) (Parent)(Thread)(Expand)
(no subject) - [info]diffrentcolours, 2003-08-28 03:20 am UTC (Expand)
(no subject) - [info]insaint, 2003-08-28 04:54 am UTC (Expand)
(no subject) - [info]diffrentcolours, 2003-09-02 02:40 pm UTC (Expand)
(no subject) - [info]acpizza, 2003-10-27 10:43 am UTC (Expand)
(no subject) - [info]mvivei8828, 2003-10-27 02:27 pm UTC (Expand)

[info]bradfitz
2003-08-25 06:37 pm UTC (link)
I hearby propose that we publicize and distribute a fre personal proxy for Win, Mac, and Linux that does nothing but remove all Referer headers on requests that look like images. It'd be a good mix of letting people load images from anywhere and still give sites navigation path stats.

I have a Perl to Win32 exe compiler license (ActivePerl), so this is all really like 20 lines or so. Mac OS X and Linux can just run the Perl.

Evan, Whitaker... we can work on this tomorrow.

(Reply to this) (Thread)(Expand)

[info]kaoslord
2003-08-25 06:48 pm UTC (link)
slickness. i'd love to test.

(Reply to this) (Parent)
(no subject) - [info]bobert225, 2003-08-25 06:49 pm UTC (Expand)
(no subject) - [info]darkpixel, 2003-08-25 07:33 pm UTC (Expand)
(no subject) - [info]bradfitz, 2003-08-25 08:05 pm UTC (Expand)
(no subject) - [info]euqsam, 2003-08-25 08:53 pm UTC (Expand)
(no subject) - [info]zanfur, 2003-08-25 09:56 pm UTC (Expand)
(no subject) - [info]flickgc, 2003-08-26 01:19 am UTC (Expand)
(no subject) - [info]evan, 2003-08-26 01:26 am UTC (Expand)
(no subject) - [info]flickgc, 2003-08-26 01:28 am UTC (Expand)
(no subject) - [info]flickgc, 2003-08-26 02:42 am UTC (Expand)
(no subject) - [info]mart, 2003-08-26 04:38 am UTC (Expand)
(no subject) - [info]kaoslord, 2003-08-26 09:15 am UTC (Expand)
(no subject) - [info]mart, 2003-08-26 09:27 am UTC (Expand)
(no subject) - [info]jackola, 2003-08-26 01:23 pm UTC (Expand)
(no subject) - [info]morganite, 2003-08-26 05:29 pm UTC (Expand)
(no subject) - [info]ruakh, 2003-08-28 11:32 am UTC (Expand)
(no subject) - [info]mart, 2003-08-26 04:36 am UTC (Expand)
(no subject) - [info]meowpurrr, 2003-08-26 04:43 am UTC (Expand)
(no subject) - [info]professorb, 2003-08-28 05:40 pm UTC (Expand)

[info]uncaring
2003-08-25 06:45 pm UTC (link)
...and what can we do about it?

Haven't thought through this too much, but...

1. Set up a system to proxy image requests (let's call it imageproxy),
2. Before serving up a LiveJournal page that links to an image on an AOL server, re-write the image URL to be something like http://imageproxy.livejournal.com/get/RealURLToImage.
3. Have the proxy strip out any referer headers (which I've never set up but am assuming is possible).

Downsides:

1. Easily defeated by AOL -- block all requests from imageproxy.
2. The LJ network is now consuming more bandwidth by sucking down all these AOL images and then serving them to your users.
3. Others?

Upsides:

1. Transparent to users (e.g. they wouldn't have to modify URLs to their images).
2. Makes AOL images viewable to LJ users (i.e. solves the original problem).

Thoughts?

(Reply to this)


[info]knew
2003-08-25 06:47 pm UTC (link)
Now would you believe it, Netscape's (now, who would they be owned by now) "My Webpage" site is also doing the same thing.

(Reply to this)


[info]flashfire
2003-08-25 06:48 pm UTC (link)
Anyone think this would be reasonable to bring up in [info]news, in addition to the [info]lj_biz mention? I'd think this information should be spread around quickly just so more people understand what's going on.

(Reply to this)

the time for playing "cowboys" is long since past...
[info]darksoul
2003-08-25 07:19 pm UTC (link)
Am I the only person who thinks "so what"?

All of the free hosting services have blocked remote loading of images. Why? Because of the massive drain on their servers that LJ has posed. Sure, they blocked remote loading to anywhere, but it was still done mostly because of LJ users.

I think that going off half-cocked and helping users be able to circumvent AOL's ban on remote-loading to LJ will only serve to disintegrate any notion that this is a legitimate business run by an intelligent and legitimate businessman (Brad). Instead, the owner and his employees, not to mention the users, will look like a bunch of hacker hoodlum degenerates who want nothing more than to "screw the man" and "hack the planet".

If AOL wants to block remote loading from LJ, let them. It's their perogative as a business. When users come to us asking why their images won't load, explain to them that AOL is blocking images from remote loading to LJ and to request further information from their higher level tech support people as to why. That's the sensible, business-minded route.

(Reply to this) (Thread)(Expand)
Re: the time for playing "cowboys" is long since past...
[info]leora
2003-08-25 07:26 pm UTC (link)
There are some merits to that.

The difference between free sites blocking remote loading and AOL doing so is that the AOL users are paying for a service they are not receiving.

But really, they should be complaining to AOL and either demanding the services they paid for or finding a new ISP. It is, of course, up to each individual what they choose to do.

(Reply to this) (Parent)(Thread)(Expand)
Re: the time for playing "cowboys" is long since past... - [info]darksoul, 2003-08-25 07:43 pm UTC (Expand)
Re: the time for playing "cowboys" is long since past... - [info]leora, 2003-08-25 08:13 pm UTC (Expand)
Re: the time for playing "cowboys" is long since past... - [info]luminifer, 2003-08-26 04:58 am UTC (Expand)
Re: the time for playing "cowboys" is long since past... - [info]the_dibbler, 2003-08-25 11:18 pm UTC (Expand)
Re: the time for playing "cowboys" is long since past... - [info]darksoul, 2003-08-26 07:20 am UTC (Expand)
Re: the time for playing "cowboys" is long since past... - [info]the_dibbler, 2003-08-26 11:50 am UTC (Expand)
Re: the time for playing "cowboys" is long since past... - [info]darksoul, 2003-08-26 11:57 am UTC (Expand)
Re: the time for playing "cowboys" is long since past... - [info]acpizza, 2003-10-27 10:46 am UTC (Expand)
Re: the time for playing "cowboys" is long since past... - [info]darksoul, 2003-10-27 10:56 am UTC (Expand)
Actually, they did/are - [info]acpizza, 2003-10-27 11:13 am UTC (Expand)
Re: Actually, they did/are - [info]darksoul, 2003-10-27 11:26 am UTC (Expand)
URL shortening sites might solve your problem.
[info]jaybonci
2003-08-25 08:33 pm UTC (link)
There are a pile of URL shortening services out there, and a referrer from there should solve your problem, correct?

The Yahoo directory on this, along with CPAN's WWW::Shorten which is an interface to a pile of these sites.

That should solve the problem, at least in the short term, but then yes, it's an arms race.

--jay

(Reply to this) (Thread)
Re: URL shortening sites might solve your problem.
[info]mart
2003-08-26 04:43 am UTC (link)

I don't think this can work. When a server responds with a “redirect” response the client requests the new URL using the same referrer as the initial request, since redirecting isn't really referring, it's saying “actually, what you're looking for is over here…”

(Reply to this) (Parent)

[info]quirrc
2003-08-25 10:23 pm UTC (link)
i'd like to note that aol is blocking not just images but all files with livejournal referrer like clicks from livejournal pages. for example here http://www.livejournal.com/support/see_request.bml?id=147731

(Reply to this) (Thread)(Expand)

[info]knew
2003-08-25 10:35 pm UTC (link)
now that's just totally unacceptable. blocking images i can understand, but blocking everything is completely unreasonable.

this isn't the only thing that AOL has been up to recently either. it seems they've completely blocked all email from a particular hosting company, and ignored a temporary restraining order requiring them to stop the blocks. more on this can be found at http://www.hostreview.com/news/news/030822CIHost.html

(Reply to this) (Parent)
(no subject) - [info]bradfitz, 2003-08-25 10:44 pm UTC (Expand)
(no subject) - [info]the_dibbler, 2003-08-25 11:20 pm UTC (Expand)
(no subject) - [info]_glittergirl, 2003-08-26 07:05 am UTC (Expand)

[info]dawn_key
2003-08-25 11:49 pm UTC (link)
AOL is being dumped by Time-Warner. They have lost over a million users in the past year and have no veteran managers left. This is just one more indication that they are desperate and floundering and illustrates why the stock is taking a nose dive.

AOL is seen on Wall Street as "training wheels" to the Internet. Most people don't need or want training wheels any more. People are switching to high speed connections and AOL's relatively poor content isn't worth keeping for the price they charge. So if it makes anybody feel better, they are likely not going to be the biggest kid on the block for too many more years.

(Reply to this) (Thread)

[info]suppafly
2003-08-26 09:43 am UTC (link)
haha.. great troll

(Reply to this) (Parent)

[info]fate_steps_in
2003-08-26 12:00 am UTC (link)
*sigh* So that's why my LJ background isn't working.

(Reply to this) (Thread)(Expand)

[info]peachiegirl216
2003-08-26 07:41 am UTC (link)
cute user pic. :)

THat is exactly why your background won't show up. I asked about my pictures not showing up on the Compuserve forums and they are clueless basically. But someone did post the link to this thread. I say we all contact AOL/CS and complain. Everything works fine on my Blurty and my mindsay journals. It's just here that things don't work.

(Reply to this) (Parent)
(no subject) - [info]cranehaven, 2003-08-29 09:45 am UTC (Expand)

[info]petfish
2003-08-26 09:48 am UTC (link)
Heh. It's not really that I don't care, it's just that it's kinda amusing to see AOL fucking up again.

Speaking from experience, AOL has a tendency to pull shit like this. I can think of several occasions where they decided all Earthlink email was spam. One time it started with the dialup users, so I could see it as a script going awry at first, but then it expanded to everyone on the earthlink.net domain, and that was kinda wacky. I seem to remember something similar happening at another isp I worked for, that did hosting. Once it was email, but a few times it was blocked sites.

Every time, the blocking didn't last for more than a few days. A few calls from angry Abuse departments at other ISPs, a couple hundred calls from their own users, maybe the threat of a lawsuit or two, and they bring it down and never admit they did it in the first place. This situation could be different, but I wouldn't be at all surprised if it blew over like that.

(Reply to this)


[info]deltatau
2003-08-26 02:59 pm UTC (link)
I was told by an AOL staffer that if we change the image url to ftp://members.aol.com/username/photo.jpg (instead of http) that the images will come up. This seems to have fixed the problem.

I'm still annoyed enough that I'm leaving AOL and moving all my images over to a paid domain instead.

(Reply to this) (Thread)(Expand)

[info]insaint
2003-08-26 09:54 pm UTC (link)
I'm still annoyed enough that I'm leaving AOL and moving all my images over to a paid domain instead.

Good move.

(Reply to this) (Parent)
(no subject) - [info]kutsuwamushi, 2003-08-27 01:34 am UTC (Expand)
(no subject) - [info]evan, 2003-08-27 11:02 am UTC (Expand)
(no subject) - [info]kutsuwamushi, 2003-08-27 11:39 am UTC (Expand)

[info]bimonsradley
2003-08-27 12:52 am UTC (link)
I hear with AOL 9.0 they are coming out with their own journal type thing, if this is the case this could be there reasoning behind blocking access.

(Reply to this)


[info]seanachie
2003-08-27 07:15 am UTC (link)
You can troubleshoot here. This might help. http://webmaster.aol.com/

(Reply to this)


[info]jennilee_rose
2003-08-27 10:07 am UTC (link)
Ok stupid question time:

Is it safe to assume that those using mywebpage.netscape.com (because they're affiliated with AIM users) is also a part of this blocking? I have my images there and mine aren't loading either.

Jennilee

(Reply to this) (Thread)

[info]evan
2003-08-27 10:10 am UTC (link)
Yes, they are doing it too.

(Reply to this) (Parent)

[info]seanachie
2003-08-27 12:39 pm UTC (link)
[info]jettdelirium uses AOL to webhost her LJ photos I think. I'm confused. Did she find away around your problem?

(Reply to this) (Thread)(Expand)

[info]evan
2003-08-27 04:11 pm UTC (link)
It looks like our workaround is working for at least one person...

(Reply to this) (Parent)
(no subject) - [info]cranehaven, 2003-08-29 09:48 am UTC (Expand)

[info]mizanchan
2003-08-28 02:56 pm UTC (link)
I don't know...maybe I'm just hopelessly naive, but I would think if a site is going to block another site from image loading, it would actually contact the other site and explicitly tell them why this course of action was required.

I work in a computer store, and part of my job is garnering AOL signups (much as I loathe AOL). If you could see all the responses I get when I mention signing up for a free trial offer of AOL as we're instructed to, your heads would spin. Believe me. I could spend my whole shift asking people, and only get one or two signups, if that. Their PR sucks royally.

I believe that if AOL's trying to stifle its biggest competition by conveniently blocking image loading while it plans to come out with its own journal service, it's starting to turn into another Microsoft.

(Reply to this)

Easy to fix.. "liveimage" service.
[info]pfak
2003-08-29 04:44 pm UTC (link)
<? if (!$url) { header("Location: $url"); } else { die("No URL Specified"); } ?>

Simple, removes headers. You could even implement this directly into LiveJournal code with a simple regex.

youd do http://liveimage.livejournal.org/?url=http://evil.aol.com/site

Simple, and fixes your problem completley.

(Reply to this) (Thread)
Re: Easy to fix.. "liveimage" service.
[info]kagekonjou
2003-10-26 05:11 am UTC (link)
When you do a Location forwarder from PHP, it sometimes will send referrer tags. Same can be said for Perl, as well as any other web scripting language. The only TRUE way to go around this is to preform a GET procedure, grab the image, then paste its raw data.

(Reply to this) (Parent)
There I am, reading Slashdot, when what do I see but...
[info]ac
2003-08-29 05:50 pm UTC (link)
[info]evan, I just wanted to be one of the first to say congratulations for making it on /. with such a well-written (and succinct) dissertation on the problem. Of course, based on the comments, it sounds like most people either didn't read it or didn't understand it, but I suppose that goes with the territory.

Hopefully, this public discussion of the problem will result in something from AOL. I guess we'll just have to see...

A.C.
******

(Reply to this)

Page 1 of 2
<<[1] [2] >>

Create an Account
Forgot your login or password?
Login w/ OpenID
English • Español • Deutsch • Русский…