Home
LiveJournal Client Discussions - LJSM: Livejournal backup tool [entries|archive|friends|userinfo]
LiveJournal Client Discussions

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

LJSM: Livejournal backup tool [Dec. 21st, 2003|11:33 pm]
Previous Entry Add to Memories Tell a Friend Next Entry

lj_clients

[ati]
Hi,

I've written Livejournal backup tool, LJSM.
It saves posts with comments, memories and graphics of any user. Converts relative links to absolute. Creates contents page, supports http proxy.

There are two companion tools:
ljpms -- script for batch posts manipulation
allin1 -- script for joining posts downloaded by ljsm into one file in reverse chronological order
linkReply

Comments:
[User Picture]From: [info]tlhf
2003-12-23 06:10 pm (UTC)

Bug! ^__^

(Link)

I found a slight bug in ljpms. If a post doesn't have a subject, it's subject is set (in livejournal) to HASH(0xwhatever) XD

Line 178 should be:
'subject' => ($mode eq 'delete') ? '' : (ref $entry->{'subject'} ? "" : $entry->{'subject'}),

It's an ugly line, I know, but it makes the patch simpler. I assume you're gonna put that logic above the TRY now it's got two ternary operators. Your coding style is sweet tho. I wish more perl coders wrote like you. (Even if you do use crazy unecessary 's :P)

xxx

(Your code also died on an entry with a ¬ in, but I reckon thats an xml library mismatch between my xml-library/encodes and the server's. XML is teh daft.)
[User Picture]From: [info]ati
2003-12-24 11:21 am (UTC)

Re: Bug! ^__^

(Link)

Yes, you are right.
Thank you for the bugfix, I'll patch my code in the next version
[User Picture]From: [info]andyli
2003-12-27 03:12 pm (UTC)

(Link)

Thanks -- just what I was looking for...

but is this too much of a hit on LJ's servers?
[User Picture]From: [info]ati
2003-12-28 01:31 am (UTC)

(Link)

since it makes incremental backup it should be ok.
[User Picture]From: [info]andyli
2003-12-28 06:32 pm (UTC)

(Link)

Everything worked fine, except all my entires from 2001 - it created the HTML files, but there were empty files (0 bytes).

Is there anyway for me to try to download 2001 posts again? (2002-2003 worked fine and I don't want to redownload it).

Also -- I notice the system is trying to get it in lynx scheme, but mine took the current view scheme. Is there something I'm doing wrong? (maybe its a journal setting).
[User Picture]From: [info]andyli
2003-12-28 06:52 pm (UTC)

(Link)

Nevermind the third part -- I changed from S2 to S1 and that fixed the linx view problem.
[User Picture]From: [info]ati
2003-12-29 10:26 am (UTC)

(Link)

you can use combination of -d and -O switches

i.e. something like this:
ljsm -d 2001/1:2001/12 -O andyli

[User Picture]From: [info]andyli
2003-12-30 11:32 pm (UTC)

(Link)

thanks for the reply.

do you know of anyway for me to fix the S2/S1 problem without switching every time? fornow, I have to keep everything in S1.
[User Picture]From: [info]ati
2003-12-31 01:35 am (UTC)

(Link)

I have to look at this issue more closely.
I think I'll make new version of ljsm at the end of january, which will address the S2 problem (and some others)
[User Picture]From: [info]shadowolf
2005-01-09 06:38 am (UTC)

(Link)

Has there been any progress on an S2 version lately?

Inquiring backup makers want to know!
[User Picture]From: [info]marymoonlight
2003-12-29 04:34 pm (UTC)

(Link)

Are there any other LJ export clients available that are a little simpler to use?
[User Picture]From: [info]ati
2003-12-30 12:33 am (UTC)

(Link)

None, as far as I know.
[User Picture]From: [info]talisker
2004-01-17 09:23 pm (UTC)

Not a bug per se, but something you should consider

(Link)

I just used LJSM, and it worked fine. Nice tool, btw. Just wanted to point out something you might want to look into. I have a post that has 50+ comments, most of which are replies-to-replies-to.... you get the picture.

When this happens, the LJ code simply cascades the outermost comments and they're displayed in the main comment page as only sumbitter and title. To view the comment, you need to click on one, and you'll see the complete hierarchy. If this sounds unclear, email me and I'll send you the file to illustrate what I mean.

So, the point is, in cases like these, LJSM misses comments, as they're not in the main thread list associated with a post. I don't know how this could easily be detected, but what would need to be done is for those such comments to be flagged as having missing comments and those comments retrieved separately.

Best regards,
R.Cote (rglcote@yahoo.ca)
[User Picture]From: [info]ati
2004-01-20 05:36 am (UTC)

Re: Not a bug per se, but something you should consider

(Link)

it's not a bug, it's a feature :)

LJSM just saves html pages, and since livejournal can't be forced to 'unfold' replies ljsm acts exactly the same way
[User Picture]From: [info]zanna_voodoo
2004-04-14 09:12 pm (UTC)

(Link)

Does ljpms only change all posts in the journal to one mode or another? I want to delete a date range of posts. Possible?
[User Picture]From: [info]ati
2004-04-14 10:20 pm (UTC)

(Link)

possible.

export posts in the date range of interest (using www.livejournal.com/export.bml or LJSM) then run

perl ljpms.pl username:password delete
[User Picture]From: [info]ati
2004-04-15 10:15 pm (UTC)

(Link)

yep, I know. planning to fix it in the next version
[User Picture]From: [info]samepaul
2004-10-06 06:04 pm (UTC)

(Link)

dead link
[User Picture]From: [info]ati
2004-10-06 10:32 pm (UTC)

(Link)

[User Picture]From: [info]mikeprime
2005-01-07 01:22 pm (UTC)

A question on LJSM

(Link)

I used LJSM on my livejournals and it worked beautifully, but I also have journals on Deadjournal.com that I'd love if there was a way i could use your tool on. unfortantely, I don't know anything about programming, or I'd try to work it out myself.
I do know that deadjournal is similar enough that most tools i use on LJ work with DJ once i changed the four letters in the server area that make it say Dead instead of livejournal. -l-

-Corin
[User Picture]From: [info]ati
2005-01-07 06:04 pm (UTC)

Re: A question on LJSM

(Link)

if you can run perl scripts, change line
use constant BASE_URL => 'http://www.livejournal.com/';

in ljsm.pl to point to any other site that uses similar interface.

There is currently no way to change BASE_URL in Windows executable version of LJSM
[User Picture]From: [info]mikeprime
2005-01-07 06:36 pm (UTC)

Re: A question on LJSM

(Link)

There is currently no way to change BASE_URL in Windows executable version of LJSM

Which is what i am using. -l- thank you anyway.
[User Picture]From: [info]mikeprime
2005-01-07 08:18 pm (UTC)

Re: A question on LJSM

(Link)

So i installed perl on my machine so I could run from the perl scripts, and for both standard and modified for DJ versions, when i try to run ljsm from my computer, i get the following, but no errors running ljpms

C:\My Documents>perl ljsm.pl
Can't locate HTML/SimpleLinkExtor.pm in @INC (@INC cont
/site/lib .) at ljsm.pl line 83.
BEGIN failed--compilation aborted at ljsm.pl line 83.

can you help at all?
Sorry if this is a simple thing, but this is my first experimentation with using perl.
[User Picture]From: [info]ati
2005-01-07 08:41 pm (UTC)

Re: A question on LJSM

(Link)

you should install several standard perl modules
if you are using ActiveStates's perl, use ppm tool to do it.
e.g.

C:\My Documents> ppm
PPM interactive shell (2.1.5) - type 'help' for available commands.
PPM> verify -upgrade HTML::SimpleLinkExtor
Upgrade package 'HTML-SimpleLinkExtor'? (y/N): y
Bytes transferred: 2194
Installing C:\Perl\site\lib\auto\HTML\SimpleLinkExtor\autosplit.ix
Installing C:\Perl\site\lib\HTML\SimpleLinkExtor.pm
Writing C:\Perl\site\lib\auto\HTML\SimpleLinkExtor\.packlist
Package HTML-SimpleLinkExtor upgraded to version 0,72,0,0
PPM>


you may also wish to open ljsm.pl in any text editor and read the text near the beginning of the file.

don't hesitate to ask me if there is any other questions.
[User Picture]From: [info]mikeprime
2005-01-08 08:18 pm (UTC)

Re: A question on LJSM

(Link)

Since ya offered.:

figured out how to use the ppm tool to get some of what i needed, but it's currently asking for apparently isn't in a repository. But i did find it online, so how do i install from a downloaded archive?

thanks for the help!

-Corin
[User Picture]From: [info]ati
2005-01-08 09:02 pm (UTC)

Re: A question on LJSM

(Link)

if you got an archive from cpan.org you should also get microsoft's nmake.exe

then unarchive perl's module, cd to that directory and run

perl Makefile.PL
nmake
nmake install

this is a standard process for installling perl modules without ppm tool
[User Picture]From: [info]justinitsuj
2005-02-08 07:44 pm (UTC)

Export comments in XML as well?

(Link)

I tried exporting my posts as XML, and it seems the comments weren't pulled as well.

Is this possible?
[User Picture]From: [info]ati
2005-02-09 07:45 am (UTC)

Re: Export comments in XML as well?

(Link)

Not yet. There soon will be new version of LJSM, which may save comments in XML too.
[User Picture]From: [info]justinitsuj
2005-02-09 02:52 pm (UTC)

Re: Export comments in XML as well?

(Link)

Good news. Thanks for letting me know.
[User Picture]From: [info]other
2005-12-26 07:04 pm (UTC)

(Link)

Howdy. This is a great program. Thanks!

I haven't used it in a while and now I get the error:
"got login page http://www.livejournal.com/login.bml but found no login form on it."

Is there a problem on my end or has livejournal changed their login page so that LJSM doesn't work anymore?
[User Picture]From: [info]mbarrick
2006-01-04 07:38 am (UTC)

(Link)

I'm seeing the same problem, and it is the result of a change on LJ. Looking at it quickly it appears that ljsm is failing on the login because the login form is no longer the first <form> on the page and the login subroutine is only testing the first form it finds.
[User Picture]From: [info]ati
2006-01-07 10:41 pm (UTC)

(Link)

Don't do it.
I've changed lj_login function and now it uses 'sessiongenerate' call of LJ client protocol.
See http://www.offtopia.net/ljsm/index_en.html for download links.
[User Picture]From: [info]mbarrick
2006-01-07 11:22 pm (UTC)

(Link)

I've deleted my hack so it won't get used. Using the sessiongenerate call of the LJ protocol is definitely the better way to go.