|
Total Recall - Project Log
[Recent Entries][Archive][Friends][User Info]
Below are the 20 most recent journal entries recorded in the "Total Recall - VJ Army Statistics Project" journal:[<< Previous 20 entries]
|
02:12 pm
[taren_]
[Link] |
Reverse sort added for song data
http://superwailingbonus.com/totalrecall/songdata.php?play=sp&sort=playcount
Another slow day at work
I completely overhauled my sorting algorithm for the song data page, and added Reverse sorting capabilities. With my old algorithm, when I tried reverse sort, it blew up. Literally.
Click on the orange table headers to sort in reverse by that key.
EDIT
Also fixed the history bug, so history data should be updating normally again.
"What do you mean MySQL client ran out of memory?!"
|
|
01:28 pm
[taren_]
[Link] |
new features
Added functional for Age/Grade breakdown so you can see what songs are listed under age/grade breakdown! check out the links on your user page.
|
|
10:59 pm
[taren_]
[Link] | 30 days down out of 64 at Ingersoll and still counting.
I don't really mind going in anymore, now that I spend a majority of my time on Putty. I mainly monitor my scripts/php code when it gets boring. I even wrote a Status Page to help me find out when stuff craps out.
Today I also wrote the Grade Totals pages (who has the most AAA's/AA's/A's, etc.), and added date/grade breakdowns, so you can see entry grades by month. Fixed some other bugs in the users.php page... I've almost run out of things to do with this project! I'm going to be really sad when it's over, unless people can give me ideas of things they'd like to see implemented?
|
|
10:51 pm
[taren_]
[Link] |
Fixed
I fixed some stuff you guys
First, cleared up a typo in one of my scripts that was causing sigs not to get updated for nearly two weeks. I was missing a " mark. Other than that, will be getting into individual nonsense i.e. autorival ASAP.
Expect sigs to be working again come tomorrow...
|
|
10:49 pm
[taren_]
[Link] |
slight update
Comments (Count / Pct / Weight) are now online
Percentiles (Avg, Bolds, Crowns, % ranks) are now online
Expect to see some progress over the next few weeks, as I have discovered a new use for the time I spend at work staring at the ceiling.
PPK asked about Autorival- I plan to add user data after I finish all the junk in the header:
-Camerawhore (easy)
-Avg Entry Age (easy)
-Oldest Entries (very easy)
-Sauna Mode (moderate)
-Song data (play count, aaa count/rate, extotal/avg/%, etc) (hard!)
I have disabled High Guarantee DJ for now. I don't think it was that interesting to begin with.
Then I can get to stuff like Autorival later
*zzz*
|
|
08:22 pm
[taren_]
[Link] |
fixings
The History page is back up, and I'll be putting all the old data back in it shortly. DJ Points page is now a bit prettier and has bar graphs. (But no bubbles, yet)
Sig images should start displaying "average age" tomorrow.
Still working on getting the rest of the stats pages up and running. Slowly...
|
|
12:48 am
[taren_]
[Link] |
sigs are back...?
It's been a while, but sigs should be coming back within the next few hours. There still may be a few bugs. Avg Age does not work, for now. I have blown away all of the old TR pages (20,000 of them, HTML and PNG files) and in about two hours the script should kick in. I hope it works, but I'm certain there are bugs remaining.
So far I'm liking this PHP/SQL method, except for the fact that now my phpMyAdmin page is fucked up. I bet my host upgraded them. It was FINE this afternoon, god damn, just leave it alone.
If the sigs are a success, then it won't be much trouble at all to produce the rest of the pages, since all of the data can be automatically processed by scripts. You will know if sigs are a success because I put a little mark "V2" in the corner of the image.
|
|
11:14 pm
[taren_]
[Link] |
->mysql?
I've been getting a lot of errors with TR recently; mainly issues concerning the archive being fubar'd. Incomplete download, failed server-side download, 404, who knows. I would also like to stop using my main PC for running the data. With my new-found knowledge of PHP and mysql, I can try to set up a completely server-side automated script.
It's just an idea in my head right now, but I'm trying to think of how I'd go about doing all the calculations and junk without the use of my familiar C++. A lot of that stuff is reeeeally esoteric (read: sloppy) code.
It'd probably be a shitload easier with SQL, too- my C++ data structure is a correlated linked list, whatever that means. There is one giant array of all the entries, and each entry has a pointer to another entry: one to the next entry for the same user but different song, one for the same song but different user, and one that goes through all entries, one user at a time. It's a mess. With PHP I could simply call "all songs by user X sort by title." bam, done.
The only other problem would be sig banners. It's all bitmapped right now, from my crazy calculator font. I'd have to make separate images for each character. Argh.
Oh yeah, and I'd have to write some NASTY sed scripting to convert XML output to SQL queries. I could do it, but it's simply a matter of HOW.
Let me think this over some more.
|
|
01:36 pm
[taren_]
[Link] |
New features
Added AUTORIVAL support. This compares each user against every other user in the database, and ranks them by number of battleground songs. A battleground song is an entry that both users have entered that are within 10 EX of each other. This feature added like 80 minutes onto the nightly runtime, ugh.
Also, I finally added separate pages for PC/FC/PC+FC counts. Originally, it was ALWAYS PC+FC, but now there are separate pages for FC-only, and PC-only counts as well.
|
|
08:39 pm
[taren_]
[Link] |
Bugfix
I fixed the problem on my end that was causing the program to lag forever.
I ported in some code from a different that determined all users' confidence interval and such, and left in the for-loop the code that said "for all users, do XYZ". This was placed inside of a loop that said "for all users, do XYZ". So, if you understand coding, you could be groaning right about now. The program was calculating each user's stats once for each user in the DB.
This thing was getting like 500 sigs per hour, it was so bad. And I have to do about 3600 of 'em. I ran it with some debugging info, and witha handful of users, it was showing 7000 per hour. Then as I added more users to the pool it dropped to 3600. I pinpointed the section of code, and removed the offending for-loop, and it still runs fine at about 8000 per hour.
No actual sig data was changed in the process of fixing this bug. This was merely an internal issue that will shave about 5 hours off the daily runtime of this program.
Awesome. Now it will actually finish before lunch. Today I sat there and watched as it ran until well after 11:00, eight hours in. Tomorrow, watch the timestamp in "History Data" page, it should be 5:00 (at worst) instead of 9:45. Nice.
|
|
06:30 pm
[taren_]
[Link] |
Rankings / High Guarantee DJ
-Added Ranking by Crowns, Bolds, and Percentile counts.
You can now sort by the number of crowns, bolds, and counts in SP, DP, and combined. For example, I can go view who has the most 80-90%ile entries in Single Play only, or check who has the most bolded entries. Remember, a crown counts and a bold, and a bold counts as a 90% entry.
-Added High Guarantee DJ Page (version 1?)
This is the "hardcore ranking" page. The list is sorted by the lower bound of the 95% confidence interval of your percentiles. What?!
Here's how it works:
I find the standard deviation of your percentiles, then multiply it by 1.96 and subtract it from your percentile average. Then I sort by which users have the highest value after all this.
The problem right now is that it seems to give strange output- users with really shitty averages are getting ranked above better players simply because their standard deviation is very low, in turn because they have SO FEW ENTRIES. But I was thinking about it, and it may be just the data set I'm working with in my sandbox. When I run the full version, my distribution resembles a normal curve, but in the sandbox it better resembles the inside of a hillbilly's mouth. So when the full set runs tonight, my Std Dev will decrease dramatically. The output should be interesting.
I was thinking about putting the value there on the sig, but if it's going to be negative for half of the fucking database, I might not.
|
|
08:14 pm
[taren_]
[Link] |
Avg Entry Age Rankings
Added user ranking pages for Average Entry Age. Sorts include by SP, DP, and All. ALL is an average age for ALL entries, not the average of SP and DP. Page also includes a quick breakdown of entries per 6-month period.
Will hopefully be adding more soon.
|
|
10:20 pm
[taren_]
[Link] |
User page updates
Sig changes:
*Repositioned Average Age number.
*Added Normal Clear count. Hard Clears also count as Normal Clears.
User page changes:
*Repositioned VJA/comment/oldentry links
*Added inlined sig image links alongside VJA sig image.
*Added bar graphs of Percentile data
*Added bar graphs of Entry Age. Age is broken down by month, from X to X+1 months. A month is defined as exactly 30 days, for simplicity's sake.
*The one-pixel resize trick for lazy ass bar graphs is Truly A Fantastic.
*Praying that my new "trigger" script won't fuck up again.
Note to self: Don't ever say, "Nothing can POSSIBLY go wrong with this..." because something probably will.
Have an enjoy.
|
|
08:12 pm
[taren_]
[Link] |
Scripted Connections
-Changed gauge color on sig to a blue->skyblue gradient.
I've been having problems keeping the timings between scripts after adding the sigs. The size of my upload RAR file went from 45 MB to almost 100 MB. So I eliminated the problems with this:
-Changed server-side script to not suck. Originally, it ran once daily at 6 am local time to execute another script to extract and slice/dice the .RAR that should have been uploaded by then. More often than not, I'd wake up at 6:30 only to find something had gone wrong. This new version checks hourly for two .rar files in my Unix directory, the actual uploaded file, and the dummy file.
I made a dummy .rar file "donotdelete.rar" because if I had zero, the script would return an error when checking for .rar files (cannot find...). While the file is uploading, its filename is appended with ".filepart" so a command ls -l *.rar | wc -l does not see the temp file. Once it is completed, it sets the .rar count to 2, triggering the ixion script.
ixion script consists of:
~a call to acidvision to delete all html files and whatnot in the /totalrecall/ directory
~extracts the .rar file
~moves all the extracted files to /totalrecall/ since they wind up in root when SYSTEM runs the cron script.
~converts my fucked up conversion script, replacing all the \r characters with newlines
~executed fixed conversion2 script which individually converts all ppm files to png, and deletes the originals.
~moves all html/gif/jpg/png files to /totalrecall/
(Pardon the LED references, heh. Originally the cron script was named nebula and its first command called script grasper)
trigger then renames the .rar to .rar.done so the ls -l *.rar | wc -l doesn't see it any more.
-Changed computer-side scripts as well.
Originally, I had four separate Windows Scheduled Tasks to run Total Recall. They all had to run at separate times. Of course, this rarely ever worked. So I wrote a new EXE file to execute all four commands in order in the same program. No more praying that the timings were correct, everything gets run at the same time now. It'll be interesting to see how much earlier the program finishes. You can see when the program finishes in the History Data page's second column. Right now, it's some time around 4:50 AM, with the program beginning at 3:59 AM. Now that the buffers between tasks are gone... hah!
Hopefully, hopefully, the program should be TOTALLY AUTOMATED now. Wow, and I thought I'd never use Unix outside of CSCI330...
I'm thinking about adding more cool features to the sigs. I'll update when I come with something cool.
|
|
11:12 pm
[taren_]
[Link] |
Toying with sig images
Added the following features to the sig images:
-added purple gauges behind digits in graph to signify relationship between numbers. Unless all ten gauges read zero, the highest number will fill the bar completely up, and the rest of the gauges will be relative to that. I chose purple since it was the only color that would work with the green-to-red range in use now for the digits. (this was a few days ago)
-converted hard-coded graphics to overlays (percentile graph and lettering, white lettering along side, URL at bottom)
-average percentile rating (###.### format), green
-hard clear count (####), red
-total EX (#######), orange
-added SP and DP logos to top right corner
-added support for the following characters:
! # > . < ( ) & $ * _ ^
Sig images will hopefully update tomorrow. I fixed the little glitch where it wasn't properly archiving the "conversion" script file for execution on the Unix box.
|
|
08:33 pm
[taren_]
[Link] |
Sig images
features: dj name, vj ID, crowns, bolds, and breakdown by percentile.
Crown = 1st place ranking
Bold = top 5%
percentiles are broken down into groups of 10%. The leftmost represents the count of entries you have that rank in the top 10% (90-100%) for that song. The next one is all of the 80-90's, 70-80, etc. The red one on the right is 0-10%, which includes italics (bottom 5%).
As you can see, most of my entries fall into the 50-80% range.
To access your sig image, you may hotlink the following:
http://superwailingbonus.com/totalrecall/#sp.png
http://superwailingbonus.com/totalrecall/#dp.png
Where # is your ID number with no leading zeroes.
Image will update daily, just like the rest of the pages.
This is what I threw together yesterday. Today I spiffed them up a bit. I added a background image, brighter pct colors, a 1-pixel border, 100-70-50-30-0 markers for the percentile, and a website address in really tiny font.
Things to add:
More features.
Character support for punctuation and stuff.
Other information?
Cautions: although you may hotlink my image, keep in mind that its size may change at any point, if I add new information and resize the image.
Hope you enjoy it.
|
|
10:04 pm
[taren_]
[Link] |
Looking for image conversion software
I'm looking for some sort of image conversion software to convert PPM format to any common, sane format like PNG, JPG, or GIF. I plan to use it to make alternate sig images for Total Recall, but I only know how to generate PPM files.
This program would ideally be:
-for windows
-free
-spyware-free
-command-line compatible <- really important
-preferrably stand-alone, but not required.
Anyone have any suggestions? I've been scouring the interwebs but can't find anything trustworthy AND freeware.
Once I find anything, any suggestions for what to put in an alternate VJArmy sig-include image?
|
01:43 pm
[taren_]
[Link] |
Individual Song Stats
Coming tomorrow: individual song statistics. These include:
-Play Count
Shows most played songs.
-AAA Count
Shows most AAA'd songs.
-AAA Rate
Shows songs with abundant AAA's
-Scoring Total
Total EX among all players for song. (If I remember right, V has always been on top of this list.)
-Average Score
Total EX divded by Play Count. Shows average score for song per user.
-Average Percentage
Average Score converted to percent (includes grade image).
-Highest Score
Highest acquired EX by song.
-Highest Percentage
Highest EX, converted to percentage (includes grade image). Songs with 204620's go to the top of this list.
-Clear Count
Number of registered CLEARs OR BETTER.
-Clear Rate
Clear Count divided by Play Count
-FC Count
Number of registered FULL COMBOs OR BETTER.
-FC Rate
Duh
-PC Count
Number of registered PERFECTs.
-PC Rate
Duh
-Comment Count
Number of registered non-empty comments.
-Comment Rate
Duh
Breakdowns by SINGLE, DOUBLE, or ALL.
Links will be available in the header to allow you to sort by any of these fields.
|
|
09:38 am
[atmotas]
[Link] | Now i know i have said this before, but i have a good reason why you might look more into it. Make a .cgi file, if your web host provides server side scripts.
Reason why i think you should look into it. Becasue as i was looking on total recall, i noticed you can click on a users name and it will go to a page with their stats. So i believe that means your program makes an html file for each user and there are 1753 users, So that means your program makes 1753 html files, one for each user. Now with cgi, you can write the cgi program to execute, and will write the html file for the user to see each time he click on the link, so your program doesn't write any html pages for each user, it doesn't even write the cgi file each time. The CGI file will access the data base of the user and write the webpage right then and there. Now with each user like number 280. you want to see his stats, well lets name the cgi files userstat.cgi, in the userstat program you would have something that would take in a variable, and that variable would be what user you wanted to see and the url would looke like this "~totalrecall/userstat.cgi?usernum=280"
not sure if i got it right but it is something like that. Well that calls the userstat program with the usernum equal to 280, and the program will run, get the information of user 280 and write the webpage with the users info.
If you interested search for online guilds on how to write and work cgi files, and most of them will sugest or use perl as the programming language, but you can use c++ for it.
http://www.cgi101.com/book/
this seems like it has good information, but teaches it in perl.
http://hjs.geol.uib.no/cplusplus/
this one teaches it n c++
http://www.coding-zone.co.uk/cpp/cgi/170401cgicpp.shtml
and another link.
this is just a suggest/idea i had, so you are not uploading all those html files.
|
|
04:33 pm
[taren_]
[Link] |
New feature for 12/18
A new, "building" feature will be available tomorrow. A link should appear in the header to "History Data". This page will update daily with historical information about the number of users and entries each time the program is run. It will contain a table listing the Date and Time, along with the number of Entered Users and User Entries available at that time. It will append a new row to the table each day, so you can look at growth trends.
One potential problem is that I've noticed that occasionally the RAR archive gets corrupted when downloading from Remy's site, so everything will look fine, but only half of the user database will show up on a given day. These errors will be included in the history report, but can be EASILY edited out at my own discretion, since it's only a text file on my end that grows.
Tomorrow there will be two lines, one for today's data, and one for tomorrow's data, when it updates in about 12 hours. Enjoy.
|
[<< Previous 20 entries] |
![[info]](http://l-stat.livejournal.com/img/community.gif){kind=small}
totalrecall_
