Things have been pretty busy lately, so I haven't had much of a chance to work on StormChat. However, users over at Evo have been having a lot of problems with it lately, so I need to get some things done.

Todo list:

1. WEB: Rework the CPSP field. This should be renamed connection_mode. Values should be:
   • 0 = SlimChat
   • 1 = Client Pull
   • 4 = Server Push
   • 5 = Server Push (with Proxy)
2. SERVER: Adjust code to work with new CPSP values & name.
3. WEB: Autodetect Firefox 1.5 and push them over to using the "with Proxy" setting. Direct other Server Push users directly to the server. This will help prevent sessions timing out so often as the server has anti-timeout code (keep-alive settings) that the PHP based proxy doesn't (can't?) support.
4. WEB: Work on the client pull code. This should operate on a timeout basis for checking new events rather than a meta-refresh. Meta refreshes break down when the browser can't connect to the server (fast enough) and the page does not complete loading. This causes it to never be loaded again because it doesn't have a meta tag to keep refreshing.
   • Revamp non-AJAX client-pull version. Store the new MID via JS in the content page and use a timeout to build and adjust the URL of the event page, either loading the new URL or refreshing the old URL.
   • Work on the AJAX code version. This will load the events via XML.
5. WEB: Extend STC() to set the title of the page to include the number of unread events. When someone types in the submit pane, the # should be cleared and the title updated.

I also need to work on testing the 2x18G drives that I got recently and hopefully get them installed in sunrise soon (need to talk to Curt as we're way past the beginning of Feb. to go down to HostTech). Need to upgrade sunrise from Mandrake 9.1 to Mandriva 2006 as 9.1 is way out of date (and I believe causing some wierd obfuscation of errors with the StormChat server due to the SMP nature of the box-- mainly, seeing EINTR errors that mask the true error on a socket).

Did more work on the FALs now that I got the general "API" figured out... renamed *_db.php to *_fal.php and added the functions. The FALs for SMF and Invision both need work (and testing). The phpBB FAL seems to work fine. I did sit down and write an initial LiveJournal FAL, but it needs further work and testing. Also need to work on how to support the different FALs during installation.

Code feels much cleaner with that reworked. Lot of config variables and whatnot that were able to be removed, which makes me happy.

Looking over the code, I found that it should be pretty trivial to generate a clean FAL... since I've already moved away from using the actual forum database as much as possible (mainly done when I was hosting StormChat for Fantastic Byways, and when Evolution was still being hosted on Pele).



<?php

function getAvatar($userinfo) {
    // returns URL to the avatar to display for the user
}

function checkAuth($username, $password, $modGroup, $checkAdmin) {
    // success: return user info array
    // failure: return error string
}

function checkBoardBan($ip, $userid) {
    // returns boolean result
}

function checkValidAlias($alias, $uid) {
    // returns boolean result
}

?>


• ADMIN: Converted to template system:
  • Admin navigation header
  • Manage Rooms
  • Manage Groups
  • Delete Room
• ADMIN: Added code to handle:
  • Delete Group
  • Add Group
  • Change Dynamic Group

Haven't updated in a bit, but that doesn't mean that I've been idle.

I've been working on testing some AJAX-based code for the submit page. It works, but it's not where I want it to be right now, so I'm holding off on those changes at the moment rather than committing them to CVS. I think I'm going to split that development off to a separate branch while I work on getting things ready for release.

I am looking at working on the code for the FAL before release, though, because I really don't like how the conditional code is scattered all throughout the project.

I think I found a way around the problem with the dual ORDER BY statements in the C code that caused problems in MySQL 5.x and in PostgreSQL as well.

In PostgreSQL, I need to have the row # when retrieving a result when using libpq (PHP automatically handles this). In MySQL, I don't... but I can use mysql_data_seek() to go to a specific row. So on rows I'm doing:

sprintf(buffer, "(SELECT * FROM xtable ORDER BY xfield DESC LIMIT mylimit) ORDER BY xfield ASC");

pthread_mutex_lock(&db_mutex);
results     = db_query(dbconn, buffer);
pthread_mutex_unlock(&db_mutex);
num_rows    = db_num_rows(results);

for (i = 0; i < num_rows; i++) {
    row     = db_fetch_row(results, i);
    // further processing
}

I can now just do:

sprintf(buffer, "SELECT * FROM xtable ORDER BY xfield DESC LIMIT mylimit");

pthread_mutex_lock(&db_mutex);
results     = db_query(dbconn, buffer);
pthread_mutex_unlock(&db_mutex);
num_rows    = db_num_rows(results);

for (i = num_rows - 1; i >= 0; i++) {
    row     = db_fetch_row(results, i);
    // further processing
}

Still need to work this all the way in (I'm looking at how master_poll_events operates and I'm trying to determine if I need to handle it the way I currently am or not).

• WEB: Convert options page to template system.


• WEB: Merge functionality of the whoson-lite page into whoson and use template to show slim display.

• ADMIN (TODO): Manage config


• ADMIN (TODO): Ban user/ip address


• ADMIN (TODO): Manage current bans


• INSTALLER: Finish installer.

Tried running this on my home machine under Solaris 10, with not-so great results. I'm not sure if it's a PHP5 issue or a Solaris issue but the proxy that I wrote for server push does not work correctly. You have to log out to see new events... not useful at all.

Also, doing double ORDER BYs doesn't seem to work, either, though that's probably a MySQL issue with my setup (MySQL 5.0.16-max). I will try to work around this, since PostgreSQL doesn't let you do double ORDER BYs, either. In the PHP code for client pull, I've already fixed this (simple call to ksort($events);) but the server push code is a bit more difficult.

• WEB and SERVER: Fixed the Server Push issue with Firefox 1.5 by writing a PHP proxy script. Using this, I never have to set document.domain, so I've removed those calls in the web code and the server. All is good. :)


• THEME: Committed the gray theme to CVS as well, I'm personally digging this a lot.

• WEB: Reworked help file to use a template (rather than globbing everything into the overall_header.tpl) and added filter entries to the help system.


• WEB PLUGIN: Added /wartimers command to the Word War plugin. Currently fixed at a 1 hour timer.


• WEB PLUGIN: Added /cleanup command to the Misc Commands plugin. Removes junk entries (PMs to self, /help entries, /ping entries, etc.)


• WEB PLUGIN: Added /canceltimers command to the Misc Commands plugin. Sets all active alarm timers to inactive. (Moderator only.)


• WEB PLUGIN: Added URLFilter plugin to generate clean links.


• WEB and WEB PLUGINS: Added a field to events to track whether or not the plugin parsed the event (only for commands, not filters... filters are irrelevant to this). If no plugin parses the command, then it's not a valid command and an error message is raised. This fixes a rather annoying bug where the text of the offending command was passed directly to chat.


• WEB: Added code to the header to allow for a navigation bar to be used in the template through tweaking the overall_header.tpl file. Demonstrated this with the testing template:

Some of these were already done, but as SOMEONE has been complaining about me announcing things, I'm mentioning them here and now.

• SERVER: Added client pull timeouts and idle outs (ignored for moderators).


• SERVER: Added time based alarms (low-hanging fruit while working on client pull timeouts).


• SERVER: Shutdowns now add event into database (for CP/SC users).

Reviewed and updated the roadmap again. Not going to aim for PostgreSQL out of the gate; I don't have the time to find and work around all of the MySQL specific code I've been using (UNIX_TIMESTAMP(), LIMIT, GROUP BY, multiple ORDER BY statements, etc.). That can come later.

This basically leaves 3 things left on my plate before I can comfortably release this to the public. Mainly the installer, the admin panel, and the conversion to phpBB-style templates. Along with the last piece, I want to go through and do a general cleanup of the code... basically look for various debugging print() statements, and kill them. Use addToLog rather than syslog. Add calls to showFatalError as appropriate. Add comments as appropriate.

It's been a long time coming, but this is really starting to shape up. The server code still needs work, as I haven't been able to track down that crash that occurs randomly when a user logs out. But all in all, this is starting to be quite a good framework.

I do want to OOPify some things later in the web code-- notably what I intend to call the FAL (Forum Abstraction Layer)-- but that's not a major deal. Once everything is using templates, it won't matter how I change the back-end code as long as I'm still filling out the templates the same.

• WEB: Converted topic page to template system.


• WEB: Merged functionality of the main and main_slim pages via conversion to template system.


• WEB: Merged functionality of the events and events_slim pages via conversion to template system.


• WEB (BUG): If command is not found, the plugin code will not trap this and the text is blindly passed to the room without note of who entered it, etc..

• WEB: Working on converting the code to use phpBB style templates. So far I've got the help, index, contents, transcripts, client pull event, avatar, submit, and Who's On pages converted.


• WEB: Added error function (showFatalError(msg)) which will be wrapped within the template system.


• WEB (TODO): Convert options, main, and topic pages to template system.


• WEB (TODO): Merge functionality of the whoson-lite page into whoson and use template to show slim display.


• WEB (TODO): Merge functionality of the slim pages into main code with different template files?


• WEB: User-specified CSS works in Client Pull now.


• WEB: Client pull events page will reset the pull_timer field in the sessions table to 0 when it checks for events. This will allow for the chat server to eventually track people who have wandered off and closed their browser window without logging out properly and time them out.

Also cleaned up and updated the roadmap post.

Including template files and code, the web code comes to 9000+ lines of code. The server code brings that to just shy of 14000 lines.

• SERVER: Added support to the code to use database prefixes. This will allow users of one database to run multiple chat systems with different prefixes.


• WEB: Check user/IP bans on login


• WEB: Added logging table, will eventually convert all syslog calls to addToLog...


• SERVER: Added autoconf support (did this a bit ago, but noting it now...)


• WEB PLUGIN: /userban command


• WEB PLUGIN: /ipban command

Upon further testing with PostgreSQL, I found most of my trouble was a result of differences with the GROUP BY clause. I've reworked the code to bypass this in the few cases I used it, and it works with SlimChat. Do need to test the server with Server Push.

• WEB: Moved report code to a plugin.


• WEB PLUGIN: Implemented ignore, unignore, and listignore commands as a plugin. (I'm bad, I know.)


• WEB: Implemented support in Client Pull and SlimChat for ignoring. (Yes, it's possible to ignore yourself.) The Client Pull code isn't tested, but I'm pretty sure it'll work given the similarity between events.php and events_slim.php.


• WEB: Fixed global events (chat server starting, /wall messages, etc) in Client Pull.

• WEB/SERVER: Implemented the SendToChat changes I talked about earlier. This should help quite a bit with bandwidth usage. :) Didn't take as long as I thought that it would.


• SERVER: Also added code to the chat server to detect the config value for document.domain and send that in the JS. I think I deleted it at some point when I was removing hard-coded references to Evolution.


• SERVER (TODO): Need to abstract the references to the database tables in the chat server so they can use a different database prefix. All the code for that support is in the web interface, but it's not in the server.


• MISC: Added some preliminary support for user ignoring; mainly because it was similar to where I was working with the SendToChat code. Have the database tables made and the way to tell the server about changes. Don't have the user functions (which will be a plugin) or the chat server support yet (will also need to put it into client pull as well... will probably do that before the chat server, actually. Especially given that I'm using FF 1.5 so of course I'm going to be using Client Pull myself anyways...)


• WEB: Also did a major revamp of the command parsing system. If a command inserted multiple entries into chat_events, then the first one was handled by the main code and the subsequent ones were handled directly within the command. Which of course meant that I had to remember to call filter_msg, etc... ugly. Especially if a plugin was going to do that... really not good. So I converted it to an array, and pass the array to plugins, they can add new entries, and then the main code just loops through the array to do filtering... nice and clean (or at least, cleaner by far).


• WEB: Also adding preliminary support for filter plugins, because that just makes sense.

Been working on porting StormChat to PostgreSQL. Managed to get the initial bulk of things done, though there is some work left. Mainly my LEFT JOIN statements that I use in MySQL don't work in PostgreSQL. I'll have to dig into this further.

I also need to brush up on how LIMIT is handled, because I know that there are some differences in how PostgreSQL handles them.

I have the Momjian elephant book, though, so it shouldn't be too bad. The worst part was PostgreSQL's C API (libpq) not having an equivalent to mysql_fetchrow... you can't get an array of the columns in a row, you can only query column by column in the row. My current implementation creates a pointer array; this is a memory leak as it is never freed. Something to fix later; it's not like pointers take up much space so I'm not too worried about it at the moment, but it is something that will need to be dealt with eventually.

Also PostgreSQL does not have an MD5 function, so I had to move that code back out into PHP. No sweat.

After running into a quagmire of XSS issues with FF 1.5, I have managed to get client-pull working with FF 1.5. Basically since client-pull is all served from the same website, then there is no need to set document.domain at all. So I pass cpsp to a few more pages and don't set it if cpsp is not set to 1.

Server push is still broken with 1.5, but the Mozilla folks have basically broken changes to document.domain in so many ways (see my post on my journal with test case) that I'm content to say "it's a Mozilla bug" and tell folks to use client pull instead.

Fixed client pull scrolling as well... added a basically blank html page that tracks the value of the scrolling policy. This makes for a pretty content Stormy, since I can actually use chat again without having to use IE.

Some ideas I was thinking about...

The topics page and avatar pages use meta-refreshes. This is bad for bandwidth when they aren't necessary.

Change SendToChat(msg) to SendToChat(msg, reload).

If reload = 0, do nothing.
If reload = 1, reload the topic page.
If reload = 2, reload the avatars page.

This allows me to very easily tag calls which modify the topic and the avatar list and only reload them when needed. It's also easy for the server code to tell the browser to reload the avatar list when someone is disconnected...