Wikimedia technical discussions

Templates Explained?

Wed, 08 Nov 2006 03:19:55 GMT

I'm interested in learning how to create complex templates for my wiki, but I don't find Wikimedia's Template FAQ useful at all. Does anyone know of somewhere that has a more user-friendly explanation of template creation?

Squid replacement?

Thu, 14 Sep 2006 03:16:23 GMT

Hey!
Just wanted to point at varnish.
I don't think they have had an official release yet but it looks like there is a lot of promise.

sourceforge: http://sourceforge.net/projects/varnish/
homepage: http://varnish.projects.linpro.no/

I Broke It Somehow...

Sat, 01 Apr 2006 14:18:48 GMT

I've been a happy user of an offline (personal use w/ EasyPHP) MediaWiki for several months, but in the past week I've been having a problem. I can make edits and see those edits on the edit page... but the actual page itself doesn't update. The edits are saved. I can shut the machine down, reboot, and then see the changes on the edit page. But the actual article page won't update.

My setup:
WinXP Home SP2
Firefox (also tried in IE)
(according to the Special:Version page...)
MediaWiki: 1.5.6
PHP: 4.3.10 (apache)
MySQL: 4.1.9-max

I've tried stopping and restarting EasyPHP (PHP/MySQL), dumping the internet cache, and the thing of it is.... I don't recall making any changes to my system.

Any suggestions would be welcome. (Or if there's a better forum for asking this question...) Thanks.

~ Isis

Wikimedia Projects

Mon, 13 Feb 2006 20:16:00 GMT

For those who weren't aware, one of the Florida servers fell over this morning - apparently due to physcial hard disk failure - which caused onward problems throughout the server farm resulting in a need for multitudes of reboots and relinking. This happened around 11pm GMT and techs from west coast and australia (amongst others) were up during the middle of their nights to sort it. Many of the issues have been sorted but there are still some on-going problems, as such no wikipedia or wikimedia project should be considered as stable for the moment.

initial thoughts on the T2000

Sat, 11 Feb 2006 12:11:18 GMT

so we received a T2000 for evaluation a couple of days ago. i spent yesterday fiddling with this and running some benchmarks.

first thoughts:

it shipped with Solaris 10 3/05 HW2 and the Java Enterprise System pre-installed, rather than the current 1/06 release. it wasn’t hard to upgrade it to 1/06 remotely, but i found this a bit surprising.
the disk was partitioned with Sun’s standard pre-installed scheme, which left 50GB of the first disk and the entirety of the second disk available. setting this up as a mirrored disk was quite simple
the ALOM - remote management - is very nice. you can connect to it via a serial console or the network, power cycle the machine, connect to its console, and a few other things. the SPARC OpenBoot PROM is much more suited than a PC BIOS to this kind of remote management, since it’s controlled entirely via a command line. in general, managing the server was very pleasant.

more importantly, though, how does it perform? from benchmarks seen elsewhere, i was expecting the T2000 to be as fast as, or faster than, our existing dual-Opteron systems - the 8-core, 1.2GHz model is reported to be comparable to or faster than a 2-CPU dual core Opteron for web serving workloads. ours is a 6-core, 1.0GHz version, so this seems like a reasonable expectation.

i started by installing our standard Apache software on the machine. software versions are as follows:

SunOS hydra.pmtpa.wmnet 5.10 Generic_118822-27 sun4v sparc SUNW,Sun-Fire-T200
Server version: Apache/1.3.33 (Unix)
PHP 5.1.2 (cli) (built: Feb 10 2006 17:26:41)
/opt/mysql/mysql/bin/mysql  Ver 14.12 Distrib 5.0.18, for sun-solaris2.10 (sparc) using readline 5.0

i compared this to a standard Linux host, srv54:

Linux srv54 2.6.9-1.667smp #1 SMP Tue Nov 2 15:09:11 EST 2004 x86_64 x86_64 x86_64 GNU/Linux
Server version: Apache/1.3.33 (Unix)
PHP 5.1.1 (cli) (built: Jan 13 2006 08:11:22)
mysql  Ver 11.18 Distrib 3.23.58, for redhat-linux-gnu (x86_64)

i then than “ab” against both machines (running MediaWiki, our PHP web application), using an increasing number of threads. the results were as follows:

threads	req/s (Hydra)	time/req (Hydra)	req/s (srv54)	time/req (srv54)
1	2.34	455 ms	9.38	100ms
2	4.84	208 ms	17.84	54 ms
4	8.87	130 ms	25.05	37ms
6	13.7	71 ms	29.05	33 ms
8	16.5	59 ms	27.45	40 ms
12	20.32	47 ms	29.27	33 ms
16	19.98	48 ms	29.79	33 ms
24	19.29	53 ms	29.16	33 ms

hmm. the T2000 seems to be almost a third slower than the Opteron system. after this, i wondered whether the difference was caused by some network issues, rather than performance issues, so i made a simple, cpu-bound benchmark:

<?php 
  for ($i = 0; $i < 100000; $i++) {
    $n += $i * (int)($i / 2);
  }
  print $n;
?>

the T2000 consistently served this at 40 req/s, while the Opteron managed 60 req/s. so, about a third slower seems a reasonable result.

this is obviously a very rough benchmark, but i don’t think it too likely that more fiddling will produce a more impressive result (if anyone has any suggestions, though, please let me know!)

is it worth buying, then? well, a 6-core, 1.0GHz T1000 with 2GB RAM costs $2,995 list price, while the dual Opteron system with 4GB costs $2,613. an 8-core T1000 with 8GB is $6,995. a T2000 is even more expensive ($12.395 for 8 cores, 8GB). so, the T1000 isn’t too far off - and a 2/4GB, 8-core model might be even better, if i could find a price - but it’s still a bit more expensive. Sun claims that the T1 CPU saves money on pack space and power rather than initial cost. well, i haven’t managed to measure how much power the T2000 is using yet, but i’ll see if i can get our colo person to do that at some point soon. i think it’ll have to save a lot of power to make it a worthwhile purchase, though.

update (2006-02-28):

so based on Sun's SPECweb2005 submission for the T2000, i made some changes to /etc/system:

set rlim_fd_max=500000
set rlim_fd_cur=500000
set sq_max_size=0
set ipge:ipge_bcopy_thresh = 1536
set ipge:ipge_dvma_thresh = 1536
set ipge:ipge_tx_syncq=1
set consistent_coloring=2

and IP parameters:

psradm -i 1 2 3 5 6 7 9 10 11 13 14 15 17 18 19 21 22 23
ndd -set /dev/tcp tcp_smallest_anon_port 2048
ndd -set /dev/tcp tcp_conn_req_max_q 102400
ndd -set /dev/tcp tcp_conn_req_max_q0 102400
ndd -set /dev/tcp tcp_max_buf 4194304
ndd -set /dev/tcp tcp_cwnd_max 2097152
ndd -set /dev/tcp tcp_recv_hiwat 400000
ndd -set /dev/tcp tcp_xmit_hiwat 400000
ndd -set /dev/ipge instance 1,2,3,4,5,6,7,8,9
ndd -set /dev/ipge rx_intr_pkts 600
ndd -set /dev/ipge rx_intr_time 600

after this, the T2000 serves Special:Version at 29.75 requests/sec, vs 28.21 requests/sec for the Opteron system (at a concurrency of 40). so, it's now slightly faster, but still quite a bit more expensive: assuming an 8-core T1000 can process 38.65 requests/sec, it works out to $180/req/s, vs $92/req/s for the Opteron. a 6-core T1000 would be $100.67/req/s - close, but not quite there.

Tim's also run some more benchmarks focusing more on raw CPU power than transactional speed.

update (2007-06-21):

frostcs mentioned below that performance for this kind of PHP application might have improved a lot recently. the impression we got from talking to Sun at the time was that PHP5 had a particular problem with concurrency that caused the poor performance. unfortunately, we don't have this machine anymore, but i'd be interested to see the results if anyone else can repeat the benchmark.

PHP references must die

Fri, 10 Feb 2006 09:42:30 GMT

Debugging an oddity in a new special page, I found this gem of a behavior... using array references is not such a hot idea. Take this simple case, of an outer loop and an inner loop iterating over the same array:

echo "Nested iteration over a short array:\n";
$set = range( 1, 3 );
foreach( $set as $i ) {
	echo "- $i\n";
	foreach( $set as $j ) {
		echo " -> $i, $j\n";
	}
}

This works just like you'd expect:

- 1
 -> 1, 1
 -> 1, 2
 -> 1, 3
- 2
 -> 2, 1
 -> 2, 2
 -> 2, 3
- 3
 -> 3, 1
 -> 3, 2
 -> 3, 3

But here's the case my code was encountering:

echo "\nSame thing, but the variable was set as a reference:\n";
$ref =& $set;
foreach( $ref as $i ) {
	echo "- $i\n";
	foreach( $ref as $j ) {
		echo " -> $i, $j\n";
	}
}

which renders:

Same thing, but the variable was set as a reference:
- 1
 -> 1, 1
 -> 1, 2
 -> 1, 3

Apparently if the thing you're iterating over happens to have been set as a reference, it happily stomps over the array cursors, and the outer loop ends after the first iteration of the inner loop. Nice!

Really, references in PHP just plain suck. At least with PHP5 we'll have the chance to get rid of most of them, since object assignment will work as you'd expect instead of a copy...

HOWEVER!

The real ugly part comes when you realize that the "global" statement actually works via hidden references:

function iter() {
	echo "\nSame thing, but the variable is a global:\n";
	global $set;
	foreach( $set as $i ) {
		echo "- $i\n";
		foreach( $set as $j ) {
			echo " -> $i, $j\n";
		}
	}
}

Since global $set; is equivalent to something like $set =& $GLOBALS['set'];, you get the broken reference behavior even though you thought you were doing something that made sense.

Thu, 09 Feb 2006 13:22:16 GMT

so we just got our hands on a nice new Niagara server for evaluation:

SunOS hydra.pmtpa.wmnet 5.10 Generic_118822-27 sun4v sparc SUNW,Sun-Fire-T200

i’ve not had time to benchmark it yet, but first impressions are good... the ALOM management is useful (and a lot better integrated than Sun’s AMD stuff). it’s a bit more expensive than the equivalent AMD stuff, but hopefully if it’s fast enough we can save a bit of money on rack space and power (which we’re using an awful lot of right now) - one 1RU T1000 is rumoured to be as fast as two dual-CPU Opterons, as well as using a third of the power and half the space...

MSIE strikes again

Wed, 08 Feb 2006 10:27:10 GMT

So I'm trying to clean up the MSIE hacks stylesheet to get right-to-left languages on Wikipedia working properly with IE7 Beta 2. I end up with a couple style rules that I want to apply conditionally when dir="rtl" is present on the <html> element, and IE7 now supports CSS attribute selectors, right?

Well... sort of.

I try something like html[dir="rtl"] #column-one and have no luck; it just doesn't apply. After some further testing, I find that apparently the dir attribute is some kind of magic exception -- you can make selectors on random attributes you make up if you like, but you can't make them with dir!

Here's a little test page I whipped up, and rendering in IE7 beta 2 vs Firefox 1.5:

Sigh...

Posted a report on MS's feedback newsgroup.

Fri, 06 Jan 2006 21:47:24 GMT

I'm having a problem with specifying a default for a parameter in a template.

If I give the parameter in the template as: {{{Param|None}}}
When the value is unspecified in a page I get: {{{Param|None}}} (printed plaintext)
When the value IS specified in a page I get: {{{Param|None}}} (printed plaintext)

If I give the parameter in the template as: {{{Param}}}
When the value is unspecified in a page I get: {{{Param}}} (printed plaintext)
When the value IS specified in a page I get: (the value printed plaintext)

Basically it doesn't seem to recognize the parameter-with-default as being a parameter at all, it just prints it out plaintext regardless of whether it is specified or not. If I don't give a default, then without a value it prints the plaintext parameter, and with a value it prints the parameter properly.

Does this make any sense? What might I be doing wrong? I have read the appropriate meta help carefully and as far as I can tell I'm specifying my default properly.

Thanks in advance for the help!

site status

Sun, 27 Nov 2005 20:02:59 GMT

Edward Z. Yang (GPGKey:0x869C48DA) is working on an interface to show up wikipedia status: http://www.thewritingpot.com/wikistatus/

It's currently alpha but you can probably test it and report bugs / wishes & issues to him. One day, maybe it will replace the old berlio status page at http://openfacts.berlios.de/index-en.phtml?title=Wikipedia_Status

squids mrtg is no more !

Tue, 15 Nov 2005 20:38:32 GMT

Hello,

A long time ago, gwicke set up mrtg to monitor squids usage. The system was available at http://wikimedia.org/stats/live/ .

Someone else is generating new statistics and take care of Amsterdam, Seoul and Paris cluster as well. So you should update your bookmarks and have a look at the new system:

http://noc.wikimedia.org/stats.php

commons file server switched

Tue, 01 Nov 2005 00:56:29 GMT

Yesterday, Brion announced a switch of the file server for http://commons.wikimedia.org/ . Should enhance download of pictures for all wikimedia projects (as long as the picture is on common).

Commons is a wikimedia foundation project that aims to provide free multimedia materials. If you never browsed it, you should have a look at it RIGHT NOW.

I even got a shell script that grab the picture of the day and set it as my wallpaper :)

horsepower!

Thu, 22 Sep 2005 08:04:31 GMT

new application server boxes entering service:

20 dual-opteron servers...

but we still have to repair old db servers and obtain/deploy new ones.

default search all?

Wed, 21 Sep 2005 09:27:10 GMT

Hi.

My library is going to be using a wiki for subject guides. We're using templates as side bar info thingies. Is there a way to set the search settings to default to searching templates as well, WITH OUT having to be logged in? We want our users to be able to find info in these templates with a minimal amount of effort.

any help is appreciated!

Fri, 09 Sep 2005 00:38:33 GMT

well, the first three machines in the new Yahoo! cluster are now online. after some issues with installing the rest of them are resolved, we should have a nice shiny new cluster to do things with.

Wikimedia download portal updated

Fri, 26 Aug 2005 19:40:07 GMT

I changed the download portal at http://download.wikimedia.org/ . Should be easier to find and download the dump.

Portuguese translations updated

Thu, 25 Aug 2005 14:49:33 GMT

Hello !

Thanks to Get_It, we updated the Portuguese translation files on WikiMedia servers. The old one was Brazilian portuguese and got moved under PT_br.

Every Portuguese speaker should now have a correctly translated interface on every WikiMedia sites.

If you want to help with MediaWiki localisation, have a look at the guide: http://meta.wikimedia.org/wiki/MediaWiki_localisation

1.6 alpha

Sun, 21 Aug 2005 11:25:02 GMT

The sites are now running MediaWiki 1.6 alpha, the CVS HEAD.

This means new features will appear live on the sites much more quickly. On the other hand changes will have to be tested thoroughly, possibly on their own experimental branch.

Users will see little new with this change for the moment, except a kind of 'edit conflict' page if a page you're editing gets deleted.

More from Tim Starling: http://mail.wikipedia.org/pipermail/wikitech-l/2005-August/031103.html

Mon, 15 Aug 2005 19:45:06 GMT

well, we've now had two of our nine Sun v20zs fail (in two months), the second one leaving mail offline for the last day.

i shall think twice about recommending Sun x86 hardware in future ;)

Tue, 09 Aug 2005 23:58:39 GMT

i've been fiddling with mail.wikimedia.org a bit recently. while moving it to another machine, i thought i'd look at adding some anti-spam filtering, for those poor mailing list admins. i've written a brief thingy here describing how it works, if anyone's interested. seems quite effective so far...

Memory leaks are awesome

Sun, 10 Jul 2005 06:43:02 GMT

[21:12] 	innocence:	brion!
[21:12] 	innocence:	your lucene updater is using 3 trillion MB of ram

Mon, 04 Jul 2005 11:14:00 GMT

we now have the webalizer access stats back, although only for hits that go through knams at the moment, so it only includes a subset of wikis and users. at some point the rest of them should appear there...

edit: stats are now for all wikis, and per-country stats are available too.

Upgradey!

Sun, 26 Jun 2005 05:06:40 GMT

woo

Thu, 02 Jun 2005 13:30:45 GMT

some pics of the new dutch servers. more to be written on this later...

Wed, 25 May 2005 01:38:36 GMT

yeah, site's broken. some kind of issue with albert (fileserver)... looking at it at the moment.

looks like i'm the only person around. excellent timing.

edit: fixed (it'll take a few minutes for it to recover). somehow eth1 died - doesn't look like the cable was unplugged, but there's no traffic on it. we use eth1 for internal (including NFS) and eth0 for external traffic, so the apaches couldn't load any files, and hence nothing worked.

normally, this would be a real PITA, but we've recently started using a neat Linux feature, 802.1q (VLAN trunking). this is normally used to bridge vlans between switches or routers, but you can use it on hosts too. so i configured albert's eth0 on the switch as a trunk port, and set up the new eth0.2 interface to handle internal traffic, and it magically works. clever.

so, apologies for the interruption, your regular Wikipedia service will resume shortly.