Tigers, and lions, and bears, oh my!

Shame on me

Fri, 26 Dec 2008 07:57:15 GMT

Shame on me. I had a mistake in my previous pos. Thanks to xaprb for pointing this out.

I misplaced quote marks in my second query.

Instead of


date BETWEEN '2008-01-01 12:00:00.000' AND '2008-01-14 12:00:00.000' AND accomodation=1

I wrote this:


date BETWEEN '2008-01-01 12:00:00.000' AND '2008-01-14 12:00:00.000 AND accomodation=1'

Note the last quote mark after accomodation=1

:)

Those evening queries...

Wed, 24 Dec 2008 09:35:53 GMT

I'm dealing with databases right now while keeping an eye on «High performance MySQL». Finally decided to try something out. Here we go...

We have a table split into partitions be week:


CREATE TABLE `price_detail` (
  `accomodation` int(11) NOT NULL,
  `date` datetime NOT NULL,
  `price` int(11) NOT NULL,
  `room_id` int(11) NOT NULL,
  KEY `idx_price_date` (`date`) USING BTREE,
  KEY `idx_price` (`price`) USING BTREE,
  KEY `idx_price_accomodation` (`accomodation`) USING BTREE
) 
PARTITION BY RANGE (TO_DAYS(`date`)) (
PARTITION p20080101 VALUES LESS THAN (733407), 
PARTITION p20080108 VALUES LESS THAN (733414),
/* and so on — an entire years worth */

Yep... We have indexes as well...

We have 26 million rows per week in the worst case. Needless to say that we have the worst case for testing purposes .

Let's see how a simple query will work on such a database:

Query I:


explain partitions 
  select 
    accomodation 
  from 
    price_detail 
  where 
    accomodation=1 and date BETWEEN '2008-01-01 12:00:00.000' AND '2008-01-14 12:00:00.000'\G;

*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: price_detail
   partitions: p20080108,p20080115
         type: ref
possible_keys: idx_price_date,idx_price_accomodation
          key: idx_price_accomodation
      key_len: 4
          ref: const
         rows: 40292
        Extra: Using where

Query II:


explain partitions 
  select 
    accomodation 
  from 
    price_detail 
  where 
    date BETWEEN '2008-01-01 12:00:00.000' AND '2008-01-14 12:00:00.000 AND accomodation=1'\G;

*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: price_detail
   partitions: p20080108,p20080115,p20080122,p20080129,p20080205,p20080212,p20080219,
                     p20080226,p20080304,p20080311,p20080318,p20080325,p20080401,p20080408,
                     p20080415,p20080422,p20080429,p20080506,p20080513,p20080520,p20080527,
                     p20080603,p20080610,p20080617,p20080624,p20080701,p20080708,p20080715,
                     p20080722,p20080729,p20080805,p20080812,p20080819,p20080826,p20080902,
                     p20080909,p20080916,p20080923,p20080930,p20081007,p20081014,p20081021,
                     p20081028,p20081104,p20081111,p20081118,p20081125,p20081202,p20081209,
                     p20081216,p20081223,p20081230,p20090106,p20090113,p20090120,p20090127
         type: ALL
possible_keys: idx_price_date
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 274845000
        Extra: Using where

What a nasty surprise!

I'm off learning about covering indices...

For whom the headhunt tolls...

Fri, 17 Oct 2008 08:17:00 GMT

We're hiring now.

The first line in job description reads: Good or excellent knowledge of PHP

Here's what we get: "I've just started learning РНР...»

Perhaps, we should've stated our requirements more clearly... Somehow... :)

A hundred wise men

Fri, 29 Aug 2008 10:31:30 GMT

via http://andrzejn.livejournal.com/1152676.html

- A hundred wise men know that something is impossible. Then a fool comes along...

- And he makes a discovery?

- No. However, the wise men remember him with tenderness for long afterwards.

Musings

Thu, 24 Apr 2008 13:08:16 GMT

As I promised here are my thoughts:

1. First of all, pattern matching lets you describe what yo eed with little or no effort. For example, consider these rules:

{field}
{field, Func}
{field, {'=', field2}}

Frankly, I have a very vague idea how these could be matched using ifs only. I guess you would create a Rule class and inherit a slew of classes from it. Something like RuleField, RuleFunc, RuleOperator and so on. If you use pattern matching, however, the rules are easily parsed::

%%{field}
validate_rule({FieldName}) ->
    ok.

%%{field, Func}
validate_rule({FieldName, Func}) when is_function(Func) ->
    ok.

%%{field, {'=', field2}}
validate_rule({FieldName, {Operator, FieldName2}}) ->
    ok.

Such ease, however, may bring (and it does bring me) to my second thought:

2. WTF-ish code. The thought that you can easily slice through complex constructs makes you write before you think.

In my case I knew what I wanted to pass to the function, but I had no idea what I expected the function to return. Ok, I've parsed the rules, what now?

As a result, the first version of the function would return a deeply nested list of lists that looked something like this:


[[Field, [Error1, Error2]], [Field2, [Error3, Error4]]]

It took me two additional refactorings to make it return a properproplist.

This same "wow, look at how I handle things!" approach resulted in an ugly preprocessing of results before I return these to the user:


lists:filter(
    fun(Elem) -> 
        case 
            Elem of {} -> false; 
            {_, []} -> false;
            _ -> true 
        end 
    end, 
    lists:flatten(validate1(A, ValidationRules, [])))

Yup. Get rid of those unwanted elements before the user sees them. Are you scared? I am. I m saddened as well :(

Here's where my train of thoughts stops...

Validation for erlyweb

Thu, 24 Apr 2008 07:54:44 GMT

I've written a validation function for Erlyweb. First I'll tell you how it works and then, perhaps, more thoughts on its internals.

Create a form with four fields:
- login
- password
- pasword_repeat
- email

This is a pretty standard registration form. Naturally we'd have to validate input coming from this form:
- login has to be 4-16 symbols in length
- password has to be 4-16 symbols in length
- password has to be the same as password_repeat
- email has to be a valid email address

Erlyweb's validation functions cant cope with this. My function can :)

Suppose you have a function called process_signup, which accepts the yaws_arg record. Then the validation will look like so:

process_signup(A) ->
	F = fun(A, Field) ->
		{ok, Val} = yaws_api:postvar(A, Field),
		L = string:len(Val),
		if
			L < 4 orelse L > 16 ->
				{Field, length};
			true ->
				{}
			end
	end,
	EmailCheck = fun(Args, Field2) ->
		{ok, Email} = yaws_api:postvar(Args, Field2),
		Match = regexp:match(Email, "^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]+"),
		Match /= nomatch
	end,

	%% magic is here:
	buktu_form:validate(A, [
		{login, F},
		{email, EmailCheck},
		{password, [{'=', password_repeat}, F]}
	]).

If we don't input any field at all, we'll get back the following list:

[{login,invalid_field},
 {email,invalid_field},
 {password,[{invalid_fields,[password,password_repeat]},
            invalid_field]}]

If we input values that don't match our criteria, we'll get:

[{login,length},
 {email,invalid_value},
 {password,[{not_equal,password_repeat},
            length]}]

The callback function that you can pass can return the following:
- a tuple {FieldName, Error}
- true if the field is validated and false otherwise (then the validation function will return {FieldName, invalid_value})
- any value Value which will be transformed into {FieldName, Value}

if any of the fields in the rule don't exist or are empty, for each such rule the validation function will return:
- invalid_field if the field is compared against a value
- {invalid_fields, [field1, field2]} if the field is compared against another field

If you need to bind several rules to a field, pass a list of rules. If you just need to check if a field exists, pass in a tuple containing the field's name:
- Does the field exist?
  {field_name}
- Compare a field field_name to field field2_name (you can use '=', '/=', '<', '=<', '>', '>=' )
  {field_name, {'=', field2_name}}
- Compare a field to any value:
  {field_name, {'=', Value}}
- Use a callback function (function/2, first parameter is yaws_arg, the second is the field's name). The function can be a lambda or any function of any module in the form of module:function/2 or {module, function}
  {field_name, F}
- Use a callback function with an additional value (function/3,first parameter is yaws_arg, the second is the field's name, the third is the value). The function can be a lambda or any function of any module in the form of module:function/3 or {module, function}
  {field_name, {F, Value}}

The validation function returns a proplist:

[{FieldName(), Errors()}]


where
FieldName() = atom()
Errors() = Error() | [Error()]
Error() = user_defined_values | absent | invalid_value | 
          invalid_field | {invalid_fields, [FieldName(), FieldName()]} |
          ComparisonError()
ComparisonError() = {not_equal, value_or_field} | {equal, value_or_field} |
                    {not_greater, value_or_field} | {not_less, value_or_field} |
                    {not_greater_or_equal, value_or_field} | {not_less_or_equal, value_or_field} |

Apple...

Thu, 06 Mar 2008 15:27:24 GMT

there it is.... quite unusual, but there is no way back...

Somebody stop me

Fri, 24 Aug 2007 07:44:46 GMT

Or shoot me before I suffer too long :)

I've managed to get one of the oldest Firefox bugs going: https://bugzilla.mozilla.org/show_bug.cgi?id=69230 (see comment 86). I have know idea what to do with it now :) And there's the possibility that I will have to compile Mozilla from the source...

"Tell my wife and children that I love them..." :))

Erlang Bit Syntax

Tue, 21 Aug 2007 13:39:02 GMT

I've decided to see whether all is as well with Erlang's bit syntax as advertised..

A friend of mine has been struggling with SOny's OMA format which is being used in Sony's players. The only feature this format has that distinguishes it from "normal" formats is its lack of documentation and whatever documentation there is differs significantly from whatever actual files have to offer :)

There are some good folk out there and they've reverse engineered the format and wrote a Java program to maniplate it. The program is available here: http://dmitriid.com/files/projects/erlang/OMA.zip. The program is accompanied by documentation on Sony players' directory structure and, what's more important, on OM file format as well.

You can see the OMA header format in this file: http://dmitriid.com/files/projects/erlang/OMA.html. It doesn't look to horrible, does it? Let's see how Erlang handles it.

The first obstacle comes from the sample file that comes with the application (trunk/dist/OMGAUDIO/10F00/10000001.OMA). The spec says it has to start with

"E" "A" "3" 3 0 0 0 0 17  76 "T" "I" "T" "2" 0 0

You wish. In reality it starts with

"e" "a" "3" 3 0 0 0 0 17 "v" "G" "E" "O" "B" 0 0

And there's lots of info before we reach TIT2. Oh well, hex-editor to the rescue. The files does contain the tags shown in the spec. However, they are placed differently and there's a bunch of other, unknown tags. Whatever shall we do? I propose moving byte by byte until we reach tags that we know of or the end of the header.

I'll tell you from the start that I cheated. I don't read the codec info from the header, I stop as soon as I reach

"E" "A" "3" 2 0 60 ff ff

So, the code

Open the file and read data from it

parse_header(File) ->
    case file:open(File, [read, binary, raw]) of
        {ok, S} ->
            {ok, Header} = file:pread(S, 0, 16#0c60),
            H = case read_header(Header) of
                error ->
                    {error, invalid_header};
                Data ->
                    Data
            end,
            file:close(S),
            H;
        _ ->
            {error, file_cannot_be_opened}
    end.

Well, that was easy. Open the file. Read it. Call internal read_header function to actuall parse the header and return the value that we receive from the function.

The read_header function couldn't be simpler

read_header(<<$e, $a, $3, Rest/binary>>) ->
    Data = decode_header(Rest, []),
    Data;

read_header(_) ->
    error.

It couldn't be simpler because of the pattern-match. If the chunk of data that comes in starts with

"e" "a" "3"

it's passed to the upper function. Otherwise, the lower function is called.

The actual parsing is contained in several decode_header functions which parse a single tag at a time. Here' an example:

% Title
decode_header(<<$T, $I, $T, $2, _:2/binary,
      Var:2/integer-unit:8, _:4/binary,
      Rest/binary>>, L) ->

        TitleLength = Var - 2,
        <<Title:TitleLength/binary, Rest2/binary>> = Rest,
        decode_header(Rest2, [L|{title, Title}]);

What happens here? Keep the spec before your eyes.

Here we go. The song title is stored as follows

T I T 2 0 0 Var1 Var1 0 0 2 TitleString

where TitleString is [0x0 String]. That's exactly what we've specified in the pattern:

T        $T
I        $I
T        $T
2        $2
0        _:2/binary            simply skip the two zeroes
0
Var1     Var:2/integer-unit:8  a number containd in two bytes
Var1
0        _:4/binary            simply skip 0 0 2 0
0
2
0
TitleString Rest/binary

All we have to do now is get the title from Rest. Empirically I've guessed that the length of this title in our case is Var - 2. Having decided on that, we can get the title using pattern matching, again:

        TitleLength = Var - 2,
        <<Title:TitleLength/binary, Rest2/binary>> = Rest,

This way all we have to do is to parse all the remaining tags according to the rules and test them against a real file. The real file may give us a headache because it doesn't confirm to the spec (albeit informal) and adds several new and unknown tags. That is why we need to augment decode_header with:

decode_header(<<>>, L) ->
    L;

decode_header(Bin, L) ->
    {_, NewBin} = split_binary(Bin, 1),
    decode_header(NewBin, L).

We move forward byte by byte. If we match a known tag, it is caught by the corresponding decode_header. If we've reached the end of the header, this will be natched by decode_header(<<>>, L). Otherwise we move forward by one more byte and repeat the steps nce again.

The entire file is available here: http://erltag.googlecode.com/svn/trunk/src/oma.erl. If you want to practice, download this file: http://erltag.googlecode.com/files/erltag-release.zip(6.1 MB). You can find the sample audiofile in the test folder.

By the way. The code doesn't pretend to be the most effecient and correct way of writing Erlang code. Most likely the code is rather opposite of that :)

Programming Erlang

Tue, 31 Jul 2007 11:44:38 GMT

It has finally arrived!

It arrived at Moldova on Juky 21st. It got out of customs on July 25th. And it is only today that I finally noticed the invitation to pick it up at my local post office. :)

Hurrah!

Now if I only could find time to study it...

Keep'em languages coming!

Wed, 18 Jul 2007 15:12:18 GMT

Past Friday I got myself acquainted to Python.

Different people view Python as "nothing to learn" or as "if only I had time to learn" or even as "very difficult to learn".

I had to learn it quick for a very simple reason. I have lots of photos taken at different resolutions. I need to resize them to different sizes (much like Flickr does), create several thumbnails of each image, upload them to a host (Amazon S3) and load information on uploaded images to a database.

The process must be automated, of course.

It all started when I convinced myself that Python Imaging Library is the best library to manipulate images. Indeed:

import Image
im = Image.open('path/to/image')
im.thumbnail((800, 800), Image.ANTIALIAS)
im.save('path/to/image')

The line with im.thumbnail((800, 800), Image.ANTIALIAS) is absolute genius. If your image is, say, 1024х768px, then what you get in the end is not a skewed 800х800px, but a proportional 800x600px.

Combined with MySQLDb and Boto this yields a very easy solution to my task. However, that's not the point.

As it turns out, getting to know many languages (even superficially) may give you a huge head start in learning a new programming language. I've had an overview or, sometimes, more than an overview of such languages as Lisp, Haskell, Erlang... How does that help me with Python? Well, I got an imeddiate understanding of the following:

Note: I can bear no responsibility for Haskell codes in the examples :)

Tuples and destructuring assignment

A tuple is a fixed-length list. This list may contain elements that are very different in nature:

# Python tuple
(1, 2, 'hello')

-- Haskell tuple
(1, 2, "hello")

%% Erlang tuple
{1, 2, 'hello'}

Looks similar, doesn't it? :)

Let now get data out of our tuples:

# Python
# А will contain 1
# В will contain 2
# С will contain 'hello'
A, B, C = (1, 2, 'hello')

-- Haskell
-- a will contain 1
-- b will contain 2
-- c will contain 'hello'
(a, b, c) = (1, 2, "hello")

%% Erlang
%% А will contain 1
%% В will contain 2
%% С will contain 'hello'
{A, B, C} = {1, 2, 'hello'}

Certainly, the principles of theses examples are very different. Haskell and Erlang use pattern matching that Python doesn't have. However, it looks very similar and is very easy to understand once you know the other examples. PHP, by the way, has a similar thing:

// А will contain 1
// В will contain 2
// С will contain 'hello'
list($A, $B, $C) = array(1, 2, 'hello')

This, of course, is very different and the inner workings are absolutely different :) One thing about this is quite peculiar, though. Once you get used to these sorts of destructuring assignments, you start using PHP's list and Python's tuples much more often. Because it's so convenient.

Named parameters

I once talked about named parameters. As it turns out, Pythons has them.

# define a function
def myfunc(param='', another_param=1)
print param, another_param
# call the function
# will print hello 5
myfunc(another_param=5, param='hello')
# will print hello 1
myfunc(param='hello')
# will print 1
myfunc()

Anonymous functions (lambdas)

# Python
# define a list
li = [1, 2, 3, 5, 9, 10, 256, -3]
# get only evens out of the list
li_even = filter(lambda item: item % 2 == 0, li)

Very similar to code in other languages:

-- Haskell
-- define a list
li = [1, 2, 3, 5, 9, 10, 256, -3]
-- get only evens out of the list
li_even = filter (\item -> (mod item 2) == 0) li

%% Erlang
%% define a list
Li = [1, 2, 3, 5, 9, 10, 256, -3]
%% get only evens out of the list
Li_even = lists:filter(fun(Item) -> Item rem 2 == 0 end, Li)

List comprehensions

Remember how we used to define lists in mathematics? For instance:

S = [1, 2, 3, 4]
M = {x | x in S, x even}

M is all x's where x belongs to S and x is even. Nice languages have ways to define lists in a very mathematical way. This is called list comprehensions. One downside to Python's comprehensions is the syntax. Here are previous examples rewritten to use list comprehensions instead of lambdas:

# Python
# define a list
li = [1, 2, 3, 5, 9, 10, 256, -3]
# get only evens out of the list
li_even = [x for x in li if x % 2 == 0]

-- Haskell
-- define a list
li = [1, 2, 3, 5, 9, 10, 256, -3]
-- get only evens out of the list
li_even = [x | x <- li, (mod x 2) == 0]

%% Erlang
%% define a list
Li = [1, 2, 3, 5, 9, 10, 256, -3]
%% get only evens out of the list
Li_even = [X || X <- Li, X rem 2 == 0]

Anyhow, thanks to other languages the new language (a more or less mainstream language, not K :)) ) is not just easy, it's extreamly easy and can be learned in, say, 15 minutes.

Update: Corrections to Haskell code. Thanks, deni_ok

Pragmatic Erlang

Mon, 28 May 2007 11:04:35 GMT

Almost 60 dollars left my pocket to get Joe Armstrong. "Programming Erlang. Software for a Concurrent World". It's finally mine! As a PDF only so far. I'll get to reading it eventually......

Yeah, baby! :)

NY

Fri, 29 Dec 2006 13:39:55 GMT

HAPPY NEW YEAR TO ALL FRIENDS AND ENEMIES!!!

Orcas official site

Sun, 08 Oct 2006 08:37:35 GMT

Orcas has been moved to http://orcas.dmitriid.com/

Project description etc etc can be accessed via project wiki at http://orcas.dmitriid.com/wiki

Refactoring, pattern matching, function guards and Erlang

Wed, 04 Oct 2006 09:01:21 GMT

I'm still late on my promise to talk about pattern matching even though today's post is related to patern matching.

In brief here is what pattern matching all about. The all-knowing Wikipedia defines pattern matching as

the act of checking for the presence of the constituents of rigidly specified pattern

What does this mean though? This means that when a function (especially in dynamically typed languages) accepts parameters of different kinds/types we can define this function like this::

fun([H|T]) ->
    do_smth();

fun({a, tuple}) ->
    do_smth_else();

fun(value) ->
   do_a_third_thing();

fun(DefaultValue) ->
   do_default_thing().

What happens in the code above?

1. First function accepts a list/array as its parameter. The list consists of a Head and a Tail. So, if we pass [1, 2, 3, 4] to this function, the variable H will contain 1 and variable T will contain [2, 3, 4]

2. The second function accepts a tuple, that consists of two immutable values, "a" and "tuple"

3. The third function accepts an immutable value of "value"

4. The fourth function accepts everything rejected by the first three functions

So. If we call a function like this:

fun([2, 5, 72]).

we will trigger the first function, since it accepts an array/list.

However, if we call the function like so:

fun({another, tuple}).

we will trigger the fourth function. Why? Because even though we pass a tuple, the third function accepts only one tuple, {a, tuple} whereas we pass a different tuple, {another, tuple}.

So what's refactoring got to do with all this? And what's function guards?

A couple of things about function guards. Consider the ubiquitous Fibonacci function. First, pattern matching:

fib(1) -> 1; 

fib(2) -> 1; 

fib(X) -> fib(X-1) + fib(X-2).

Now, function guards:

 
fib(X) when X =< 3 -> 1; 

fib(X) -> fib(X-1) + fib(X-2).

The first line reads: "If X is less than 3, return 1".

Guards can check for a wide range of conditions, including such thnigs as is_list(X), is_tuple(X) etc. They can also be stacked using such keywords as "and", "or" etc. More on them in forecoming topics. Back to the topic at hand.

A question was raised over at RSDN with regard to implementation of the Factory pattern. For instance, in creating a wrapper for various database connections. The following С++ code was given as an example:

class Driver
{
public:
  virtual void connect() = 0;
};

class MysqlDriver : public Driver {}; //implements connect() using some mysql API calls.
class OracleDriver : public Driver {}; //implements connect() using some oracle API calls.

And, finally,

class DriverFactory
{
public:
  static Driver * createDriver( string name ) //creates necessary driver
  {
    Driver * driver = 0;
    if("mysql" == name ) driver = new MysqlDriver();
    else if( "oracle" == name ) driver = OracleDriver();
    else throw 1;
    return driver;
  }
};

The key to this factory, is of course the following:

if("mysql" == name ) driver = new MysqlDriver();
    else if( "oracle" == name ) driver = OracleDriver();
    else throw 1;

Ok. Let's rewrite it using Erlang.

First that springs to mind is the obvious line-by-line translation of the C++ code:

start(Driver) ->
    case Driver of
        mysql ->
            % connection happens here
            {};
        mnesia ->
            % connection happens here
            {};
        odbc ->
            % connection happens here
            {}
    end.

It's clear that you would call the function as follows:

start(mysql).
start(mnesia).
start(odbc).

Note that the "case" construct in Erlang also employs pattern matching.

What if connection i a bulky piece of code spanning 10-20 lines? We could employ function guards of course::

 
start(Driver) when Driver == mysql ->
    % connection happens here
    {};

start(Driver) when Driver == mnesia ->
    % connection happens here
    {};

start(Driver) when Driver == odbc ->
    % connection happens here
    {}.

This is more like it. Function call hasn't chaned a bit: start(driver_name). Guards, however, are not expressive enough. You have to read the definition of the function and only after that your eye cathes the guard. Is there a better way to do this? Of course!

Recall that pattern matching involves a "rigidly defined pattern". What's more rigid than

start(mysql) ->
    % connection happens here
    {};

start(mnesia) ->
    % connection happens here
    {};

start(odbc) ->
    % connection happens here
    {};

We can improve on our example and introduce options that will be passed to the connection code:

start(Driver) ->
    start(Driver, []).
        
start(mysql, Options) ->
    % connection happens here
    {};
    
start(mnesia, Options) ->
    % connection happens here
    {};
    
start(odbc, Options) ->
    % connection happens here
    {};
    
start(_, _) ->
    {error, driver_not_supported}.

Last lines simply states that for any other type of connection we through an error. Underscore means "any variable", a sort of "joker".

That's a part of what, to me, is the "Erlang Way". Very shortly and quite inaccurately this could be defined as: focus on what you need. A slight improvement of this definition is described in the concept of "aggressve programming". See. Erlang programming rules for a description.

That is, a program/module/function should do only the things it's meant to do. Under all other circumstances it should fail specifying why it failed, because it really should be the problem of the person who decided to (ab)use the program/module/function in the wrong way. As a result you end up with a small number of functions/modules which only do what they are meant to do. And this is quite handy

The same page lists other things in the "Erlang way", in particular:

- top-down programming (you start with abstractions and work towards details),
- "make it work correctly now and worry about otimizations later",
- principle of least astonishment (the system must behave predictably)
- "aggressive" programming etc.

Actually, this approach can be used - and is used! - in mainstream languages such as C++, Java, C#. However, in order to employ this approach in these languages, you need experience (quite a lot of it some times - a nod to C++). Erlang, on the other hand, encourages this approach right from the start.

I general, I'm not going to give up on Erlang in the forseeable future :) Unless it's in favour of Ne,erle, perhaps :))

erlsoap

Mon, 25 Sep 2006 07:23:17 GMT

Opensource projects, eh? They make you involve in all sorts of things :) Thanks to the ever unpredictable Lady Luck, Anton Fedorov decided to bring the aforementioned erlsoap library up to date.

So the battle ensued between me and the shiny new 0.4 version of erlsoap. This version decided it didn't like RSDN's webservice. After some fruitful discussion with Anton a new, 0.4.1, version of erlsoap appeared. This version, however, didn't suit qrilka in the least :)) The problem lay with the inets library distributed with Erlang and used by erlsoap. Inets, as it turned out, doesn't support proxy authentication.

So what's a developer got to do? I geared up and went to hunt... ehm... to add support for a more advanced ibrowse library. As a result, I've officially added my first patch to an opensource project which even brought me a mention in the soapclient module in erlsoap's new version 0.4.2.

Now I'm tuly happy :)

The cavalry is coming!

Tue, 19 Sep 2006 13:41:51 GMT

Camrade qrilka has shown interest in Orcas. Right now we are not quite sure as to what is the best approach to SOAP in Erlang.

Since RSDN's webservice is quite small, one approach would be to implement all calls and parse all responses by hand. But that's boring, isn't it? :)

Today I've braced myself and went to war with erlsoap, which has been stagnating for the past two years. Well, you can check out the code yourself. The file is test_soap.erl, and you run it by

test_soap:run().

qrilka suggested that we use Smerl. This still leaves quite a lot to do by hand. Anyway, tomorrow, time permitting, we'll take a look into the matter.

Orcas

Fri, 15 Sep 2006 14:45:33 GMT

Well, I've slowly started working on an offline client for RSDN.ru in Erlang.

Here's the project page: Orcas
Here's anouncement of the fact on RSDN (in Russian)

Mon, 04 Sep 2006 07:35:58 GMT

Kosma Prutkov. Fruits of thought. # 66

Many a thing we do not understand not because our reasoning is weak, but because we have no notion of them

Dynamic languages rule

Mon, 21 Aug 2006 14:26:22 GMT

It seems to me that dynamic languages...That is, languages which have more dynamics built in... That is, nice languages to put it simply :)) Well, they rule.

I've been browsing Jakarta Commons' source code using http://www.koders.com/ (I actualy only needed their WordUtils in order to rewrite them in ColdFusion). An then I see Validate.java:

    public static void isTrue(boolean expression, String message, Object value) {
        if (expression == false) {
            throw new IllegalArgumentException(message + value);
        }
    }

    public static void isTrue(boolean expression, String message, long value) {
        if (expression == false) {
            throw new IllegalArgumentException(message + value);
        }
    }

    public static void isTrue(boolean expression, String message, double value) {
        if (expression == false) {
            throw new IllegalArgumentException(message + value);
        }
    }

    public static void isTrue(boolean expression) {
        if (expression == false) {
            throw new IllegalArgumentException("The validated expression is false");
        }
    }

:)) Static typing is the answer to everything they say, but at what cost? :))) I wonder if this could be rewritten... ColdFusion-style:

<cffunction name="isTrue">
    <cfargument name="expression" type="boolean" required="yes">    
    <cfargument name="message" type="string" required="no">
    <cfargument name="value" type="any" required="no">  
	
	<cfset msg=IIF(IsDefined('arguments.message'), DE(arguments.message), DE("The validated expression is false"))>
	<cfset v=IIF(IsDefined('arguments.value'), DE(arguments.value), DE(""))>
	
	<cfif arguments.expression EQ false>
		<cfthrow type="IllegalArgumentException" message=msg & ToString(v)>
    </cfif>
</cffunction>

Key moments here are required="true|false" in argument definitions and the IsDefined('arguments....') function which, well, defines whether optional arguments exist. Oh, and don't forget about type="any" which allows you to pass arguments of any type. And the ToString function :)

Function overloading - is, undoubtedly, great. But I'll trade it IsDefined most of the time (when it's uses are justified, of course :) )

PS. Correct me if I'm wrong, but does Java really not have access to the array of arguments passed to a function? And I'm not talking about Variable arity.

Coldfusion и XML

Thu, 17 Aug 2006 07:46:03 GMT

It is a major pain in the nether regions of the body to work with a language which provides reasonable tools for working with XML and provides no sane tools to modify XML. I'm talking about ColdFusion.

Suppose we have the following piece of XML:

<xml>
    <root>
        <elem>
            <value>Text</value>
        </elem>
    </root>

If we read this piece into a variable called xmlDoc, we have immedate access to the value "Text":

txt = xmlDoc.root.elem.value.xmlText;

Now txt contains, as you may have guessed it, a string, "Text".

Moreover, if we have quite a handful of such elements, we may use an extremely convenient function XMLSearch which accepts (an unspecified subset of) XPath and returns an array of found elements.

A real-world example now. Here's a stripped-down localization file from the project I'm working on::

<?xml version="1.0" encoding="UTF-8"?>
<strings>
	<nspace NAME="global">
		<string CRC="73FB418E8CCD929E219338A555AA7EA4">
			<original>You need to login first</original>
			<localized/>
		</string>
		<string CRC="99DEA78007133396A7B8ED70578AC6AE">
			<original>Login</original>
			<localized/>
		</string>
	</nspace>
	<controller NAME="admin">
		<action NAME="cities">
			<string CRC="FD8459135F9464065B708800B0BDF6D8">
				<original>Add a new city</original>
				<localized/>
			</string>
			<string CRC="F67FDD86A499050E0585BCA9EA023188">
				<original>Add city</original>
				<localized/>
			</string>
			<string CRC="4505DE1F3D02176AA6F1403778C5ADD1">
				<original>Region:</original>
				<localized/>
			</string>
		</action>
	</controller>
</strings>

Searching across this monster of a file is trivial:

searchString = "/strings/controller[@NAME=""admin""]/action[@NAME=""cities""]/string[@CRC=""CRC""]";
elems = XmlSearch(xmlDoc, searchString);

/* If there's only one elems, we return it (simplified example): */

return elems[1].localized.xmlString;

where CRC is calculated using obscure Vodoo rituals :)

Now, the question is: what if we want to change the value oа this element? The procedure required to do that evokes an unbearable desire to nuke Macromedia offices :)

Scanning thorugh the docs reveals that the only standard procedure to change the desired value of a tag involves the use of ArrayAppend function on the array of "value" from the array of "elem" from the array of "root"...

That is... You can find an element, but you cannot manipulate it directly. You have to take the root element. Then you have to take an array of first-level elements anв find the desired one. In that element you once again take an array (of second-level elements) and find the desired one. In that element you take an array of third-level elements and find the desired one. And finally, you take an array of fourth-level elements and find the desired one. If such an element does not exist, you append it to the array of fourth-level elements. Save.

Anyway, here's what it looks like in the end:

// Add a new string
ArrayAppend(xmlDoc.strings.xmlChildren[controllersIndex].action[actionsIndex].XmlChildren, XMLElemNew(xmlDoc, "string"));

// Find its position
stringIndex = ArrayLen(xmlDoc.strings.xmlChildren[controllersIndex].action[actionsIndex].XmlChildren);

// Change its attribute
xmlDoc.strings.xmlChildren[controllersIndex].action[actionsIndex].string[stringIndex].xmlAttributes.crc = Hash(t);

// Add "original" to it
ArrayAppend(xmlDoc.strings.xmlChildren[controllersIndex].action[actionsIndex].string[stringIndex].xmlChildren, XMLElemNew(xmlDoc, "original"));

// Find original's position
orgIndex = ArrayLen(xmlDoc.strings.xmlChildren[controllersIndex].action[actionsIndex].string[stringIndex].xmlChildren);

// Add text
xmlDoc.strings.xmlChildren[controllersIndex].action[actionsIndex].string[stringIndex].original.xmlText = t;

// Add "localized"
ArrayAppend(xmlDoc.strings.xmlChildren[controllersIndex].action[actionsIndex].string[stringIndex].xmlChildren, XMLElemNew(xmlDoc, "localized"));

// Obtain its index
locIndex = ArrayLen(xmlDoc.strings.xmlChildren[controllersIndex].action[actionsIndex].string[stringIndex].xmlChildren);

// Add text
xmlDoc.strings.xmlChildren[controllersIndex].action[actionsIndex].string[stringIndex].localized.xmlText = '';

:)))

I am only grateful that I had to write this function only once. And it took me the better part of the day...

Named Parameters

Fri, 28 Jul 2006 07:53:00 GMT

Ever since I saw Lisp I've sorely missed named function parameters. What the heck is that? Ok, here goes...

Lisp examples are quoted directly from Practical Common Lisp

Let's refresh our knowledge of WinAPI - the dearly beloved API of all Windows developers. For instance, ShellExecute:

HINSTANCE ShellExecute(      
    HWND hwnd,
    LPCTSTR lpOperation,
    LPCTSTR lpFile,
    LPCTSTR lpParameters,
    LPCTSTR lpDirectory,
    INT nShowCmd
);

On the whole I think I remember which parameters this function requires and why. Not too clearly though. I do remember the first three. The fourth might spring to mind while writing the function. The last too however, are quire elusive and I'm never quite sure which goes where and when. The problem gets worse if you remember that som parameters, hwnd, lpParameters, lpDirectory in our case, are optional and can be replaced with NULLs. So, we would quite often invoke this function like this::

 hResult = ShellExecute(NULL, "open", "path/to/file", NULL, NULL, SW_SHOWDEFAULT);

Scary, huh? Now imagine if we could write this function like this:

 hResult = ShellExecute(operation="open", file="path/to/file");

Beautiful. Now, this is what named parameters are all about.

Let's go back to Lisp. Let's consider the following function:

 (defun foo (&key a b c) (list a b c))

This function accepts three parameters and simply listst them in their order. This is how this function behaves when we invoke it in numerous ways:

(foo)                ==> (NIL NIL NIL)
(foo :a 1)           ==> (1 NIL NIL)
(foo :b 1)           ==> (NIL 1 NIL)
(foo :c 1)           ==> (NIL NIL 1)
(foo :a 1 :c 3)      ==> (1 NIL 3)
(foo :a 1 :b 2 :c 3) ==> (1 2 3)
(foo :a 1 :c 3 :b 2) ==> (1 2 3)

Note the last line. Even though the order or parameters is incorrect, the function still behaves as designed. Now, that's the beauty and power of named parameters. A developer now only needs to know which parameters a function requires, not their not their order.

Unfortunately none of the mainstream languages support named parameters, not natively at least. That is why invoking a function (especially a WinApi function) is a long and tedious process whereas we could simply make do with passing a limited number of named arguments (as my example with ShellExecute shows).

I am currently developing in ColdFusion and it supports named parameters natively (though you cannot honestly call it a mainstream language):

<cffunction name="function" access="public">
    <cfargument name="arg1" type="numeric" required="yes">
    <cfargument name="arg2" type="string" required="no">
    <!--- и так далее --->


</cffunction>

This function is invoked as follows:

<cfset function(arg1=1, arg2='string')>

As in Lisp, the order of arguments is not important as long as they are named.

Some languages (scripting languages, mostly) emulate this behaviour through the use of hashes. PHP anyone?

function a_func($params)
{
    if(isset($params['arg1'])) /* do stuff */
    if(isset($params['arg2'])) /* do stuff */
}
 
/* Calling the function */
$result = a_func(array('arg1' => 123, 'arg2' => 'a string'));

That is, we actually pass an array, but it serves our purpose well enough. Ruby employs exactly the same technique:

# find from ActiveRecord
      def find(*args)
        options = extract_options_from_args!(args)
        validate_find_options(options)
        set_readonly_option!(options)

        case args.first
          when :first then find_initial(options)
          when :all   then find_every(options)
          else             find_from_ids(args, options)
        end
      end

 
# Calling the function:
Person.find(:first, :order => "created_on DESC", :offset => 5)
Person.find(:all, :group => "category")
Person.find(:all, :offset => 10, :limit => 10)
# where Person is an object of type Activerecord

However, none of these can even compare to Lisp's implementation. ColdFusion, for instance, doesn't accept an arbitrary number of parameters (see Lisp's &rest). Hashes both in PHP and Ruby are rather cumbersome because implementation of the function may become convoluted (or you have to resort to using less-than-safe extract or extract_options_from_args! functions). I'm not even talking other languages like С/С++, C# and Java :). Some languages though, like Haskell, Erlang and Nemerle, have pattern matching allows the developer to work arounв named parameters in a very elegant manner, but that is a story for another day.

Running along the rails

Wed, 26 Jul 2006 07:29:20 GMT

In July 2006 CNN Money gave a fellow named David Heinemeier Hansson the 50th place in their list of most influential people in contemporary online business (also known as Business 2.0).

What? You don't know who this fellow is? Well, he is the creator of Ruby on Rails.

Ruby on Rails is an iPod of online technologies:

Like iPod, Rails doesn't use any new revolutionary technologies:
- - Ruby itself appeared in 1995
- - The MVC (Model-View-Controller) pattern was first described back in 1979
and so on

Both iPod and Rails, when they appeared, were the cause an unprecedented buzz and discussion.

Rails's fans, much like iPod's fans, are blind to the shortcomings of their favorite toy and are ready to rip any critic to pieces (or worse).

Rails, like iPod, has created numerous clones which are striving to duplicate the functionality without actually introducing any new ideas:
- - PHP: CakePHP and Zend Framework
- - ColdFusion: ColdFusion on Wheels
- - .NET: MonoRail
- - Perl has Catalyst, though this framewrok isn't originally based on Rails
- - Python has TurboGears, though its creator stated that this is not a Rails clone

So, why Rails has not only appeared, but also become very popular and eagerly cloned?

I suppose this happened because Rails (and iPod, too) showed us that technology can be humane.

In the case of Rails you can obtain interesting result fast - lightning fast (nonbelievers can check out Rails screencasts). Most of the time you spend in Rails is not fighting databases, or sessions, or files, or anything else, but actually writing the logic of your application.

Additionally, Rails lowers the entry level for the MVC pattern (quite a complex pattern, actually). You simply forget about it, it comes to you naturally.

Rails is, in essence, a ready and sound foundation that let's you build your own house. You may choose either a number of available blocks or just build everything brick by brick. Most other framewroks, on the contrary, have been providing you with either a finished house or with a number of raw materials and a dozer.

So, this David Heinemeier Hansson fella has again proved that no "designed by committee" system/framework (even if this committee is a multi-billion dollar corporation) can ever compete with frameworks that people build for themselves. After all, people know better about their own needs than a committe ever will.

Anyway, I can't wait to see what happens to web technologies 5 years from now. After all, too many developers have accustomed themselves to thу idea of humane technology to ever turn back. I hope.

Winter has come...

Thu, 20 Jul 2006 09:44:14 GMT

After some consideration we have decided to develop our project in ColdFusion. The reason is simple - there are some developers who can help us should the need arise. There are no Rubyists in sight, however...

I have mixed feelings about ColdFusion so far. I'm quite unnerved by CFML (sort of SGML-based, but not quite) and DreamWeaver's stupid-stupid-stupid wizards which break all code formatting once applied.

Stay tuned for more rants...

Web programming. Revisited

Tue, 04 Jul 2006 16:13:13 GMT

On the heels of the last post:

Found here

Where else, other than in Web programming that is, can you find such a mash-up? :)