lectures in computer science

Lazy Evaluation: an optimization technique

2006-04-19T22:48:49Z

Imagine for a moment that you have an object that you can ask a particular question about, but the question is expensive to determine the answer to. Sometimes you need the answer a lot, sometimes you never need the answer. There are several ways to go about this; the simplest can be called "reactive" and "proactive".

To show this, I'm going to need an object with an expensive question, so here's a heavily stripped down 3D vector in C++, with a method that returns the length of the vector:

class Vector3D {
private:
  double x, y, z;

public:
  Vector3D(double new_x, double new_y, double new_z) {
    x = new_x;
    y = new_y;
    z = new_z;
  }

  void setX(double new_x) {x = new_x;}
  void setY(double new_y) {y = new_y;}
  void setZ(double new_z) {z = new_z;}
  double getX() {return x;}
  double getY() {return y;}
  double getZ() {return z;}

  double getLength() const {
    return sqrt(x*x + y*y + z*z);
  }
};

This is a reactive design: the length is recalculated from scratch each time it gets asked for. This is just fine most of the time, but sometimes you find that the length is asked for repeatedly between changes. Since the method is expensive (well, okay, this one really isn't, but for our purposes we will call it so), we don't want to have to recalculate it each time it gets asked for. So instead, let's try getting it to change every time something is changed. This is called proactive design: the question is answered before it is asked. Here's how that would work (added changed sections are in bold):

class Vector3D {
private:
  double x, y, z, length;
  
  void recalcLength() {
    length = return sqrt(x*x + y*y + z*z);
  }

public:
  Vector3D(double new_x, double new_y, double new_z) {
    x = new_x;
    y = new_y;
    z = new_z;
    recalcLength();
  }

  void setX(double new_x) {
    x = new_x;
    recalcLength();
  }
  
  void setY(double new_y) {
    y = new_y;
    recalcLength();
  }
  
  void setZ(double new_z) {
    z = new_z;
    recalcLength();
  }
  
  double getX() {return x;}
  double getY() {return y;}
  double getZ() {return z;}

  double getLength() const {
    return length;  
  }
};

This isn't too bad either for most situations, you find; however, you now have a different problem: sometimes you call setX etc a lot of times on an object, but never ask for its length! This is a lot of wasted effort to update something that never gets looked at.

So what do you do? Neither calculating from scratch every time it gets asked for nor recalculating whenever it changes gives acceptable performance. You can't win.

Or can you?

What if you instead calculate the answer only when it needs to be calculated, and hold on to it for when you need it again? You'd need to be able to tell when the answer is valid, and when you need to recalculate it...

class Vector3D {
private:
  double x, y, z, length;
  bool length_is_valid;
  
  void recalcLength() {
    length = return sqrt(x*x + y*y + z*z);
    length_is_valid = true;
  }
public:
  Vector3D(double new_x, double new_y, double new_z) {
    x = new_x;
    y = new_y;
    z = new_z;
    length_is_valid = false;
  }

  void setX(double new_x) {
    x = new_x;
    length_is_valid = false;
  }
  
  void setY(double new_y) {
    y = new_y;
    length_is_valid = false;
  }
  
  void setZ(double new_z) {
    z = new_z;
    length_is_valid = false;
  }
  
  double getX() {return x;}
  double getY() {return y;}
  double getZ() {return z;}

  double getLength() {
    if (!length_is_valid)
      recalcLength();
    return length;
  }
};

Now, we have three ways of designing our object to get the answer to the question. Let's try using them.

This table compares Reactive, Proactive, and Lazy evaluation methods by the number of times the evaluation happens, over the same sequence of test calls. The test calls are Initialization, Querying, and Changing, which correspond to the methods Vector3D, getLength, and setX (or setY, or setZ), respectively. There is an X in a cell if the answer is calculated during that row's call under that column's design. At the bottom is the number of times the answer is calculated for that design.

	R	P	L
I		X
Q	X		X
Q	X
Q	X
C		X
C		X
C		X
Q	X		X
C		X
Q	X		X
C		X
C		X
Q	X		X
C		X
Q	X		X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
C		X
C		X
C		X
C		X
C		X
C		X
C		X
C		X
Q	X		X
C		X
Q	X		X
C		X
	19	18	7

As you can see, the lazy evaluation beats them both.

Now, you can't always use lazy evaluation. It requires that you have absolute control over the data the question is based on - if something that your question depends on is directly accessible from something other than your object's set methods, you can't use it. It also requires that the method that embodies the question is allowed to write to the object as opposed to simply read from it - unlike reactive and proactive evaluation, the question method cannot be const. Then there are places where it's not appropriate - an inexpensive calculation doesn't gain much from lazy evaluation; sometimes it's better to have consistently timed responses than high speed most of the time and a relatively epic slog when it needs recalculating; it adds data to the object as a whole, which makes them cost more memory and which can also prevent libraries that copy large chunks of data to somewhere else (OpenGL, for instance) from operating efficiently.

Vorn

3D Graphics Primer, Part III: Projective Space, and Canonical Coordinates

2006-02-26T15:38:34Z

So what's this fourth coordinate do for us? We're still in a 3D world, aren't we? So if we added a fourth coordinate, we'd start having things overlap. This leads to the question: is there a way to get a canonical version of a 4-vector?

Before we can really talk about this, I need to prove something to you.

Consider the situation where we multiply a vector v by a scalar n and then transform this new vector by a matrix A. Can we express the transformed new vector in terms of the transformed old vector?

Let\'s apply the matrix × vector operation (Av)_j = &Sigma_kv_k×A_j_k to a scaled vector (v × n)_k = v_k × n:

(A × (v × n))_j = &Sigma_kv_k × n × A_j_k
= n × &Sigma_kv_k × A_j_k
= n × (A × v)

So, in short: the size of a vector doesn't change the way a matrix affects the vector's direction.

It's pretty obvious that w only affects translation - the only part of the 4×4 matrix that allows w to affect the outcome is the last column, which is where translation comes into play. So in order to find out what 4-vectors correspond to each other, we will try to figure out how to convince a translation matrix to turn a particular vector v to <0, 0, 0, w>.

This isn't even vaguely hard:

1	0	0	-v_x/v_w
0	1	0	-v_y/v_w
0	0	1	-v_z/v_w
0	0	0	1

...which also happens to affect the vector <x/w, y/w, z/w, 1> in exactly the same way as it does <x, y, z, w>.

So, in short: In all cases where w ≠ 0, the canonical version of a 4-vector v is v/w.

But what happens when w = 0? Translation doesn't affect these at all, and we can't get a canonical version of it using the normal formula.

Well, where can I put a point so that translation doesn't affect it at all?

How about infinity? After all, infinity plus any number is still infinity. Space with an additional area for things at infinity is known as projective space, and it has some strange properties. For instance, parallel lines meet twice at infinity, once in one direction and once in the opposite direction. The only property we're really concerned with is the fact that the area at infinity exists, and that it has one fewer dimension than the normal sections of space - so a 3-space turned projective has a plane at infinity.

But that still leaves us with a problem: the chunk of space that can be called "infinity" has only two dimensions (you can think of them as, for instance, latitude and longitude), but we've got three dimensions to our vectors that talk about this chunk. Fortunately we can use our multiplication result from before to come up with a canonical version of vectors with w = 0, choosing our canonical version to have a length of 1:

v' = v / ||v||

Next time: The Orthogonal Camera, Points, Lines.

Vorn

A 3D Graphics Primer, Part II: Common Transformations and The Fourth Coordinate

2006-02-12T07:14:28Z

Now that we have our tools, let's see how to apply them.

This is a scaling matrix:

2	0	0
0	3	0
0	0	4

When applied to a given vector, the vector will get bigger: twice as big in the X direction, three times as big in the Y direction, and four times as big in the Z direction. I hope I don't have to tell you how it does this. You can use any numbers in a scaling transformation.

Rotation is a bit more tricky. I'll start with the two-dimensional case, where things are rotated around the origin.

Here's two vectors being rotated counterclockwise by 53.13 degrees. The starting positions are thin, and the ending positions are thick. From this diagram we will figure out how to rotate things counterclockwise.

To do this, we must figure out the various trigonometric values for the angle we're rotating: sin(53.13deg) = 0.8; cos(53.13deg) = 0.6.

Now we take these and figure out where each of them come in:
for x, 0.6 appears as the new x component, and 0.8 appears as the new y component.
for y, 0.6 appears as the new y component, and -0.8 appears as the new x component.

So here's a matrix that rotates things around the origin by θ:

cos(θ)	-sin(θ)
sin(θ)	cos(θ)

We can then take this matrix, and extend it to three dimensions, making a matrix for rotating around each of the three primary axes x, y, and z respectively:

1	0	0
0	cos(θ)	-sin(θ)
0	sin(θ)	cos(θ)

cos(θ)	0	sin(θ)
0	1	0
-sin(θ)	0	cos(θ)

cos(θ)	-sin(θ)	0
sin(θ)	cos(θ)	0
0	0	1

(for the record, you might see these with the signs on the sin terms the other way around; those are clockwise rotations. Both are valid, but make sure you know which ones you're using.)

From here, it is now possible to create a matrix that will rotate around an arbitrary axis... but I don't really feel like stepping through four matrix multiplications and then a heinous simplification step to get there, so I'll just give you the result.

cos(θ) + (1 - cos(θ))x²	(1 - cos(θ))xy - (sin(θ))z	(1 - cos(θ))xz + (sin(θ))y
(1 - cos(θ))xy + (sin(θ))z	cos(θ) + (1 - cos(θ))y²	(1 - cos(θ))yz - (sin(θ))x
(1 - cos(θ))xz - (sin(θ))y	(1 - cos(θ))yz + (sin(θ))x	cos(θ) + (1 - cos(θ))z²

And that's rotation.

You may be staring at this at this point and wonder why the hell I haven't done translation yet, seeing as it looks like the easiest of the transformations to do. And if you're really good you'll start to realize why: matrices don't provide for the possibility of adding a constant to the vector it's being applied to. So we're going to have to come up with a way to translate things.

Enter the fourth coordinate.

Earlier, when I said that things in 3-space were represented by 3-vectors, I lied. usually they are represented by 4-vectors <x,y,z,w>, where w is called a "scaling factor". For most purposes, w is set to 1; other values are possible, but are not used as often. To go with our 4-vectors, we must have 4×4 matrices. This, finally, gives us the ability to represent translations. Here's a translation matrix for 3D, to translate things by a 3-vector v:

1	0	0	v_x
0	1	0	v_y
0	0	1	v_z
0	0	0	1

So the fourth column is how much w contributes to each new coordinate, and the fourth row describes how much each old coordinate contributes to the new w. The fourth row never changes from what it is in the above matrix for most purposes. To extend a 3-matrix to a 4-matrix, just put 0s in all the new cells except for A₄₄, which gets a 1.

Next time: the consequences of the fourth coordinate.

Vorn

3D Graphics Primer, Part I: Vectors and Matrices

2006-02-11T07:38:50Z

All points in 3D space can be represented by three numbers, the distance from three planes that are perpendicular to each other. These three numbers are usually called x, y, and z, and they're usually written something like this: <x, y, z>. This is called a vector. A vector, when named, is displayed in bold, i.e. v. Vectors can have any number of components - the most common in graphics work are 2, 3, and 4 component vectors. The fourth component, when it exists, is called w. You can talk about an individual component of a vector like this: v_x (the first component of vector v.

Here's some things you can do with vectors:

You can add two vectors: (u + v)_k = u_k + v_k. This of course means you can also subtract them.
You can multiply a vector by a scalar:(v × n)_k = v_k × n. You can also divide a vector by a scalar, and multiply a scalar by a vector.
You can multiply two vectors together to get a scalar, using an operation called a dot product: u·v = Σ_ku_kv_k.
You can multiply two 3-vectors together to get a vector, using an operation called a cross product: u×v = <u_yv_z - v_yu_z, u_xv_z - v_xu_z, u_xv_y - v_xu_y>. This can be generalized to other numbers of dimensions, with number_of_dimensions - 1 vectors being needed, but it is not common.

In addition to being able to represent points, 3d graphics requires that we be able to move points around according to some rule The mechanism for doing this is known as a matrix. Matrices are square arrays of numbers which describe certain kinds of transformations easily. They are generally denoted by a sans-serif capital letter: A, and are shown like this:

1	0	0
0	1	0
0	0	1

This particular matrix is called the identity matrix, for reasons I'll get to in a minute. You can address an individual element in the matrix as A_j_k, where j is the row the element is in, and k is the column, starting with the topmost or leftmost line as 1. here are the most important things (for graphics) that you can do with a matrix:

You can multiply a matrix by a vector, to get another vector: (Av)_j = &Sigma_kv_k×A_j_k. This is called applying a matrix to a vector, or transforming a vector by a matrix.
You can multiply a matrix by another matrix, to get a third matrix: AB_j_k = &Sigma_iA_j_iB_i_k. applying a product of two matrices to a vector is equivalent to applying the two original matrices to the vector in turn.

Applying the identity matrix I to a vector v results in v, and multiplying a matrix A by the identity matrix results in A.

Next: common transformation matrices, and why something I said is a convenient lie.

EDIT: fixed the subscript to iterate over for applying matrix to vector.

Integer division and processor-specific operations

2005-07-12T00:22:10Z

Integer division a/b, defined:

For any two integers a and b > 0, there exist unique integers q and 0 ≤ r < b such that a = q × b + r. For instance:

a	b	q	r
10	3	3	1
-10	3	-4	2

So far, decently simple. r is positive in all of these.

So let's extend division, so that we can use b < 0 as well:

For any two integers a and b ≠ 0, there exist unique integers q and 0 ≤ r < |b| such that a = q × b + r.

a	b	q	r
10	3	3	1
-10	3	-4	2
10	-3	-3	1
-10	-3	4	2

Woohoo. Definition of integer division we can use on computers, where nearly every number can be positive or negative.

Except that many computers don't USE this definition. Here's the one that my two machines (Mac G4, Pentium III) use:

For any two integers a and b ≠ 0, there exist unique integers q and r &isin [0,b) such that a = q × b + r.

a	b	q	r
10	3	3	1
-10	3	-4	2
10	-3	-4	-2
-10	-3	3	-1

In the former, and r is always positive. in the latter, r gets its sign from b. Some computers still use the former, though, so not all computers divide the same way.

Moral of the story: when writing a program for multiple platforms, make sure you ALWAYS get the answer you intend on getting from division. If this means hacking apart your numbers before or after you get them, so be it.

EDIT: fixed the third table to correct signs for q in the bottom two rows.
EDIT AGAIN: fixed the third table again because I'm an idiot.

Vorn

Calling Functions, the Program Stack, and the Void data type

2005-06-10T03:18:24Z

Computer programs aren't monolithic; they are composed of littler pieces of code known as functions. Each of these functions acts on its own, using its own data, and acting essentially like its own program. And each function can call other functions, and each of those functions can act on its own. But all this makes one wonder how the heck it does that.

Enter the program stack.

The program stack is basically a sequential area of memory - an array, almost, except that you can put anything in it (I'll explain how in a few moments) - used to store all the data that the program uses. When you call a function, you increase the size of the stack, and when you return from a function, you decrease the size of the stack.

This is the program stack before we make the function call.

This is after we finish making the function call. We have pushed a placeholder for the return value, and all the arguments, onto the stack. We're now at the start of the function itself.

Now, the function adds its own variables to the stack. These are known as local variables.

The function runs until it arrives at a return statement, at which point it pops all the stuff it has been using other than the return value off the stack.

And that's how it works.

"BUT WAIT!" you cry, confused. "isn't an array supposed to hold the same thing, all the time? You described it as an array earlier, and here we are stuffing everything from booleans to doubles in it!"

And you would be right. The program stack actually uses a special type, known to C programmers as "void*". the void*, or void pointer, data type is special because any pointer can be turned into a void pointer, and any void pointer can be turned into a pointer of any other type. A void pointer points at an object a single byte wide: when we put things onto the program stack, we remember where they are as distance in bytes from the bottom of our portion of stack, which for each function is addressed as index 0 in the array.

But how do we change the indexing of an array that already exists? Remember that arrays are passed around as pointers. In fact, we can use any pointer as if it were an array, including pointers we create by finding the location of a specific index of an array. So all the function sees is a pointer to the bottom of its section of stack, and it knows where in the program stack to find the other things it needs, using an index into the 'array' created by the pointer to the function's section of stack.

To summarize:

Functions use a program stack to store their return values, arguments, and local variables. When called, their return value placeholder and arguments are already there;when they return, they leave only the return value. They see only their section of the program stack, through a void pointer acting as an array. They know where to find their stuff on the program stack by using constant indices into that array.

Have I confused you yet?

Vorn

cs_lectures @ 2005-05-07T16:04:00

2005-05-07T20:04:26Z

Going off of derakon's entry about Gray Codes, here's how to generate the gray codes mentioned there (there are lots and lots and lots of gray codes, something like 120 million for 5-bit gray codes).

The 1-bit sequence of gray codes is obviously (0,1). To get the list of n-bit gray codes, you take the n-1-bit list, append its own reflection to itself, with the most significant bit turned on:

This kind of gray code is the most commonly used, and it's known as the Reflected Binary Gray Code.

It's possible to go from a binary number to reflected binary gray code, and vice versa, using operations that work on one bit at a time.

From Binary to Gray:

start at the second least significant bit (the one that's worth 2).

If the bit we're looking at is on, change the state of the next less significant bit (so if we're on the bit that's worth 4, change the state of the bit that's worth 2).

move to the next more significant bit (if we're on 4, go to the one worth 8), and go back to step 2.

So with 13 (0b1101) we get 1101 -> 1101 -> 1111 -> 1011.

From Gray to Binary:

start at the most significant bit

If the bit we're looking at is on, change the state of the next less significant bit.

move to the next less significant bit, and if we're not on the least significant bit, go back to step 2.

So with gray 13 (0b1011) we get 1011 -> 1111 -> 1101 -> 1101.

Vorn

Stacks, Queues, Doubly Linked Lists (old)

2005-05-04T05:04:08Z

Before I actually get into stacks and queues, one of the comments on the last entry asked about doubly linked lists - where each list item knows not only where the next is, but where the previous one is:

This has the advantage that we can 1. iterate backwards through the list (something that sometimes you need), 2. get to a specific item that's late in the list a lot faster , and 3. delete from the end of the list in constant time, so in effect we get a constant-time unrestricted deque (more on what that is later).

A very common use for data structures is for storage of things that we need a lot of together to work properly. A good example of this is HTML; it takes a lot of tags to get an HTML page working properly, and a browser needs to consider each one in its context to deal with the whole. The tool it uses to work its way through the file is known as a Stack, where you can hold data like you would a pack of cards: you can see the one on top, and you ccan put things on the top. Often this is all you need.

A stack is a data structure with four basic operations:

push object will put the object onto the top of the stack,
pop will remove the topmost object from the stack,
peek will look at the topmost object on the stack (but will not change the stack),
length will tell you how many objects there are on a stack (but will not change the stack).

This table demonstrates the operation of a stack. Peek and length are the values that those two operations would provide if run after the command.

Command	Stack State	peek	length
(start)		*	0
push a	a	a	1
push b	a b	b	2
pop	a	a	1
push c	a c	c	2
pop	a	a	1
pop		*	0

* peek can't return anything in these states because there's nothing to peek at.

There are two basic ways to represent a stack using the base data structures we have: one is an array, and the other is a linked list. Here's how we work in both of these, using st as the underlying data structure, and len as the length.

Command	array implementation	list implementation
push object	st[len] = object; len = len + 1;	st.insertAtStart(object); len = len + 1;
pop	len = len - 1;	st.removeFromStart(); len = len - 1;
peek	return st[len - 1];	return st.firstObject;
length	return len;	return len;

Arrays have the advantages that you never have to allocate any more memory to add something to the stack and that you never have to clean up after removing something from the stack (you just move the length down and leave the object there, where it will get overwritten when you push enough stuff in), and the disadvantage that you have to limit the size of the stack from the beginning. Lists have the advantage that you can't run out of space on the stack (until you run out of memory on the computer entirely), but that it's slower since you have to allocate and deallocate memory as you add and remove things from the stack.

A queue is similar to a stack, but it does things in a different order: peek and pop look at what (from the stack point of view) would be the bottom of the queue, not the top. It's like the line at the store: the cashier only works with the first person in line, and when you get into the line you are the last person in line. A queue is often used when you are recieving messages or something that have to be processed in the order they arrive.

This table demonstrates the operation of a queue. Peek and length are the values that those two operations would provide if run after the command.

Command	Queue State	peek	length
(start)		*	0
push a	a	a	1
push b	a b	a	2
pop	b	b	1
push c	b c	b	2
pop	c	c	1
pop		*	0

* peek can't return anything in these states because there's nothing to peek at.

A queue cannot easily be implemented in a single array; it is possible to implement it using two stacks, but it is complicated and not often done, in particular because it has a tendency to do a lot of the work in spurts.

Changing a linked stack into a queue involves replacing just the push() function with one that inserts at the end instead of the start.

Vorn

Linked Lists (old)

2005-05-04T05:03:23Z

So our basic data structure, the one that is used a whole lot, is the array. It has one primary advantage: you can access any element in the array in constant time, and you don't have to think too much about pointers when you do. Oh, and it is the most efficient storage technique in terms of memory usage. But it also has disadvantages: How do you insert an object into an array at a given point and have it shove all the other ones farther in? How do you delete an object from an array and have all the other objects move up to fill the space left?

All of these can be answered, but they take a considerable amount of time when you have a big array: you have to actually use the move/copy commands that the memory may or may not provide, and in many cases the insertion needs to move the whole array because you've got stuff around the array that you don't want to step on - and since pointers to that array can be stored in many places, you're liable to break your program if you try that.

So, instead, we invent a new data structure: a Linked List.

A Linked List, instead of storing a lot of objects right next to each other, stores each individual object along with a pointer to the next object in the list, like this:

That symbol over on the right that the last pointer's arrow ends in signifies that it is a null pointer - basically the address in the pointer is 0, and the computer never lets you touch the stuff there, so that you can use the null pointer to say "there's nothing here". The symbol itself comes from electronics, where it means that the wire goes to ground. In our case, the null pointer means that we're at the end of the list.

So let's see how well this linked list works. First, the random access test: can we get the nth object in the list, and how long does it take?

Well, to get the nth object, we need to get the pointer to it... which is in the n-1th object. But we need to get the pointer to that, so we go to the n-2th object... etc. So random access takes linear time. Suckage, right? Wellllllll.... sorta. The thing is that often when you deal with lots of data at once, you iterate through the array - get the 0th item, then the 1st, then the 2nd, then the 3rd, etc. And with a linked list, this takes pretty much exactly as long as it would with an array: to get the next object, just follow the pointer from the current object. So with a linked list, as with an array, iterating through the whole list takes linear time.

Now, the insertion test: can we insert an object into the list at a specific point, and how long does it take?

First we have to get there. We know that this takes an amount of time equal to the number of nodes we have to pass through. However, once we ARE there, it's very simple: given the node we want to insert N, and the node that we want to have before it P, we do this:

put N somewhere and assign the pointer FROM P to the pointer FROM N.
assign the pointer TO N to the pointer FROM P.

Like this:

So, yes, it still takes linear time, but there's still several nice features: you never have to move the parts of the list you already have, and you can do it multiple times in linear time too (technically O(M+N) where M is the number of nodes you have to go through to get to the last node that you want to add one after and N is the number of nodes you want to insert).

Thirdly, the deletion test: can we delete a specific object from the list and have the list still work, and how long does it take?

And the answer is still yes, and like insertion it takes linear time for one or many.

To delete a node D from a linked list, go again to the node before it (P), and change P's node from pointing at D to pointing at the thing D is pointing at. Then we can safely free up the memory previously used by D. Thus:

Now that we've done the basics of linked lists, let's consider a common construction, where we store pointers to the first and last nodes on the list somewhere that we can access quickly. How much work does it take to insert before the first node? after the last one? How long does it take to delete the first or last node? In all cases we must maintain the pointers to first and last.

Inserting before the first object is done by setting the new object's pointer to that of the old first node and changing the "first" pointer to the location of the new first object.

Deleting the first object is done by setting the "first" pointer to that of the pointer from the first object, and freeing the memory used by the old first object.

Inserting after the last object is done by setting the old last object's pointer and the "last" pointer to the address of the new object. The new object's pointer should be set to null.

Deleting the last object is a pain: you go through the entire list waiting for a pointer to the last object to show up (which it will in the second-to-last object), then changing "last" to the second-to-last object, changing the second-to-last object's pointer to null, and freeing the memory that the old last object took up.

So inserting on either side is fine, and deleting the first object is fine, but deleting the last object takes linear time always.

Next time: Stacks and Queues.

Vorn

Let's talk data structures (old)

2005-05-04T05:02:22Z

I want to discuss data structures - the things that computers use to store, collate, and find information in as efficient a way as possible - but to do so I have to talk about a few simple things first.

Memory

Imagine, for a moment, that you have a giant-huge wall of dials, and a clerk that can read and operate these dials. Each dial is numbered, and can store a number between 0 and 255, inclusive. So you can ask the clerk to go to one of these dials (by telling him "tell me the value of the dial numbered 49,152"), and you can also ask the clerk to change the value of one of these dials (by telling him "change the value of the dial numbered 49,152 to 5"), and both of these things take the clerk one second.

This approximately describes something known as Random Access Memory. It is called "random access" because you can get information to or from any piece of memory at any time, and it always takes the same amount of time.

The only real difference between what I described and actual RAM you have in your computer is that the dials are replaced with transistors, the clerk is replaced with a multiplexer, and it takes ten or less nanoseconds to do either of the above mentioned operations. Actually there are few Really Special operations that can be pulled off with most memory, like moving or copying huge chunks of memory to other places, that go a lot faster than would usually be possible if you had to tell the memory each little tiny step.

Pointers

So imagine that you do this: you take the values of a consecutive pair of those dials - say, dials 1022 and 1023 - and put them together to get a single unique number. Then, with that single number, you ask the clerk to go get the value of the dial with that number. This is what's known as a pointer: a piece of memory that is used to refer to another piece of memory. So say that dial 1022 has the value 192, and that dial 1023 has the value 0. To get our number we'll multiply the first number by 256 (because that's one bigger than the largest number we can store on a dial) and then add the second number: 192*256 + 0 = 49152. Then we ask the clerk for the value of dial 49152, and he says "5" back to us.

Pointers are represented in a lot of ways. The most common is a box with an arrow leading out from the center of it pointing at another box:

So that the left box is the pointer, and the right box is what the pointer is pointing at.

Multi-byte data types

Most of the time, one dial on the memory isn't enough to store any useful values; 0..255 can't store, for instance, any decent amount of money. We saw this earlier with our pointer, which we defined as two bytes put next to each other. Most modern computers actually store pointers as 4-byte (or 32-bit, which you've probably seen it described as) unsigned numbers. Similarly, we can define other common data types like this, which is a list ganked from Java:

bool: 1/8 of a byte (a bit, which can hold a single value of either 0 (false) or 1 (true)), will group together with other bools in the same record to fill out bytes efficiently. no matter how many bools there are in a record, they will ALWAYS take up a whole number of bytes, to make life easier.
byte: 1 byte, 2's complement if signed (defaults to signed)
short: 2 bytes, 2's complement if signed (defaults to signed)
int: 4 bytes, 2's complement if signed (defaults to signed)
long: 8 bytes, 2's complement if signed (defaults to signed)
char: 2 bytes, interpreted as UTF-16
float: 4 bytes, interpreted as IEEE 754 "float" type
double: 8 bytes, interpreted as IEEE 754 "double" type

Each of these will be represented in graphics as a rectangle of appropriate size, in this and following posts. Note that I won't necessarily use many of these types. In those rare cases that I need a char, float, or double and need to distinguish them from the integral types of the same size, I will use a letter inside the rectangle to signify which I mean.

When I say something is "a pointer to an int", for instance, what I mean is that when you follow the pointer, get not only the byte that it points at directly, but get enough bytes after it so that you have 4 bytes.

Structured Data

Most applications need to keep data together - names with addresses, account numbers with dollar figures, etc. Structured data does this. By putting together a lot of littler types of data, you can create a larger piece of data that can be talked about via a pointer all at once, by using a lookup table.

So we could have a structure that we define like this (in C++ish):

struct a_struct {
  byte a,
  byte b,
  short c,
  int d,
  long e
};

So it looks like this:

and we can point at an entire structure by pointing at the first byte in the structure. Then we can just add a number to that pointer to get a pointer to any attribute of the struct: 0 for a, 1 for b, 2 for c, 4 for d, 8 for e.

Arrays

An array is a special kind of structured data, in that all the objects in it are the same type. This also means that they are all the same size, which makes it a lot easier to look up an object in an array - we don't have to store a lookup table, we can just multiply. An array is thrown around in programs as a pointer to the start of the array. Encoded into the program is the kind (and thus the size) of data. Often included along with the array is the total size of the array (but not in C/C++!)

An object in an array is referred to by its index, or location in the array. Most languages (notable exceptions are SQL and VB) start indexing at 0, so that the first object in the array has an index of 0. To get a pointer to an object in an array by its index:

p = pointer to the array as a whole
s = size of the objects stored in the array
i = index that you want to point at

objectpointer = p + s * i

It's really that simple.

Vorn

Heap Sort and Priority Queues (old)

2005-05-04T04:58:15Z

in my last post I mentioned a data structure called a Priority Queue. It turns out that one of my friends wants to implement Dijkstra's Algorithm, and was totally lost on this structure. So, here's how it's done. Or, rather, one way.

The way we're going to do this is through a common data structure called a heap, which is basically an array that sorts itself out partially when you add or remove a piece of data from it. Note that I said "partially" - the order that a heap imposes is not total, it is just enough to get by for what it's doing. How much sorting does it do, exactly? Well, it's like this.

A heap is an array arrenged in such a way such that for every index k, the numbers at indices 2k + 1 and 2k + 2, if they exist, are both greater than (or less than, for a maxheap) the number at index k. This may not sound like much ordering, at first, but this is actually the simplest of several equivalent metrics for ordering things. The most significant and telling of these metrics is that every possible path through the packed binary tree that the heap represents, from the root of the tree to any leaf, is always in the correct order.

"But wait!" you say. "What are all these paths?" Well, here's the 8 paths through a 15-node packed binary tree, as indices:

0 1 3 7
0 1 3 8
0 1 4 9
0 1 4 10
0 2 5 11
0 2 5 12
0 2 6 13
0 2 6 14

You can say "oh, shiny" now.

So all of these paths, once all the heap's sorting is done, are in order. So how does it know when to sort these things so that things stay that way even through a flurry of insertions and deletions? Well, let's watch one in action. Let's take the data [50, 65, 2, 80, 57, 1, 7, 95, 53, 56] and turn it into a heap.

This is representable as five paths of varying length:

50 65 80 95
50 65 80 53
50 65 57 56
50 2 1
50 2 7

These are obviously not in order, so let's go through them to make them that way. We'll start at the 57, because it's the last actual node with leaves attached, and we will swap the root of each trio of numbers with the smallest of that trio:

57: 56

Since 56 is the smallest in this group of numbers, we swap it with the parent:

56: 57.

And then we look at where the 57 is now, and see that it has no leaves anymore, so obviously it's the smallest of those three. We know that the parent of the 56 will be checked later, so we'll leave it there for a while. Next:

80: 95, 53 -> 53: 95, 80

the 95 hasn't moved, and we've already technically checked it (we know it's sorted because it has no leaves) we're not going to look at it. the 80 now also has no leaves, so it is sorted. Finally, the area above will be checked later. Next:

2: 1 7 -> 1: 2 7

Both 2 and 7 are leaves, so we'll go on.

65: 53, 56 -> 53: 65, 56

The 56 hasn't moved, but the 53 and 65 have! We have to look at the 65 again and see whether we need to swap...

65: 95, 80

Nope. Next:

50: 53, 1 -> 1: 53, 50

In a few steps we've managed to move the 1 to the root of the tree. However, there's a bit more to do: the 50 there has moved, so let's check its children:

50: 2, 7 -> 2: 50, 7

The 50 is a leaf, so we don't check further along there.

Now we have completed the heap sort, and these are the results:

1 53 2 65 56 50 7 95 80 57

Which leaves the five paths as:

1 53 65 95
1 53 65 80
1 53 56 57
1 2 50
1 2 7

As you can see, all 5 paths are sorted, which was our original goal. So now that we have that, let's add a number to it: 44. 44 will appear just after the 57 at the start, adding one more path:

1 53 65 95
1 53 65 80
1 53 56 57
1 53 56 44
1 2 50
1 2 7

So now we check 44's parent (because we know that that section has changed):

56: 57 44 -> 44: 57 56

And then we check the 56's position now and see it's a leaf, and then we go up to 44's parent:

53: 65 44 -> 44: 65 53

And we check 53 because it moved:

53: 56 57

and we check 44's parent now:

1: 44 2

and that's fine too, so we're all done shoveling stuff to add the 44 in there.

Our new array:

1 44 2 65 53 50 7 95 80 56 57

And the paths:

1 44 65 95
1 44 65 80
1 44 53 56
1 44 53 57
1 2 50
1 2 7

Which are all sorted. So now let's try removing that first number there, because it's the only one we can absolutely guarantee the relative value of (since it's the lowest number on all the paths, it's the lowest number on the whole array)

__ 44 2 65 53 50 7 95 80 56 57

We need a new number to put there! Shoving the whole array over won't work because then none of the paths will be valid at all anymore. So let's instead take the last item and make it first:

57 44 2 65 53 50 7 95 80 56

57 44 65 95
57 44 65 80
57 44 53 56
57 2 50
57 2 7

Then let's start at the top of the tree and see what happens:

57: 44 2 -> 2: 44 57

57: 50 7 -> 7: 50 57

2: 44 7

and that's all we have to do because everything else is a leaf, or nothing has changed. So our new heap:

2 44 7 65 53 50 57 95 80 56

2 44 65 95
2 44 65 80
2 44 53 56
2 7 50
2 7 57

And lo, once again, all the paths are sorted and the lowest number in the heap is at the front.

Fun.

so, now, how does this apply to priority queues? Well, rather simply: by re-sorting the heap before and after each removal, we always guarantee that we get the lowest thing out of the heap. And since we can put in any number and ensure that it fits in the heap, we can do that too.

Here's the entire thing about heaps:

1. the heap operation is "swap the node index k with the smallest number in indices k, 2k+1, 2k+2.
2. whenever you change any part of the heap, either through addition or deletion of nodes, or by swapping of nodes, run the heap operation on all trios of numbers where at least one number has changed.
3. when adding a node, add it at the end, and do rule 2.
4. when deleting a node, replace it with the one from the end, and do rule 2.

That's it. There's nothing more to it.

There's another kind of heap, which is used for memory management, but this ain't it. The ancient Mac error "stack collision with a heap" is an emergency with just such a structure.

Vorn

Dijkstra's Algorithm (old)

2005-05-04T04:56:05Z

Imagine for a moment that you have highway map of the United States, and you want to find the shortest highway route from New York to Los Angeles. How would you do it? What's the fastest way to find this?

This looks like a job for Dijkstra's Algorithm.

What you actually have in that map is known to mathematicians as an edge-weighted graph: you have a set of points (cities, junctions), and the distances between any directly connected pair of points. What you want to do is take this graph and repeatedly find the shortest path from your start point (New York) to any point you haven't yet visited, and do this until you have visited Los Angeles.

You're going to need a couple of toys for this one, so I'll describe them.

1. a dictionary (aka a mapping or an associative array) - something you can look places up in quickly. As we visit each point, we will put it in the dictionary, with some other information.
2. a priority queue - a container that sorts things as you put them in, and when you take something out, you always get the smallest thing that you've put in so far and haven't taken out.

Both of these structures can be found for just about any language you want to use. Anyway. Our dictionary will store the name of a point as the "key" - the thing we've made it look up easily - and a "predecessor" (a point name) and a "distance" (a number) as the "value". Our priority queue will store all three of these things as well, but will sort on the distance. NOTE: you can also store the dictionary information in the points themselves... especially if you have a much simpler way (references, pointers, indices into an array) to get to any particular point.

Now, for the algorithm itself.

1. The dictionary starts empty (or, if you're storing with the points themselves, start each one's distance at -1, and its predecessor set to null or another invalid value. We'll use this invalidity to show that the point hasn't been visited.). the priority queue starts with our starting point in it, with distance 0 and an invalid predecessor (an empty string would work in our case).

2. Remove a point from the priority queue, and examine it. If this point is already in the dictionary, throw this point out and start this step over. Now look at the graph (or roadmap). You'll notice that you have several possible points directly connected to it, and each one has a distance from our current point. For each one of these that isn't already in our dictionary, put an object into the priority queue, with the same name as the point, a predecessor with the name of the point we just took out of the queue, and a distance that's the sum of the distance in the object we took out of the queue plus the distance along the road (graph edge) between these two points.

3. go back to step 2, until the priority queue is empty or you have put your destination point into the dictionary.

4. look up your destination point in the dictionary, and add it to an empty list. This will be your current point.

5. change the current point to the one listed in the predecessor field of the current point. Add this new point to that list.

6. repeat step 5 until you reach a point with an invalid predecessor field... this is your starting point.

7. reverse the list, and you have a complete set of directions from start to finish.

Now, this looks like a hell of a lot of work to get one set of directions, and sure, it is. But get this: Once you've done steps 1-3 once for a given origin point, you never have to do them again for that origin. You can start at step 4 for every single destination point from that origin once these steps are done.

Dijkstra's Algorithm runs in O(n²) time. This can be proven to be the fastest graph traversal algorithm... but I'm not going to do it right now. You can look it up.

There are other pathfinding algorithms that work better when you can learn more about the layout... but if I go any deeper into this I'll be into Artificial Intelligence, and I don't know jack about it.

Vorn

Floating Point Precision (old)

2005-05-04T04:54:31Z

Imagine, for a moment, that you have a number storage format, that allows you to store five consecutive digits that appear anywhere in a number. These digits can be ANYTHING... but the rest of the digits in any number must end up being zero. So you can store 12345, or 1234500, or 0.00000012345, or 1.0005, but not 123456, or 1.02345 , because in each case we have more than five consecutive digits to store.

So what happens when we try to do 12345 + 0.1 ? Well, we probably get 12345. And that's it.

The scary thing about this is that it's not associative: (a + b) + c does not necessarily come up with the same result as a + (b + c):

(1000.4 + 0.2) + 10000 = 1000.6 + 10000 = 11001
1000.4 + (0.2 + 10000) = 1000.4 + 10000 = 11000

In a computer, numbers are not stored in decimal, but the same principle applies: you can only store a number that you only need 52 bits to store, so that the smallest number above 1 that you can store is 1.0000000000000002220446049250313080847263336181640625, and not one iota less. That excess is known as epsilon, and you'll see it a lot. If you go through my previous posts, any time a difference number repeats, it's a power of 2 times this number. The reason the algorithms I use get this close but no further is because I have intentionally limited myself to using the standard internal format of the machine, and have decided not to use more accurate but more difficult methods. Yes, I'm lazy.

There. I haven't done anything too frightening in this one, so NO CUT FOR YOU! COME BACK, ONE POST! It'll probably be about graphs.

Vorn

Exponential Functions (old)

2005-05-04T04:51:23Z

This is the last, I think, of the elementary functions that I'm going to cover.

The function a^b can look astonishingly easy (if b is an integer) or astonishingly hard (if it is not). In both situations, however, it is rather simple to get a result. Note: A must be positive, or this won't work. Like, at all.

First, we need to note that we can find e^x through a Taylor series, and an especially simple one: e^x = 1 + x/1 + x²/2 + x³/6 + .... + x^k/k! .

Now: a^b = e^{b ln(a)} . We already know how to get logarithms, so this is pretty simple. Note that you're going to need a LOT of precision in calculating ln(a) to get it to work properly.

Here's our whole process.

1. get the natural logarithm of a, and multiply it by b. Call this x.
2. set n and k to 1, and result to 0.
3. add n to result.
4. n = n * x / k.
5. increment k.
6. go to step 3.

And here's a table.

k	n	result	\|result - right answer\|
1	1	1	7
2	2.07944154167984	3.07944154167984	4.92055845832016
3	2.16203856263191	5.24148010431174	2.75851989568826
4	1.49861093395018	6.74009103826192	1.25990896173808
5	0.779068457717906	7.51915949597983	0.48084050402017
6	0.324005462958211	7.84316495893804	0.156835041061959
7	0.112291736567752	7.95545669550579	0.0445433044942076
8	0.0333577288294788	7.98881442433527	0.0111855756647286
9	0.00867068088301367	7.99748510521828	0.00251489478171507
10	0.00200335266919865	7.99948845788748	0.000511542112516139
11	0.000416585476296685	7.99990504336378	0.000094956636219834
12	7.87513768247098E-05	7.99998379474061	1.62052593948303E-05
13	1.36465737028153E-05	7.99999744131431	2.55868569176698E-06
14	2.1828655584023E-06	7.99999962417987	3.75820133236004E-07
15	3.24224380145992E-07	7.99999994840425	5.15957534474865E-08
16	4.4947042993398E-08	7.99999999335129	6.64871002697964E-09
17	5.84154677350884E-09	7.99999999919284	8.07163225147178E-10
18	7.14538531088241E-10	7.99999999990738	9.26245746768473E-11
19	8.25467280375432E-11	7.99999999998992	1.00781605283373E-11
20	9.0342681795271E-12	7.99999999999896	1.04360964314765E-12
21	9.39311627559246E-13	7.9999999999999	1.03916875104915E-13
22	9.30116009014092E-14	7.99999999999999	1.06581410364015E-14
23	8.79146303511527E-15	8	1.77635683940025E-15
24	7.94840584667887E-16	8	8.88178419700125E-16
25	6.88676887821455E-17	8	8.88178419700125E-16

Next Time: probably floating point precision.

Vorn

Trigonometry (old)

2005-05-04T04:48:04Z

So I said to myself, self, I you've written short bits on square roots, and on logarithms, why not do something on other common functions?

Self, said I, that's a great idea.

The trigonometric functions - sine, cosine, tangent, cotangent, secant, and cosecant - are extremely common in many branches of math and science, including graphics processing. Fortunately they're easy to deal with.

Tangent, Cotangent, Secant, and Cosecant are generally not calculated directly, because they can be derived easily:

tan(x) = sin(x)/cos(x)
cot(x) = cos(x)/sin(x)
sec(x) = 1/cos(x)
csc(x) = 1/sin(x)

So all we're left with is sine and cosine. Technically we can derive one from the other, but we're not going to do that. Why not? Because they're both exceedingly beautiful.

Each is generally modeled by a Taylor series, a polynomial that is defined in such a way that it gets closer and closer to the right answer the more terms you figure. Here are the Taylor series for sine and cosine:

sin(x) = SUM((-1)ⁿx²ⁿ⁺¹/(2n+1)!,n=0,inf)
cos(x) = SUM((-1)ⁿx²ⁿ/(2n)!,n=0,inf)

That's all well and good, but it doesn't get us immediately to a quick way to do it. So here's some observations:

1. in both series, each successive term is reachable from the previous by multiplying by x² and then dividing by 2n and one of 2n-1 or 2n+1, depending on which one you're doing.
2. in both series, we have the sign flipping back and forth, and they both start with the sign positive.
3. we can determine the next term given 2n, the sign, and the current term. With a little bit of trickery, we can even do the next term for both of them in the same way, after a few little bits of startup.

So let's do that.

1s. (sine) current = x; k = 1; s = 1
1c. (cosine) current = 1; k = 0; s = 1
2. result = result + current * s
3. current = current * x * x / ((k + 1) * (k + 2))
4. k = k + 2
5. s = -s
6. go to step 2, until you think you've got it close enough. There's a trick to figuring out how close you need to be for a certain thing, but it's more involved than I like.

So, here's a sine and a cosine, done the Taylor Series way. Also listed is the difference between the current result and the actual result.

SIN(pi/3)

current	k	s	result	difference
1.047 197 551	1	1	1.047 197 551	0.181 172 147
0.191 396 769	3	-1	0.855 800 781	0.010 224 622
0.010 494 502	5	1	0.866 295 283	0.000 269 880
0.000 274 012	7	-1	0.866 021 271	4.132 128 066*10^-6
4.173 443 408*10^-6	9	1	0.866 025 445	4.131 534 248*10^-8
4.160 629 841*10^-8	11	-1	0.866 025 403	2.909 559 260*10^-10
2.924 769 985*10^-10	13	1	0.866 025 403	1.521 116 566*10^-12
1.527 318 662*10^-12	15	-1	0.866 025 403	6.217 248 937*10^-15
6.157 692 399*10^-15	17	1	0.866 025 403	1.110 223 024*10^-16
1.974 463 547*10^-17	19	-1	0.866 025 403	1.110 223 024*10^-16
5.155 337 069*10^-20	21	1	0.866 025 403	1.110 223 024*10^-16

COS(pi/3)

current	k	s	result	difference
1.000 000 000	0	1	1.000 000 000	0.500 000 000
0.548 311 355	2	-1	0.451 688 644	0.048 311 355
0.050 107 557	4	1	0.501 796 201	0.001 796 201
0.001 831 636	6	-1	0.499 964 565	3.543 467 108*10^-5
3.586 810 400*10^-5	8	1	0.500 000 433	4.334 329 148*10^-7
4.370 419 717*10^-7	10	-1	0.499 999 996	3.609 056 886*10^-9
3.630 834 484*10^-9	12	1	0.500 000 000	2.177 757 973*10^-11
2.187 722 833*10^-11	14	-1	0.499 999 999	9.964 251 646*10^-14
9.996 277 270*10^-14	16	1	0.500 000 000	3.330 669 073*10^-16
3.582 400 223*10^-16	18	-1	0.500 000 000	0.000 000 000
1.033 826 696*10^-18	20	1	0.500 000 000	0.000 000 000

As you can see, this converges very quicky. By the time we've done ten iterations, we have exceeded the ability of the machine (actually Excel) to add the newest current value to the result, and we're devastatingly close to the correct value.

I think I'll do maybe one or two more of the elementary functions (I should cover exponents because they do something cute to be more general, and then maybe some scary tricks), and then we'll move on to something different.

Vorn

Logarithms (old)

2005-05-04T04:44:02Z

If you've ever done math beyond algebra you've probably used logarithms. Logarithms log_b(x) are defined such that b^log(x) = x. You probably knew that. You probably didn't know how to calculate them. Shockingly, it's really easy.

This sort of thing is easiest to do when b is 10 or when b is 2; if you need a different b (say, e), you can later divide by log_b(e) or similar.

Start by dividing x by b until 1 ≤ x < b. The number of times you have to do that is the integer part of the logarithm. If x < 1 already, multiply instead of dividing, and the integer part is negative.

Now, we need some numbers: size = 0.5.

Square the current value of x. If it is now > b, add size to the integer part and divide x by b. Halve size.

Do this again, until you have enough precision.

Here's log₁₀(2).

1 ≤ 2 < 10, so integer part is 0.

x	bit	size	result
4.000 000	0	0.500 000	0.000 000
1.600 000	1	0.250 000	0.250 000
2.560 000	0	0.125 000	0.250 000
6.553 600	0	0.062 500	0.250 000
4.294 967	1	0.031 250	0.281 250
1.844 674	1	0.015 625	0.296 875
3.402 823	0	0.007 812	0.296 875
1.157 920	1	0.003 906	0.300 781
1.340 780	0	0.001 953	0.300 781
1.797 693	0	0.000 976	0.300 781
3.231 700	0	0.000 488	0.300 781
1.044 388	1	0.000 244	0.301 025
1.090 748	0	0.000 122	0.301 025

Technically you can do this using a higher power (instead of squaring), but then you need to divide multiple times in each step.

Vorn

Squaring and Taking Square Roots (old)

2005-05-04T04:43:02Z

Last night I dreamed of algorithms. That sadness out of the way, let me show you a way to square a number, and then reverse the process to get the square root.

Before this will make any sense to you:

	(x + k)²
=	x² + 2xk + k²
=	x² + (2x + k)k
=	x² + ((x + k) + x)k

What this means is essentially this: if I know the square of one number, and I need the square of another number, I can add the first number and the second number, multiply it by the difference between the two numbers, and add that to the square of the original number. For example:

30² = 900
32² = 900 + (30 + 32) * 2 = 900 + 124 = 1024

Now, we know that 10² = 100, and that 100² = 10,000. Each digit we go away from the decimal point in the original brings us 2 digits away in the square. So what we're going to do is:

0. set up a table so that one digit in the top row is equivalent to two digits in any other row.
1. start at the leftmost digit. Square it, and put the result directly under that digit.
2. advance to the right one digit. If there are no more digits, go to step 6.
3. take the digits before our current digit, treat them as a number, double it, and append the current digit to the end of that.
4. multiply that number by the current digit, and put the result under the current digit, using cells to the left if we have to.
5. go to step 2.
6. add all rows except for the topmost row. This is your result.

8	6	7	.	5	3	0	9
64			.					8	*	8	=	64
9	96		.					166	*	6	=	996
1	20	89	.					1727	*	7	=	12089
	8	67	.	25				17345	*	5	=	86725
		52	.	05	09			173503	*	3	=	520509
			.			0		1735060	*	0	=	0
		1	.	56	15	54	81	17350609	*	9	=	156155481
75	26	09	.	86	24	54	81

Now you're probably looking at this and going, "Man. That's a really crappy way to do this." And you'd be right. There's much easier ways to square a number; the usual method is generally much more efficient because you're always multiplying by the same number. But the thing is that the fact that we can square a number this way means that we can reverse the process to get the square root of a number.

here's how we take the square root:

1. chop up the number into 2-digit chunks, starting at the decimal point: 6 35 . 50 is okay, 63 5.5 is not.
2. bring down the leftmost unused pair of digits and place it to the right of any remainder (the remainder comes from step 4). If the remainder is 0 and there are no more non-zero number pairs, the algorithm terminates.
3. take the result so far and find the largest digit k such that (20*result + k) * k is less than the number on the remainder line.
4. subtract (20 * result + k) * k from the remainder line to make a new remainder line. Append k to the end of the result.

8	6	7	.	5	3	0	9
75	26	09	.	86	24	54	81
64			.					8	*	8	=	64
11	26		.
9	96		.					166	*	6	=	996
1	30	09	.
1	20	89	.					1727	*	7	=	12089
	9	20	.	86
	8	67	.	25				17345	*	5	=	86725
		53	.	61	24
		52	.	05	09			173503	*	3	=	520509
		1	.	56	15	54
			.			0		1735060	*	0	=	0
		1	.	56	15	54	81
		1	.	56	15	54	81	17350609	*	9	=	156155481
			.				0

Yes, there are faster ways of getting square roots... but how many of them actually give you an exact answer when it exists?

I'm mainly proud of myself because I figured out how they managed to come up with this silly algorithm.

Also, I have Tommy Tutone stuck in my head.

Vorn

Gaussian Elimination (old)

2005-05-04T04:41:09Z

For all that I talk about code, I sure don't write it very fast.

Gaussian Elimination is a very simple process that is used to solve systems of equations in the form of an augmented matrix (a square matrix with a vector attached to the side). Basically you use row operations (swap two rows, multiply one row by a number, and add a multiple of one row to a second row) on the matrix to make it so all the non-zero values are on the diagonal. Here's some code.

template <size_t size>
MathVector<size> gaussianElimination(SquareMatrix<size> m, MathVector<size> v) {
  size_t j,k;
  double mult;
  for (k = 0; k < size; k++) {
    j = k;
    while(m.index(j,k) == 0) { // find a row we haven't used that has a nonzero value in the kth column
      j++;
      if (j == size) throw NonDiagonalizableMatrix; // no such row exists; we're screwed
    }
    m.rowSwap(j,k); // make said row the kth row.
    swap(v[j],v[k]); // do the same to the vector
    for (j = 0; j < size; j++) { // manipulate each row using the current primary row
      if (j != k) { // don't manipulate the primary row.
        mult = -m.index(j,k);
        m.rowMult(j, m.index(k,k); // multiply the secondary row by the kth value of the kth row
        v[j] *= m.index(k,k);
        m.rowAdd(j, k, mult); // add enough of the primary row to the secondary row to zero out the kth value
        v[j] += v[k] * mult;
      }
    }
  }
  for (k = 0; k < size; k++)
    v[k] /= m.index(k, k); // set all the values in the matrix to 1, but only bother with it on the vector side.
  return v;
}

Using Gaussian Elimination is fine, if you're only going to be solving the system of equations once. However, often you want to go one better than that, and solve the equations several times; in which case you're probably better off using Gauss-Jordan Elimination (basically, the same process, except the vector is replaced by a specific matrix) to get the inverse of the matrix, which will allow you to find the inverse of the matrix, and solve the system of equations very quickly for a lot of different outputs.

Vorn

	R	P	L
I		X
Q	X		X
Q	X
Q	X
C		X
C		X
C		X
Q	X		X
C		X
Q	X		X
C		X
C		X
Q	X		X
C		X
Q	X		X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
C		X
C		X
C		X
C		X
C		X
C		X
C		X
C		X
Q	X		X
C		X
Q	X		X
C		X
	19	18	7

	R	P	L
I		X
Q	X		X
Q	X
Q	X
C		X
C		X
C		X
Q	X		X
C		X
Q	X		X
C		X
C		X
Q	X		X
C		X
Q	X		X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
C		X
C		X
C		X
C		X
C		X
C		X
C		X
C		X
Q	X		X
C		X
Q	X		X
C		X
	19	18	7

	R	P	L
I		X
Q	X		X
Q	X
Q	X
C		X
C		X
C		X
Q	X		X
C		X
Q	X		X
C		X
C		X
Q	X		X
C		X
Q	X		X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
Q	X
C		X
C		X
C		X
C		X
C		X
C		X
C		X
C		X
Q	X		X
C		X
Q	X		X
C		X
	19	18	7