TriNetre - Archive for October 23, 2003

(no longer updated)

The basic way in which a DA works is explained better using a more real-life example. Suppose you have a narrow road from C to S.

  ---------------------------
C           Road                S
  ---------------------------

Let us look a bit deeper on how a DA works, assuming HTTP download. Let us simplify it further (for now) by assuming that the file is from one single server.

When you need to get a file (stones) from server (S) to client (C), you start an HTTP connection using TCP. The problem (?) with TCP is that it is a very friendly, conservative protocol. It makes no assumption on what kind of connection exists between C and S. So, in the beginning it starts by transferring a small amount of data from S to C. When that works fine (without any packet drop), TCP increases the amount of data transferred next time by a small amount and this goes on until the whole file is fetched. A lonely man at work.

What a DA does is simple. First it finds out the total size of the file to be fetched. Next it calls up its army of 10 connections (men). It then divides the file into 10 parts and tells each of them which part of the file to fetch. Then each of these connections go about doing their work in parallel, fetching their part of the file. In HTTP, this is done using the "Range" specification. Each of these connections work like a single connection. However, collectively it is able to fetch more data at any point in time. If the lonely worker was able to get 10 bytes of data in the first fetch, the army would be able to get 10*10 byes of data. Thus collectively, it is able to fetch the whole file in less time.

More evolved DA has more tricks up its sleeve. Sometimes, it is the server that is a bottleneck. Think about a Slashdotted server. It might be better to leave alone (or sparingly use) an overtaxed server. Similarly, if 5 servers have the same file, it might be better to distribute the work to these 5 server, as long as these 5 are equally responsive. So a DA finds out mirrors of the file and each parallel connection tries to fetch its piece from different server. Smart, aren't they?

If it is such an open and shut case, why so much controversy? Well, because theory and practice are not always best of friends. If your machine is such that it is low on memory or CPU power, it might actually take more time to get your army out on foot and do their work than get a lonely hard working connection do its job. Furthermore, whatever be the number of connections you use, there is a limit to the amount of data that can be fetched at any time, limited by the bandwidth of the connection. You know, the road is only that wide. If you send more soldiers, they will just knock against each other and the stones will fall off from their hands. Another thing to take into consideration (if you are a protocol buff) is the overhead involved when you use more connections. Think about the 3 way handshake, the 4 packet connection close and ACK packets flowing across the network for each connection.