Terminology and basic concepts
1.Implementation : executable computer code that follows the process defined by the algorithm
2.Running time of a sequential algorithm = the number of primitive operations or “steps” on 3.The RAM (random access machine) model:
• one memory (infinite)
• one processor
• instructions are executed sequentially. The instructions are simple: assignment, addition, multiplication, comparison, etc.
4. Message complexity = the total number of messages flowing into the channels (over all the channels) until the distributed system reaches a terminal configuration.
5. Parallel computing: many computations are executed simultaneously
6. Distributed computing: distributed use of multiple resources (e.g. memory, CPU, bandwidth) to solve a computational problem
7. Computer Cluster: multiple stand-alone machines running in parallel across a high speed network
8.Grid computing: a form of distributed computing in which a super and virtual computer is composed of a cluster of networked, loosely-coupled computers; the resources offered by these computers are coordinated to solve large-scale computation problems
9. Cluster computing: a cluster computer is a group of interconnected computers (usually high-speed networking) working together closely
10.Von Neumann model: The classical von Neumann machine has a single CPU and a single main memory (RAM); CPU has a single control unit, an arithmetic-logic unit (ALU), and (few) registers
11. Von Neumann bottleneck: There is a single bus, how fast we can transfer the instructions & data between memory and registers
*12.The first extension of Von Neumann model: pipelining
13.Classification based on the instruction and data streams: Flynn's taxonomy, the best known classification scheme for parallel computers ;
The instruction stream (I) and the data stream (D) can be either single (S) or multiple (M)
Four combinations: SISD, SIMD, MISD, MIMD
–SISD: single-instruction-single-data (Most important member is a sequential computer )
![](http://pic3.zhimg.com/v2-835cc4f925a786db33132392936c0da6_b.jpg)
– MISD: multiple-instructions-single-data ( One of the two most important in Flynn’s Taxonomy )
![](http://pic4.zhimg.com/v2-85b92ea602535edb4b716556969d94f7_b.jpg)
– SIMD: single-instruction-multiple-data ( Relatively unused terminology)
![](http://pic2.zhimg.com/v2-9dff5a5cc9a1aba4dcc1126a8e168fa5_b.jpg)
– MIMD: multiple-instructions-multiple-data(Important)
![](http://pic2.zhimg.com/v2-42e0705f1bd0b73e27f2df4e653fdc6d_b.jpg)
14. Amdahl's law:------n is fixed
![](http://pic2.zhimg.com/v2-2cbe2b40b00317396abdab1a7340e6d1_b.jpg)
![](http://pic3.zhimg.com/v2-c4e46db61b65a587910ad14e917bd882_b.jpg)
![](http://pic1.zhimg.com/v2-ac58a684925fbc89380576994afc0bec_b.jpg)
15. Gustafson's law: -----n varies
![](http://pic3.zhimg.com/v2-e9b8f9ca2b64e4dbd037a8fef61a179a_b.jpg)
![](http://pic3.zhimg.com/v2-97d923d34541ed54f86d9fa0e761bad2_b.jpg)
![](http://pic1.zhimg.com/v2-b24184321febe9e9bf34a34142b59d38_b.jpg)
16.Dependency Conditions:
• Data Dependency: two or more instructions share the same data.
• Control Dependency: instructions contain one or more control structures (if-then-else, if-then, if-else) that can change the content of data; problem arises because the order of execution in control structures is not known at run-time
• Resource Dependency: there are one or more resources that are accessed by different instructions and only an instruction can access the resource at time
17.Data Dependency:
– Flow dependence : If instruction I2 follows I1 and output of I1 becomes input of I2, then I2 is said to be flow dependent on I1.
– Anti-dependence : When instruction I2 follows I1 such that output of I2 overlaps with the input of I1 on the same data.
– Output dependence : When output of the two instructions I1 and I2 overlap on the same data, the instructions are said to be output dependent.
– I/O dependence : When read and write operations by two instructions are invoked on the same file.
18.Bernstein's Conditions:
1). If process Pi writes to a memory cell Mi , then no process Pj can read the cell Mi.
2). If process Pi read from a memory cell Mi, then no process Pj can write to the cell Mi.
3). If process Pi writes to a memory cell Mi, then no process Pj can write to the cell Mi.
19. Portions of code that are not dependent (are independent) can be parallelized.
20.Barrier:a type of synchronization method: any process must stop at this point and cannot proceed until all other processes reach this barrier.
21.Shared-Memory Programming:
int private_x;
shared int sum = 0;
shared int s=1; /* Compute private_x */
P(&s); /* Wait until s = 1 */
s = 0; /* Enter critical section */
sum = sum + private_x; /* Critical section */
V(&s); /* Exit critical section */
Barrier();
if (I am process 1)
printf(“sum = %d\n”, sum);
22.Routing: If two nodes are not connected directly then the data needs to be routed
– Store-and-forward routing: the packets of a message are collected one by one at each intermediate node; when all the pieces are received, the message is transmitted, packet by packet, to the next node
![](http://pic2.zhimg.com/v2-3286d0619e8fe9c1fa11b7629dcdc641_b.jpg)
– Cut-through-routing: pieces of a message are sent one by one and there is no waiting at the intermediate node for the rest of the packets
![](http://pic1.zhimg.com/v2-524ee3ee09dcfd145c74380dcf8b24d8_b.jpg)