Low Power Viterbi Decoder Designs - University Of Manchester

Transcription

Low Power Viterbi Decoder DesignsA thesis submitted to The University of Manchester for the degree ofDoctor of Philosophyin the Faculty of Engineering and Physical Sciences2007Wei ShaoSchool of Computer Science

CONTENTS1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1.11Digital communication and coding . . . . . . . . . . . . . . . . . . . . .11.1.1Information theory . . . . . . . . . . . . . . . . . . . . . . . . .31.1.2Coding theory . . . . . . . . . . . . . . . . . . . . . . . . . . . .61.2Decoding Convolutional Codes . . . . . . . . . . . . . . . . . . . . . . .131.3Channel Coding Applications . . . . . . . . . . . . . . . . . . . . . . .141.3.1Channel coding application considerations . . . . . . . . . . . .141.3.2Low power applications . . . . . . . . . . . . . . . . . . . . . . .151.4Objectives and summary of this work . . . . . . . . . . . . . . . . . . .161.5Contributions of this work . . . . . . . . . . . . . . . . . . . . . . . . .161.6Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .182. Convolutional Coding and Viterbi Decoding Algorithm . . . . . . . . . . . .192.1Convolutional code structure . . . . . . . . . . . . . . . . . . . . . . . .192.1.1Tree and trellis representation of convolutional codes . . . . . .222.1.2Rate m/n codes . . . . . . . . . . . . . . . . . . . . . . . . . . .252.2Distance properties of convolutional code . . . . . . . . . . . . . . . . .282.3Viterbi decoding algorithm . . . . . . . . . . . . . . . . . . . . . . . . .312.3.1Hard-decision Viterbi decoding algorithm . . . . . . . . . . . . .312.3.2Soft-decision Viterbi decoding algorithm . . . . . . . . . . . . .34Convolutional codes performance with Viterbi algorithm . . . . . . . .372.4.1372.4Error event and union bounds . . . . . . . . . . . . . . . . . . .

iiContents2.4.2Convolutional code performance . . . . . . . . . . . . . . . . . .39summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .423. Viterbi decoder and its power dissipation . . . . . . . . . . . . . . . . . . . .442.53.1Viterbi decoder design . . . . . . . . . . . . . . . . . . . . . . . . . . .443.1.1BMU design . . . . . . . . . . . . . . . . . . . . . . . . . . . . .453.1.2PMU design . . . . . . . . . . . . . . . . . . . . . . . . . . . . .503.1.3SMU design . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52Power dissipation in the Viterbi decoder . . . . . . . . . . . . . . . . .553.2.1CMOS Circuitry power dissipation . . . . . . . . . . . . . . . .563.2.2Design flow and power estimation . . . . . . . . . . . . . . . . .583.2.3Testing framework and noise generator design . . . . . . . . . .603.2.4Viterbi decoder power consumption . . . . . . . . . . . . . . . .61Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .694. Low power adaptive Viterbi algorithm and decoder design . . . . . . . . . .713.23.34.1T-algorithms and adaptive T-algorithm . . . . . . . . . . . . . . . . . .714.2A new adaptive Viterbi algorithm . . . . . . . . . . . . . . . . . . . . .734.2.1Error pattern in Viterbi algorithm . . . . . . . . . . . . . . . . .734.2.2No-error code words path identification . . . . . . . . . . . . . .754.2.3A new adaptive Viterbi algorithm and decoder design . . . . . .80BER and power analysis of the proposed adaptive Viterbi decoder . . .814.3.1Matlab Simulation Results . . . . . . . . . . . . . . . . . . . . .824.3.2FPGA simulation results . . . . . . . . . . . . . . . . . . . . . .854.3.3Estimated power consumption . . . . . . . . . . . . . . . . . . .864.3.4Comparison of other low power designs . . . . . . . . . . . . . .864.3.5Possible applications . . . . . . . . . . . . . . . . . . . . . . . .87Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .884.34.4

Contents5. Low power SMU design . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5.15.25.35.45.5iii90Design of SMU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .905.1.1Major SMU operations . . . . . . . . . . . . . . . . . . . . . . .915.1.2Asynchronised SMU timing and design . . . . . . . . . . . . . .93New Trace Back SMU design. . . . . . . . . . . . . . . . . . . . . . .955.2.1Timing feature of the trace back convergence . . . . . . . . . . .955.2.2Overview of the new SMU architecture . . . . . . . . . . . . . .965.2.364-bit global winner encoding . . . . . . . . . . . . . . . . . . .975.2.4New trace back path architecture . . . . . . . . . . . . . . . . .985.2.5Local Winner Memory . . . . . . . . . . . . . . . . . . . . . . . 1025.2.6Global Winner Distributor . . . . . . . . . . . . . . . . . . . . . 1035.2.7Output Generator . . . . . . . . . . . . . . . . . . . . . . . . . . 104Timing in the new SMU design . . . . . . . . . . . . . . . . . . . . . . 1075.3.1Positive timing skew . . . . . . . . . . . . . . . . . . . . . . . . 1075.3.2Negative timing skew . . . . . . . . . . . . . . . . . . . . . . . . 1095.3.3Timing of the scaled new trace back design . . . . . . . . . . . . 112Test results of the new SMU design . . . . . . . . . . . . . . . . . . . . 1135.4.1CMOS implementation results . . . . . . . . . . . . . . . . . . . 1135.4.2FPGA implementation results . . . . . . . . . . . . . . . . . . . 116Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1186. Low power PMU design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1206.16.2Existing low power implementations of PMU . . . . . . . . . . . . . . . 1206.1.1Analog design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1206.1.2Low power compare-select-add (CSA) design . . . . . . . . . . . 123Low power design of the PMU . . . . . . . . . . . . . . . . . . . . . . . 1246.2.1Performance analysis with BM and PM Capping . . . . . . . . . 1256.2.2Low power implementation of the PMU with PM and BM capping129

ivContents6.3Test Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1316.4Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1327. Conclusion and Future Works . . . . . . . . . . . . . . . . . . . . . . . . . . 1357.17.2Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1357.1.1Summary of proposed low power Viterbi decoder designs . . . . 1357.1.2Contributions and new methods proposed for efficient decoding140Future works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142Appendix151A. Matlab Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153A.1 Matlab program for comparing union bound with simulated BER ofViterbi decoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153A.2 Matlab program for comparing union bounds of different code rates . . 155A.3 Matlab program for comparing union bounds of different constraint lengths157A.4 Matlab program for comparing union bounds of hard and soft decision159B. The AWGN generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160B.1 The AWGN generator design overview . . . . . . . . . . . . . . . . . . 160B.2 The f function implementation . . . . . . . . . . . . . . . . . . . . . . 162B.3 The AWGN generator design verification . . . . . . . . . . . . . . . . . 165

LIST OF FIGURES1.1Simple communication system. . . . . . . . . . . . . . . . . . . . . . . .42.1A simple R 1/2, k 3 convolutional encoder . . . . . . . . . . . . . . .202.2Code tree for R 1/2, k 3 convolutional code. . . . . . . . . . . . . . .222.3Trellis structure for R 1/2, k 3 convolutional code. . . . . . . . . . . .242.4Encoder structure for R 2/3, k 4 convolutional code. . . . . . . . . .252.5Trellis structure for R 2/3, k 4 convolutional code. . . . . . . . . . . .262.6Trellis structure for R 2/3 punctured convolutional code with a mothercode R 1/2, k 3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2.7Trellis structure for R 1/2, k 3 code with two highlighted code wordpaths begin and end in the same state. . . . . . . . . . . . . . . . . . .2.828State transition diagram of R 1/2, k 3 code. The input and outputstates are zero state. . . . . . . . . . . . . . . . . . . . . . . . . . . . .2.92729The communication channel model using a convolutional code and MLdecoding. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .322.10 The trellis structure shows the Viterbi decoding process of a R 1/2,k 3 code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .332.11 The convergence of the code word paths in the Viterbi decoding process. 342.12 The normal probability distribution of a random variable. . . . . . . . .362.13 The error event with Viterbi decoding. . . . . . . . . . . . . . . . . . .372.14 Union bound and simulated BER for R 1/2, k 7 code with harddecision Viterbi decoding and QPSK (Quadrature Phase Shift Keying)modulation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39

viList of Figures2.15 Union bounds for R 1/3, 1/2, and 2/3, code with soft-decision Viterbidecoding and QPSK modulation. . . . . . . . . . . . . . . . . . . . . .402.16 Union bounds for R 1/2, k 3, 4, 5, 6, 7, and 8 code with soft-decisionViterbi decoding and QPSK modulation. . . . . . . . . . . . . . . . . .412.17 Union bounds for R 1/2, k 7 code with both Soft and Hard decisionViterbi decoding and QPSK modulation. . . . . . . . . . . . . . . . . .423.1Classical three functional block of a rate 1/2 Viterbi decoder design. . .443.2The Euclidian distance between the hypothesised code word and thereceived signal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .473.3Three-bit quantisation in a 2-dimensional space. . . . . . . . . . . . . .483.4The butterfly state transition diagram represents state transitions of aconvolutional encoder of constraint lenght k. . . . . . . . . . . . . . . .3.550A rate 1/2, n states PMU architecture shows the recursively calculationsof state metrics value. The Global Winner Generator, shown in dashedlines, provides global winners information for a trace back PMU. . . . .3.6A 4-state register exchange implementation of the SMU design. Thebold arrows indicate the ideal path of the encoder states. . . . . . . . .3.75153A possible trace back SMU implementation using memory. It also requires global winner information in order to reduce the trace back depth. 543.8Design and verification process of the FPGA implementations . . . . .603.9The test framework of the FPGA implementaion. . . . . . . . . . . . .613.10 The optimum number of test bits for power simulations. . . . . . . . .633.11 The Viterbi decoder power consumption at different Eb/No levels. . . .643.12 Energy per corrected error of the Viterbi decoders of k 3 and 7. . . . .653.13 The estimated dynamic power consumption of the Viterbi decoder withdifferent constraint lengths at Eb/No 2dB. . . . . . . . . . . . . . . .66

List of Figuresvii3.14 Energy per corrected error of the Viterbi decoder with different constraint lengths at Eb/No 2dB. . . . . . . . . . . . . . . . . . . . . . .673.15 The blocks dynamic power consumption of a standard (R 1/2, k 7)Viterbi decoder. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4.168The error pattern in the code words and the corresponding decoded dataof a R 1/2, k 7 Viterbi decoder at Eb/No 3dB. . . . . . . . . . . . .744.2Simple convolutional coded digital communication system model. . . .764.3Architecture to identify the zero Hamming distance path for a rate 1/2and k 7 convolutional codes. . . . . . . . . . . . . . . . . . . . . . . .4.4Architecture of the proposed 3-bit soft decision adaptive Viterbi decoderfor rate 1/2 and k 7 code. . . . . . . . . . . . . . . . . . . . . . . . .4.54.983BER performance of adaptive Viterbi algorithm with L from 4 to 28,R 1/2, k 7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4.882Comparison of pre-decoding and Viterbi decoding operations, R 1/2,k 7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4.781BER performance of the adaptive Viterbi algorithm with L 35, R 1/2,k 7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4.67884Percentages of pre-decoding and Viterbi decoding operations, R 1/2,k 7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85BER performance from FPGA tests, R 1/2, k 7. . . . . . . . . . . . .864.10 Estimated dynamic power consumption of the adaptive Viterbi decoderon Virtex4 XC4VSX35, R 1/2, k 7. . . . . . . . . . . . . . . . . . . .875.1Memory architecture of the one-pointer trace back SMU. . . . . . . . .925.2The SMU architecture of the asynchronous design from [1]. . . . . . . .945.3The trellis structure shows the Viterbi decoding process of a R 1/2,k 3 code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .95

viiiList of Figures5.4The new R 1/2, k 7, 64-state SMU architecture, which consists of fourmajor blocks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .965.5Trace back path of the new SMU design. . . . . . . . . . . . . . . . . .985.6The timing of the trace back path multiplexer. . . . . . . . . . . . . . . 1005.7The one stage trellis structure of the Trace back unit. . . . . . . . . . . 1015.8Local winner memory of the new SMU design. It uses latch registers tostore local winner information. . . . . . . . . . . . . . . . . . . . . . . . 1025.9Global Winner Distributor of the new SMU design. . . . . . . . . . . . 1035.10 Timing of the global winner buses A and B . . . . . . . . . . . . . . . . 1045.11 Metastability and a simple flip-flop synchronizer. . . . . . . . . . . . 1055.12 Metastability simulation circuit in measuring τ . . . . . . . . . . . . . . 1065.13 Trace back status in positive timing skew situation. . . . . . . . . . . . 1085.14 Trace back gap caused by timing skew. . . . . . . . . . . . . . . . . . . 1095.15 Trace back status at various time points. . . . . . . . . . . . . . . . . . 1105.16 Post-layout of the new SMU core. . . . . . . . . . . . . . . . . . . . . . 1145.17 BER performance from FPGA tests. . . . . . . . . . . . . . . . . . . . 1176.1The conventional architecture of the ACS unit. . . . . . . . . . . . . . . 1236.2The low power ACS unit architecture proposed in [2]. . . . . . . . . . . 1246.3The BER performance variation of the Viterbi decoder with differentBM and PM capping levels at Eb/No of 3dB. . . . . . . . . . . . . . . 1256.4The BER performance of the Viterbi decoder with the variation of BMcapping level at Eb/No of 3dB. . . . . . . . . . . . . . . . . . . . . . . 1266.5The BER performance of the Viterbi decoder with the variation of PMcapping level at Eb/No of 3dB. . . . . . . . . . . . . . . . . . . . . . . 1286.6The architecutre of the BM and PM capping ACS unit. . . . . . . . . . 1306.7The BER performance of the proposed low power ACS design. . . . . . 1316.8The power dissipation of a PMU with the new ACS architecture. . . . . 133

List of FiguresixB.1 Architecture of the AWGN generator . . . . . . . . . . . . . . . . . . . 161B.2 Uniform segementation of f (x). . . . . . . . . . . . . . . . . . . . . . 162B.3 A example of 8-bit non-uniform segementation of f (x) . . . . . . . . . 163B.4 f (x) architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164B.5 Comparison of the uncoded BER . . . . . . . . . . . . . . . . . . . . . 165B.6 Comparison of the decoded BER . . . . . . . . . . . . . . . . . . . . . 166

LIST OF TABLES3.1Branch weight scheme for 2-symbol, 3-bit quantisation. . . . . . . . . .503.2Features of the Virtex-4 SX class devices. . . . . . . . . . . . . . . . . .605.1Minimum and maximum trace back stages at a 0.18µm geometry. . . . 1125.2Minimum and maximum delays of each trace back stage for differentgeometries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1125.3Minimum and maximum trace back stages at 50MHz and L 64. . . . . 1135.4Minimum and maximum trace back stages at 100MHz and L 62. . . . 1135.5Characteristics of the new SMU core . . . . . . . . . . . . . . . . . . . 1145.6New SMU BER and power consumption in decoding 1/2 codes . . . . . 1155.7New SMU BER and power consumption in decoding 2/3 codes . . . . . 1155.8The proposed SMU power consumption comparing with other low powerSMU designs at 1.8V and 180nm. . . . . . . . . . . . . . . . . . . . . . 1155.9Number of slices the Viterbi decoder occupied with the new SMU architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1165.10 Dynamic power consumption of the Viterbi decoder with the new SMUImplemented on FPGA . . . . . . . . . . . . . . . . . . . . . . . . . . . 1176.1The optimum BM caps for R 1/2, k 7 Viterbi decoder. . . . . . . . . 1276.2The optimum PM caps for R 1/2, k 7 Viterbi decoder. . . . . . . . . 1296.3FPGA simulated power of a R 1/2 k 7 PMU with the new ACS architecture at 50MHz. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132B.1 Non-uniform segment in 8-bit binary numbers. . . . . . . . . . . . . . . 164

THE UNIVERSITY OF MANCHESTERABSTRACT OF THESIS submitted by: Wei Shaofor the Degree of Ph.D. and entitled:Low Power Viterbi Decoder DesignsDate of submission:30/03/2007This thesis represents the research work of developing new approaches for implementingViterbi decoder designs to minimize computation complexity and power consumption.This work examines the decoding process of the Viterbi algorithm, the architecture ofthe Viterbi decoder, and the implementations of the basic functions. This enables thedesign problems to be discovered. Then a variety of low power design techniques aredescribed and applied to the decoder design to improve its power efficiency.The new designs are tested by simulations on both software and hardware. The resultsgive a clear view of the improvement of the modifications and enable a novel generalmethodology for significantly reducing complexity of decoding convolutional codes tobe proposed.

DeclarationNo portion of the work referred to in this thesis has been submitted in support of anapplication for another degree or qualification of this or any other university or otherinstitute of learning.

Copyrighti Copyright in text of this thesis rests with the Author. Copies (by anyprocess) either in full, or of extracts, may be made only in accordance withinstructions given by the Author and lodged in the John Rylands UniversityLibrary of Manchester. Details may be obtained from the librarian. Thispage must form part of any such copies made. Further copies (by anyprocess) of copies made in accordance with such instructions may not bemade without the permission (in writing) of the Author.ii The ownership of any intellectual property rights which may be describedin this thesis is vested in the University of Manchester, subject to anyprior agreement to the contrary, and may not be made available for use bythird parties without the written permission of the University, which willprescribe the terms and conditions of any such agreement.iii Further information on the conditions under which disclosures and exploitation may take place is available from the Head of the Department of Manchester Business School.

AcknowledgementsI wish to acknowledge Engineering and Physical Sciences Research Council (EPSRC),who kindly funded the project, and the previous works of low power Viterbi decoderdesigns accomplished by The University of Manchester, Queen’s University of Belfast,and The University of Sheffield.Many individuals have, in one way or another, influenced me during the work onthis thesis - for this I am most grateful. In particular, I wish to thank Dr. LindaBrackenbury, who has guided me down the academic road, and without her, this thesiswould not have been possible. I am truly fortunate to have a supervisor whom I canalso consider to be a good friend and who has during the last couple of years continuedto offer help and advice on my work as well as other aspects of life.Amongst my colleagues in the Advanced Processor Technique (APT) group thanks aredue to Jeff Pepper for providing technique support and to Eve, although she finishedher PhD and left the school at the end of 2005, for advising design and testing toolsand techniques. I am also grateful to the rest of APT group members, who took timefrom their busy schedules and so patiently listened to all my seminars.Especially, I would like to give my special thanks to my parents and Qiuping, for theirlove and support during my engagement with this research.

DEDICATIONTo the memory of my grandmaYongjing Niewho taught me the value of hard work

1. INTRODUCTION“Attention, the Universel! By kingdoms, right wheel! ” This prophetic phrase representsthe first telegraph message on record. Samuel F. B. Morse sent it over a 16 km line in1838. Thus a new era was born: the era of electrical communication.Now, over a century and a half later, communication engineering has advanced tothe point that earthbound TV viewers watch astronauts working in space. Telephone,radio, and TV are integral parts of our life. Long-distance circuits span the globe carrying text, data, voice, and images. Computers talk to computers via intercontinentalnetworks. Wireless personal communication devices keep us connected wherever wego. Certainly great strides have been made since the days of Morse.This thesis describes the research work that has been done in order to improve thepower efficiency of the Viterbi decoder in digital communication systems. In particular,a novel adaptive Viterbi decoder is presented plus, a lower power Path Metric Unit(PMU) and Survivor Memory Unit (SMU) designs that have been developed. Theseare discussed together with the results from testing them.1.1 Digital communication and codingIt is remarkable that the earliest form of electrical communication, namely telegraphydeveloped by Samuel Morse in 1837, was a digital communication system. AlthoughMorse was responsible for the development of the first electrical digital communicationsystem, the beginnings of what we now regard as modern digital communications system from the work of Nyquist in 1924. His studies led him to conclude that for binarydata transmission (transmitting one of two numbers, 0 or 1) over a noiseless channel

21. Introductionof bandwidth W Hertz, the maximum pulse rate is 2W pulses per second without anycross symbol interference.Hartley extended this work in 1928 to non-binary data transmission, while Kolmogorov and Wiener independently in 1939 and 1942, respectively, solved the problemof optimally estimating a signal in the presence of additive noise. In 1948 ClaudeShannon established the mathematical foundation for information transmission andderived fundamental limits for digital communication systems. His work can arguablybe considered as the true beginning of the information age.Another important contribution to the field of digital communication is the workof Kotelnikov in 1947, who provided a coherent analysis and consequently a principlefor optimal design of such systems. His work was later extended by Wozencraft andJacobs in 1965, leading to the principles used to design the communication systems oftoday.The work of Hamming in 1950 on error control coding to combat detrimental effectsof channel noise completes the classic contributions to modern digital communicationsystems.Of more modern contributions, the Viterbi decoding algorithm for trellis codes,proposed by Andrew Viterbi in 1967 is now found in almost all wireless communicationsystems. Efficient error control decoding makes mobile communication systems whatthey are today. The latest significant leap forward for improvements of communicationssystems was in 1993 with the discovery of the “turbo principle” by Berrou and Glavieux.The special turbo codes developed based on these principles can be efficiently decodedusing a very powerful iterative signal processing approach. The resulting coding systemperforms very close to fundamental limits for a range of different channels. In practicalterms, this leads to the most efficient use of bandwidth and power, which is veryimportant for portable wireless devices.In practice, the subject of digital communications involves the transmission of information in digital form from a source that generates the information to one or more

1.1. Digital communication and coding3destinations. In particular, it includes the concepts of source coding including entropyand rate-distortion, the characterization of communication signals and systems, optimal receivers, carrier and symbol synchronization, channel capacity and coding, blockand convolutional codes, signal design for band-limited channels, adaptive equalization,etc.Theoretically, communication theory consists of two major domains: informationtheory and coding theory.1.1.1 Information theoryInformation theory is the study of how the amount of content in a stream of datamay be evaluated, and how fast it may in principle be shipped from place to placeby a given communication channel [3]. The channel may need the data in a specificform and may corrupt it by randomly introducing errors. The subject is thus builton discrete probability theory as its mathematical base. It is somewhat high level inhierarchy, giving bounds and existence proofs without always any explicit means ofimplementation.When discussing information theory it is hard not to mention Claude Elwood Shannon (April 30, 1916 - February 24, 2001), an American electrical engineer and mathematician. He has been called “the father of information theory”, and was the founderof practical digital circuit design theory. In 1948 Shannon published A MathematicalTheory of Communication in two parts in the July and October numbers of the BellSystem Technical Journal. This work focused on the problem of how to best encodethe information a sender wants to transmit. In this fundamental work he used tools inprobability theory, developed by Norbert Wiener, which were in their nascent stages ofbeing applied to communication theory at that time. Shannon developed informationentropy as a measure for the uncertainty in a message while essentially inventing whatis now known as the dominant form of “information theory”.One of the most fundamental results of this theory is Shannon’s source coding

41. Introductiontheorem, which establishes that on average the number of bits needed to represent arandom variable X which is given by the entropy H(X) and is defined as [4]:H(X) Xp(x)logp(x)(1.1)x Xwhere x and p(x) represent the possible values of X and their probability, respectively.This equation plays a central role in information theory as measurements of information, choice and uncertainty, reflecting the real life fact that an unusual messagecontains more information than a normal one and thus may be more difficult for usto understand. Therefore, more bits are required in order to describe it more clearlythan a normal message. Mackay [3] summarizes this theorem as: “ N independentidentically-distributed (i.i.d.) random variables each with entropy H(X) can be compressed into more than NH(X) bits with negligible risk of information loss, as N tendsto infinity; but conversely, if they are compressed into fewer than NH(X) bits it is virtually certain that information will be lost.” When applying the source coding theoremto communications over a noisy channel, Shannon invented the noisy channel codingtheorem. This states that reliable communication is possible over noisy channels provided that the rate of communication is below a certain threshold called the channelcapacity. This is also called the Shannon limit or Shannon capacity [4].Considering a simple communications process over a discrete channel, as shown inFig. 1.1: Simple communication system.

1.1. Digital communication and coding5Figure 1.1, here X represents the space of all possible values of the messages received,and Y similarly is the space of all possible values of messages transmitted during aunit time over this channel. The possible rate of information transmission, R, wouldbe obtained by subtracting the average rate of conditional entropy, Hy (x), from theentropy of the source, H(x) [3].R H(x) Hy (x)(1.2)The conditional entropy Hy (x) measures the average ambiguity of the received signal.The capacity C of a noisy channel is the maximum possible rate of transmission, i.e.,the rate when the source is properly matched to the channel [3]:C M AX(H(x) Hy (x))(1.3)The theorem formally states that for a source entropy of H, “if H C there exists acoding system such that the output of the source can be transmitted over the channelwith an arbitrarily small frequency of errors (or an arbitrarily small equivocation). IfH C it is possible to encode the source so that the equivocation is less than H C where is arbitrarily small. There is no method of encoding which gives an equivocationless than H C” [4].This Shannon’s information theory and his theorems have large impacts on themodern communication world:1. They suggest a methodology to quantize a piece of information;2. They describe the correlation between the uncertainty of information and itstransmission speed;3. They indicate the maximum transmission rate for a noisy channel at a certainnoise level.

61. IntroductionNowadays, Shannon’s limit becomes the ideal objective for most designers of the communication systems. Coding techniques are essential for a communication system toachieve such target.1.1.2 Coding theoryCoding theory is more practical compared with other theories in the information theorydomain. It is primarily concerned with finding the methods, called codes, for increasingthe efficiency and accuracy of data communication over a noisy channel as close tothe theoretical limit that Shannon proved as possible. These codes can be mainlysubdivided into source coding (Entropy encoding) and

A.2 Matlab program for comparing union bounds of different code rates . . 155 A.3 Matlab program for comparing union bounds of different constraint lengths157 A.4 Matlab program for comparing union bounds of hard and soft decision 159