Digital amplifier modelling is an application of digital signal processing to recreate the sound of plugging an electric guitar into a traditional physical amplifier. Physical amplifiers can be impractical and unreliable for constant live use by professional musicians, and digital amplifiers have been developed to solve these problems. This paper explores my process of designing a digital guitar amplifier as well as an analysis of the final sound output.

While neural networks are effective in classifying objects from highly structured data, their efficacy on unstructured point clouds has been more limited. In particular, their potential in classifying objects with subtle differences, such as body posture, has yet to be explored beyond simple gestures.Body language is an important type of communication as it enables people to ”speak” through their behaviors. However, the capability to understand body language varies under neu- rodivergent populations. From identifying common body poses, we could assist neurodivergent individuals in recognizing social cues from others or adjusting their own.

The aim of this thesis is to explore how deep learning can understand various poses in non-verbal communication and determine its potential in body language assistance. We test the ability of a deep neural network to classify poses from point cloud distributions recorded from a LiDAR sensor. Implementing dual-dimension blocks to the network improved performance by an average of 25% relative to the baseline provided by the original model, while adding hand-crafted features on top of that led to a 2-3% increase in accuracy. Alongside the overall improvements with dual-dimension annotations, the proposed network leads to improvements across non-background classes by an average of 2%. Based on our results and adjustments, we show how our network can identify body language based on the pose’s gesture and direction.

Augmented reality (AR), virtual reality (VR) and immersive video, as emerging types of multimedia, have recently gained more and more attention. With the development of devices that can capture 3D content and a rapid increase of related applications, the delivery and storage of 3D content have become an important research area. MPEG hosted a Call for Proposals to collect ideas to efficiently compress 3D content in three categories: static point clouds, dynamic point cloud sequences and dynamic acquisition. Among the proposals, Video-based Point Cloud Compression (V-PCC) achieves the highest quality for the second category, dynamic point cloud sequences, under a bit rate constraint.

However, the V-PCC framework is not spatially scalable. In this research, interpolation components are proposed for the V-PCC framework to make it suitable for flexible spatial resolution. As outliers might be brought in by the interpolation, a patch-aware averaging filter is applied to eliminate most outliers. Experimental results show that the interpolation component performs well both on objective evaluation and subjective visual quality.

Point cloud and mesh are the two main representations for 3D content. While compression methods for them are actively studied, there are few studies of their perceptual compression quality and none that consider observation distance. We studied the perceptual quality of compressed 3D sequences, for both a point cloud compression method (V-PCC) and a mesh-based compression method (Triangle FAN (TFAN)). Two main factors that could impact perceptual quality are considered, bit rate and observation distance. Evaluation of perceptual quality is carried out both by collecting viewer opinion scores of the compressed sequences separately, and with a side-by-side comparison. A functional model for mesh and point cloud compression quality is estimated to predict Mean Opinion Score (MOS) which yields high Pearson correlation and rank correlation scores with measured MOS.

Although V-PCC achieves the highest quality among methods proposed to MPEG, outliers and various other artifacts can degrade the V-PCC quality especially when high quantization parameter (QP) is set. After examining the causes and types of V-PCC artifacts that occur, we propose a framework to remove the highly noticeable outlier and crack artifacts caused by V-PCC so as to improve compressed point cloud visual quality. A subjective experiment showed that our approach significantly improves visual quality, and the improvement becomes more obvious with increasing QP values.

Hybrid automatic repeat request (HARQ) \cite{costello1983error} plays an important role in providing reliable and efficient data transmission. In wireless communications, the wireless channel may vary fast, due to the mobility of the transmitter/receiver and the channel. Forward error correction (FEC) and automatic repeat request (ARQ) are two basic techniques to control errors. FEC employs error correction coding, by adding parity bits to the information bits, to combat channel errors. ARQ allows the receiver to request a retransmission of the packet when an error is detected in the received packet. HARQ gives protection to the wireless transmission by combining FEC and ARQ. In typical HARQ systems, redundancy is added to the information bits, and a retransmission is performed until either the packet is successfully decoded, or a maximum number of transmissions is reached.

The motivation to optimize the energy consumption of HARQ is the high energy consumption of wireless communications on mobile devices. Wireless devices usually have a limited battery life, and wireless communications consume the majority of the battery energy of mobile devices. One example is that 3G and Wifi units consume more than 50\% of the energy for some smart phones \cite{tawalbeh2016studying}. Another example is that battery depletion has been identified as one of the primary factors that limit the lifetime of wireless sensor networks \cite{verdone2010wireless}.

Previous works on HARQ mainly use information-theoretic approach, which assumes that the number of bits in each transmission round is sufficiently large. This assumption does not necessarily hold for actual codes with finite length. Therefore, in this dissertation, we consider HARQ with actual codes. We use turbo-coded HARQ, since turbo codes are well-known capacity-approaching codes \cite{berrou1993near} and widely used in standards such as 3GPP Long-Term Evolution (LTE) \cite{3gpp2007mulltiplexing}. We study the energy optimization for HARQ in two scenarios: the energy optimization for incremental redundancy (IR) HARQ, and the energy optimization for HARQ in wireless video transmission. For IR HARQ, each retransmission contains additional parity bits beyond those of the previous transmissions. For the first scenario, we consider different cases of channel state information (CSI) at the transmitter: the transmitter has no knowledge of any CSI, or knows the CSI in previous transmission rounds through a perfect feedback channel, or knows both current and previous CSI. The transmitter decides the forward error correction code rate based on the CSI it has. We minimize the energy consumption of turbo-coded HARQ, subject to a packet loss rate constraint. Numerical results show that the energy consumption of HARQ decreases when more CSI information is available at the transmitter. We also compare IR combining with both Chase combining and the system without combining, and IR combining yields the least energy consumption.

For the second scenario, we formulate the problem as maximizing the video quality, subject to a constraint on the wireless transmission energy consumption. We consider multiple parameters in multiple layers in a wireless video transmission system: transmit power, alphabet size, FEC code rate, maximum number of transmissions and unequal video data importance. An analytical framework is proposed to include these parameters, which allows us to divide this problem into two sub-problems: data transmission and unequal error protection (UEP) for video content. The problem is tackled by solving the two sub-problems, which are done by exhaustive search and convex optimization, respectively. Simulations of different videos show that the proposed scheme outperforms methods using conventional data transmission and/or unequal error protection. For example, in the low SNR region, there is a total gain of 4.8 to 5.6dB on the peak signal-to-noise ratio of the received video compared to video transmission using conventional HARQ without any video UEP.

Images degraded by light scattering and absorption such as hazy, sandstorm, and underwater images often suffer from color distortion and low contrast because of light traveling through turbid media. This can prevent systems that operate outdoors in different lighting conditions from functioning properly, for example, video surveillance systems, autopilot systems and intelligent transportation systems, which include automatic license plate recognition, automatic traffic counting, etc. Therefore, it is desirable to develop an effective method to restore color and enhance contrast for these images. This thesis presents novel work to advance research on image restoration and enhancement for such images.

To enhance or restore such a degraded image, the image formation model is often used to describe it as a ``clear" image blended with an ambient light based on the scene transmission computed using the scene depth from the camera. The transmission describes the portion of the scene radiance which is not scattered or absorbed and which reaches the camera. By reversing the image formation process, one can attain the scene radiance from a degraded image, which is a ``clear" image. However, it involves solving an ill-posed and under-constrained problem because we need to estimate both the ambient light and scene transmission from a single degraded image.

To attack this problem, we proposed to use image blurriness to estimate ambient light and scene depth for underwater images. Furthermore, we extended it by combining light absorption and blurriness to estimate scene depth for underwater scenes in different lighting conditions and color tones. For any images degraded by light scattering and absorption, not limited to underwater ones, we proposed a generalization of the common dark channel prior approach for ambient light and transmission estimation. Additionally, adaptive color correction is incorporated into the image formation model for removing color casts while restoring contrast. Based on the experimental results, our proposed algorithms outperform, both subjectively and objectively, other state-of-the-art algorithms based on the image formation model.

The huge amount of data embodied in a video signal is by far the biggest burden on existing wireless communication systems. Adopting an efficient video transmission strategy is thus crucial in order to deliver video data at the lowest bit rate and the highest quality possible. Unequal error protection (UEP) is a powerful tool in this regard, whose ultimate goal is to wisely provide a stronger protection for the more important data, and a weaker protection for the less important data carried by a video signal. The use of efficient video delivery techniques becomes more important when 3D video content is transmitted over a wireless channel, since it contains twice as much data as 2D video. In this dissertation, we consider the UEP problem for transmission of 3D video over wireless channels. The proposed UEP techniques entail relatively high computational complexity which lend themselves to be more suitable for video-on-demand delivery, where the time-consuming computations are done offline at the transmitter/encoder side.

To adopt UEP for 3D video, we consider a general problem of joint source-channel coding (JSCC). Solving the JSCC problem yields the optimum amount of 3D video compression as well as the optimum FEC (forward error correction) code rates exploited for UEP. We first need to estimate the perceived quality of the reconstructed video at the receiver. The lack of a good objective metric for 3D video makes adopting UEP a more challenging and problematic task compared to 2D video. Fortunately, for 3D video, some quality thresholds are derived in the literature based on the PSNR (peak-signal-to-noise-ratio) metric through experimental tests. These thresholds allow us to formulate the JSCC optimization problem using the PSNR in a straightforward but different way from the typical counterpart optimization problems in the literature. More precisely, we put the constraints of the optimization problem on the quality of the reconstructed 3D video and set our goal to minimize the total bit rate. We adopt the multiview coding (MVC) extension of the H.264/AVC. We also propose a scalable variant of MVC and formulate and solve the JSCC optimization problem for it. We show that significant gains are obtained if the proposed UEP scheme is combined with asymmetric coding.

We also tackle the UEP problem for the video plus depth (V+D) format. We employ the SSIM (Structural SIMilarity) metric for designing UEP for V+D, since it has been shown that PSNR does not properly characterize the perceived quality of a 3D video represented in V+D format. Moreover, the synthesized right view always shows a huge PSNR loss (even in the absence of compression), which does not even allow us to use the asymmetric coding PSNR thresholds. This motivated us to adopt the classical JSCC problem formulation, where our goal is to maximize the quality of the reconstructed left and right views, given that there is a constraint on the sum of the number of source bits and the number of FEC bits. We show that UEP provides significant gains compared to equal error protection. We also derive several interesting results; some of them are in accordance with what have already been published in the literature and some of them are not. We show that the reason for this inconsistency is that we are solving the UEP problem in a more general situation, which yields novel solutions.

Lastly, we focus on UEP for video broadcasting over wireless channels. Our goal here is to design a UEP-based video broadcasting system that well serves all the users within the service area of a base station. In a service area, there exist heterogeneous users with different display resolutions operating at different bit rates. Spatially scalable video is an excellent video compression format for this scenario, since it allows a user to decode that portion of the scalable bit stream that fits its operating bit rate as well as its display resolution. We tackle this problem for a MIMO (multi-input-multi-output) channel which enables us to exploit either spatial diversity or spatial multiplexing in a multipath fading channel to increase channel reliability or throughput, respectively. We employ spatial diversity techniques, in particular the Alamouti code, to encode the base layer. We also adopt spatial multiplexing techniques, in particular the V-BLAST, to encode the enhancement layer. By controlling the power allocation between the base layer and the enhancement layer, we can control the level of protection we provide to each of them. We also show that the adoption of scalable video in our system yields much higher gains compared to non-scalable video.

The perceptual quality of videos has attracted much attention, because it is hard to estimate by objective metrics, and because it is affected by many conditions. Compression, transmission and viewing conditions all impact the perceptual quality. In this dissertation, we aim to preserve and enhance the perceptual quality in different cases.

The impact of ambient illumination on the perceptual quality of traditional 8-bit 2D video is first studied. Some details, especially those in dark areas of videos, are invisible in bright ambient light, because of the reflection of ambient light and the reduction of the sensitivity of human eyes. We analyze the display characteristics and human visual sensitivity, and propose methods to enhance the contrast and details without increasing the peak brightness of the display.

Another viewing condition, viewing distance, is also investigated in this dissertation. A display device held farther away may have fewer details visible compared to a device held closer. The unnoticeable details can be filtered before compression, which can reduce the bit-rate of the video. A subjective test was conducted to demonstrate the bit-rate saving without degrading the perceptual quality.

Besides the traditional 8-bit videos, a new form of video, high dynamic range (HDR) videos, is studied in Chapter 4 of this dissertation. There can be banding artifacts in the inverse tone mapped HDR videos which degrade the perceptual quality, though the impact on the objective quality is subtle. An enhancement filter is proposed to remove the banding artifacts and reduce compression artifacts, and at the same time, preserve true edges and details. The parameters of the filter are determined by minimizing the proposed perceptual distortion metric.

The perceptual quality of 3D video is explored in Chapter 5. In particular, the stereoscopic 3D video of the 2D+depth format is studied. Transmission of such videos through networks can be affected by packet losses. The importance of packets is investigated by a human observer experiment. A prediction model of the importance is developed using features such as the video type, frame type, and spatial location of the packets.