Approximate computing enables efficient trade-off among accuracy,area, latency and power for more efficient error tolerant applications implementation such as machine learning and multimedia workloads. Those workloads are heavily dominated by the arithmetic operations and hence designs of hardware-efficient approximate arithmetic units have been intensively investigated recently. However, most of the existing works lack the systematic configurability for accuracy vs. area/power/latency trade-off. This thesis merges several works into designs of fast approximate arithmetic units with improved accuracy and hardware efficiency. The first work proposes a new counter-based stochastic-behaving approximate integer unsigned multiplier and scaling method with improved counting efficiency for many emerging error tolerant application workloads such as deep neural networks, mitigating a long-standing issue of the stochastic computing that small inputs usually lead to large error. The second work proposes a novel counting-based SC divider with accelerated counting process and improved accuracy to easily interface with the existing stochastic or binary logic. The third work focuses on improving the accuracy and the hardware efficiency of the conventional approximate logarithmic multiplier at the same time with a novel error compensation scheme. Last but not least, the forth work mitigates an important problem that approximated designs might lead to unwanted higher temperature and related reliability issues due to the increased power density and proposes a power density aware approximate logarithmic multiplier design which can reduce the power density of the original approximate logarithmic multiplier design with no accuracy loss.