A set of new neuron model and neural network architectures are introduced for the exploration of spiking neural networks to classify handwritten digits and images. Brain-like neural network powered artificial intelligence has become a driving force behind everyday applications from autonomous vehicles, facial recognition applications, to business analytics and energy market prediction. Biological brain-inspired Deep Neural Networks (DNN) have been around for a while. However, spiking Neural Networks (SNNs) have come to the spotlight due to higher energy efficiency and brain-like computing power. In this thesis, various models of spiking neurons such as Leaky Integrate-and-Fire (LIF), and Memristive Integrate-and-Fire (MIF) are examined and explored for applications of spiking neural networks. It has become apparent that large-scale neural network models will not only be in data-centers but also in edge computing devices and embedded systems, thus more energy-efficient and faster neural network types are needed.