Deep Neural Network representations correlate very well with neural responses measured in primates' brains and with psychological representations of human similarity judgement tasks, making them possible models for human behavior-related tasks. This study investigates whether DNNs can learn an implicit association (between colors and emotions) for images. An experiment was conducted in which subjects were asked to select a color for a given emotion-inducing image. These human responses (decision probabilities) were modeled on neural networks using representations extracted from pre-trained DNNs for the images and colors (a square of the color). The model presented showed a fuzzy linear relationship with the decision probabilities. Finally, this model was presented as a model for emotion classification tasks, specifically with very few training examples, showing an improvement in accuracy from a standard classification model. This analysis can be of relevance to psychologists studying these associations and AI researchers modelling emotional intelligence in machines.