Successful motor interaction with a target changes memory of the target’s size, which seems larger if the action was successful than if it was unsuccessful. This has been attributed to the effect of action on subsequent perception or memory. We asked what the action provides: Is feedback from the action necessary, or only the information provided by the action? We found that perceived difficulty alone changes the remembered goal characteristics, without changes in the stimuli, and before the motor task is executed. We gave observers a marble and showed them a hole in a box. They were told that throwing the marble into the hole was either difficult or easy, depending on the condition. The hole was then covered and its size judged. Participants who were told that the task was difficult judged the hole to be significantly smaller than it was, whereas those told that the task was easy made judgements not significantly different from veridical. When observers subsequently threw the marble, their success rates were independent of their own estimates of hole size or of what they had been told about the difficulty of the task, showing that their size estimates affected memory but not action. In a second experiment, we found that the effect disappeared if the hole was visible during the size estimation.