Decreased Key Pecking in Response to Reward Uncertainty Followed by Surprising Delay Extension in Pigeons

The Pavlovian autoshaping paradigm has often been used to assess the behavioral effects of reward omission on behavior. We trained pigeons to receive a food reward (unconditioned stimulus or UCS) following illumination of a response key (conditioned stimulus or CS). In Experiment 1, one group of pigeons was trained with two 100% predictive CS-UCS associations (reward certainty) and another group with two 25% predictive CS-UCS associations (reward uncertainty) for 12 sessions. In both groups, the two CS durations were 8 s. Then, in each group, the duration of one CS remained unchanged and that of the other CS was suddenly extended from 8 to 24 s for 6 sessions. In Experiment 2, some experienced individuals (from Experiment 1) and naïve individuals formed two groups trained with a 24-s CS throughout for 18 sessions. Our results show that pigeons (a) pecked less at the uncertain than the certain CS, (b) decreased and then increased CS-pecking after extending CS duration, especially in the certainty condition, (c) were unresponsive to the 24-s CS in the absence of previous experience, and (d) decreased their response rate close to the end of a trial irrespective of the reinforcement condition, CS duration, and amount of training. These results are discussed in relation to several theoretical frameworks.

Testing the behavioral effects of reward omission can take several forms. One of them consists of repeatedly exposing organisms to reward uncertainty in Pavlovian autoshaping, a procedure in which the brief presentation of a conditioned stimulus (CS; lever insertion in rats or key illumination in pigeons) randomly predicts the delivery of food or nothing. In rats, reward uncertainty typically enhances sign-tracking (CSdirected) responses in comparison with reward certainty, where each CS presentation is food rewarded (Anselme et al., 2013;Glueck et al., 2018;Hellberg et al., 2018;Robinson et al., 2014;Torres et al., 2016). Under conditions of reward certainty, sign-tracking is both a product of associative learning (the rat has to learn that the CS predicts food) and incentive salience, a dopamine-dependent brain process that makes the CS motivationally attractive (i.e., approached, sniffed, nibbled, and pressed;Berridge, 2007). That the higher response rates to the CS under reward uncertainty cannot convincingly be explained in terms of learning or incentive salience (Anselme, 2015(Anselme, , 2018, is traditionally assumed to result from the experience of frustration, owing to the occasional nonrewarded CS presentations (Amsel, 1992), and has more recently been viewed as a product of incentive hope induced by the possible reliability of the CS on each ongoing trial (Anselme, 2015(Anselme, , 2018. Pigeons may also show an increase in sign-tracking by means of the same method as used with rats (Gottlieb, 2004;Perkins et al., 1975), although this effect might be less robust (Anselme, unpublished data;Papini & Overmier, 1984. Here, we used a slightly different procedure involving two successive cues. Serial compound CSs have extensively been used in the literature on conditioning (e.g., Collins et al., 1983;Cook et al., 1992;Kaplan & Hearst, 1982;Kehoe, 1982;Kehoe et al., 1987;Lamb & Riley, 1981;Pavlov, 1927). In our study, the succession of two cues was similar (but not equivalent) to the procedure used by Pearce and colleagues in the 1980's (e.g., Collins et al., 1983;Pearce et al., 1985), which was revealed to be effective in enhancing response rates in pigeons exposed to an unreliable cue versus a reliable cue. The first and the second cues are called initial-and terminal-link stimuli or CSs, respectively. The pigeons were not exposed to a choice between two options; each condition was presented to a different group of individuals. Specifically, in Group Certainty, the initial-link CS was reliable because it was systematically followed by a reliable terminal-link CS that was 100% predictive of food, and, in Group Uncertainty, the initial-link CS was unreliable because it was randomly followed by a reliable terminal-link CS that was 100% predictive of food or by a reliable terminal-link CS that was 100% predictive of no food.
Besides reward uncertainty, another traditional approach to reward omission is the consummatory successive negative contrast (cSNC) procedure, in which organisms are repeatedly exposed to a given reward concentration (e.g., a 32% sucrose solution), which is suddenly reduced to a noticeably lower nonzero value (4% sucrose). Rats that experienced such a violation of their initial expectation are said to be frustrated and disregard the downshifted reward during the first postshift session (Flaherty, 1996;Pellegrini et al., 2004;Ruiz-Salas et al., 2020). During the next sessions with the downshifted reward, there was a gradual recovery of consummatory behavior near the response rate observed in the last preshift sessions. In pigeons, the attempts to generate cSNC effects by means of traditional methods have all failed, suggesting that pigeons are either insensitive to frustration or, at least, insensitive to the experimental conditions that cause frustration in rats (Papini et al., 2019). One possibility to test frustration in pigeons is to use a modified cSNC procedure that could make them more responsive. Here, the idea is to reduce reward quality by means of a sudden extension of the CS duration instead of a sudden downshift in reward concentration or amount. Indeed, pigeons find delayed rewards strongly aversive, showing a steeper temporal discounting curve than rats and humans (Green et al., 2004;Mazur & Biondi, 2009;Tobin & Logue, 1994). Temporal discounting is not a measure of frustration per se, but a surprising delay extension is bad news that could facilitate the occurrence of frustration in pigeons. To test this, we analyzed the number of key-pecking responses that occurred after shifting the duration of an initial-link CS presentation from 8 s to 24 s for several sessions. Another indicator of frustration might be revealed through a second-by-second analysis of responding within a trial, as the anticipation of frustration near the end of a trial is assumed to inhibit approach behavior among uncertain, but not certain, individuals due to a possible absence of reward delivery (e.g., Amsel et al., 1964;Chen et al., 1980).
The key-pecking patterns observed in this study can inform us about the psychological processes involved in pigeon autoshaping. First, at training, higher response rates to an unreliable initial-link CS (uncertainty) compared with a reliable initial-link CS (certainty) would be consistent with the incentive hope hypothesis and frustration theory (Amsel, 1992;Anselme, 2015). In contrast, lower response rates to uncertainty as opposed to certainty would be compatible with the incentive salience hypothesis because inconsistent CS-UCS pairings should reduce the motivational value that can be transferred from the food UCS to the CS. Second, extending CS duration from 8 s to 24 s might decrease pecking rates as an effect of delay aversion, in which case this adjustment of behavior would simply result from a reduction in the attribution of incentive salience to the CS. Now, if as in the cSNC procedure in rats, that delay extension caused an abrupt decrease in performance followed by a gradual recovery period, then this might be the signature of frustration in pigeons. This effect of frustration should mainly be observed in individuals trained under reward certainty because reward expectation is stronger than under uncertainty training (Pellegrini et al., 2004). Finally, a within-trial decrease in pecking rates near the end of a trial could be a consequence of anticipatory frustration if it is only observed for the extended delay in the uncertain pigeons, especially late in training -because the learning of that anticipation may take time to express in behavior (Fuentes-Verdugo et al., 2021). In contrast, such a decrease could denote a mere expectation of reward timing if its occurrence is independent of reinforcement condition, CS duration, and the amount of training. Other psychological processes, such as stimulus generalization, perseveration following stimulus change, extinction, reacquisition, and the orienting response, represent alternative theoretical frameworks in case of an absence of evidence for frustration in the present study.

Animals and Housing Conditions
Sixteen unsexed homing pigeons (Columba livia) obtained from local breeders were used in this study (n = 8 per group). All birds were individually housed in wire-mesh home cages (40 cm × 40 cm × 45 cm) under a 12h/12h light/dark cycle (lights on at 7:00 am). They were previously involved in unrelated experiments, with half of them tested in a mirror self-recognition task, and the other half in a visual recognition task. Pigeons were maintained at approximately 85-90% of their free-feeding body weight for the duration of the experiment (range: 383-488 g) and had ad libitum access to water in their home cage. All procedures followed the German guidelines for the care and use of animals in science and were in accordance with the European Communities Council Directive 86/609/EEC concerning the care and use of animals for experimentation. The methods were also approved by our university.

Apparatus
The pigeons were tested in individual operant chambers (34 cm width × 34 cm depth × 32 cm height) equipped with a white house light. In the middle of the front panel, one transparent pecking key (4 cm × 4 cm), coupled with an electric switch, allowed the animal to respond to the CS displayed on an LCD flat-screen located behind the panel. A rotating food hopper located below the key, approximately 5 cm above the floor level, provided access to food pellets after the presentation of a rewarded CS. The house light remained on during the intertrial interval (ITI) and the food port light was on with each food delivery. Pigeon activity could be monitored with a camera fixed on the back panel of the Skinner box, but video data were not recorded. A custom-written MATLAB R2019b code using the Biopsychology Toolbox was used to control the apparatus (Rose et al., 2008).

Procedure
The individuals inexperienced with the presentation of CSs in a Skinner box received one daily session of Pavlovian autoshaping -100 trials with a 20-80 s ITI -for several consecutive days in order to learn to peck at the CS presented on a response key. Following a 10-s CS presentation, a single food pellet was automatically delivered from the food hopper. The autoshaping training continued until each pigeon reached at least 85% of responses to the trials. At this stage, all the pigeons were able to peck at a CS.
Pretraining. The pigeons were divided in two groups referred to as Group Certainty and Group Uncertainty -although their group assignment was random, each group was composed of 4 pigeons from the previous mirror self-recognition experiment and 4 pigeons from the previous visual recognition experiment. They were pretrained with one daily session for 12 days in order to learn the association between several CSs and food delivery. During the first 8 days, the two groups were pretrained with their respective terminal-link stimuli. In Group Certainty, one session consisted of 8 trials interspersed with a 20-80 s variable ITI. Each trial started with an ITI, and a yellow stimulus was then presented for 2 s and was followed by the delivery of 4 food pellets. In Group Uncertainty, the pigeons received the same pretraining with the yellow stimulus and also received 8 trials with a 2-s grey stimulus never followed by food. Presentations of the grey and yellow stimuli were randomized. However, their association with food was not counterbalanced because we were not interested in the response to these stimuli. Because the sessions were short, the pigeons received 2 sessions per day. During the last 4 days, the terminal-link stimuli and their respective association with food delivery were pretrained as before, but, in addition, these stimuli were randomly intermixed with the 8-s presentation of nonrewarded brown and blue initial-link stimuli eight times each. As the sessions were longer here, the pigeons received only one per day.
Training. Both groups received one daily session (32 trials) for 12 days, in which the initial-link and terminal-link stimuli were presented in succession. In Group Certainty, a brown or blue initial-link CS (16 times each in a session) was randomly presented for 8 s on the response key. Immediately after this, a yellow terminal-link CS was displayed for 2 s with a 100% probability and was followed by 4 food pellets with a 100% probability (Figure 1, left). In Group Uncertainty, a brown or blue initial-link CS (16 times each in a session) was also randomly presented for 8 s on the response key. Immediately after this, a yellow terminal-link CS was displayed for 2 s with a 25% probability and was followed by 4 food pellets with a 100% probability, or a grey terminal-link CS was displayed for 2 s with a 75% probability and was followed by food with a 0% probability (Figure 1, right).

Figure 1
Procedure (Pretraining, Training, and Test) for Experiment 1 Note. Left: Group Certainty. Right: Group Uncertainty. Two stimuli were presented in succession; the first one (8-s or 24-s duration) corresponded to the initial-link CS and the second one (always 2-s duration) to the terminal-link CS. All the pigeons of each group (n = 8) were exposed to the same pretraining and training conditions. At test, the CS duration was counterbalanced for color within each group (n = 4). Food delivery is depicted by a pigeon above a food cup.

Test.
In both groups, one of the initial-link CSs (e.g., brown) continued to be presented for 8 s, as in the training phase. However, the other initial-link CS (blue) was now suddenly extended from 8 s to 24 s. The two stimuli were counterbalanced for stimulus duration. Apart from this, the trials remained the same as in the training phase. For each pigeon, the extended stimulus was that for which the response rate was the most elevated on Training Session 12. When no great difference in response rates was shown between the two stimuli, the extended stimulus was chosen to satisfy counterbalancing for color.

Statistical Analyses
Pecks at the key were registered for the entire session and also in 1-s bins during the 8 s and the 24 s of key illumination for within-trial assessment of performance. To analyze the mid-trial stability of performance, key pecking was compared between Bins 3-4 and Bins 5-6 on Sessions 2 (early), 7 (middle), and 12 (late) at training and on Sessions 1 and 4 at test for the 8-s CS. Similar comparisons were also carried out between Bins 3-12 and 13-22 on Sessions 1 and 4 at test for the 24-s CS. The first 2 s of a trial (Bins 1-2) were removed from those analyses to avoid any bias related to initial overstimulation produced by key illumination. Responding at the end of a trial (few last bins) was also analyzed to determine how it changed relative to the previous bins. The key-pecking value at a bin corresponded to the averaged sum of responses of each pigeon to that bin within a session. To make the response rates to the 8-s and the 24-s CSs comparable in a session, the mean values for the long CS were divided by three. All statistical analyses involved the computation of mixed ANOVAs Group × Session or Group × Bin, and planned comparisons were carried out from the main ANOVAs. Repeated measures ANOVAs and t-tests were used as appropriate. The results were expressed as means and standard errors, and the significance level was accepted at p < 0.05. For some session or bin comparisons, a Bonferroni correction was used as a posthoc test for multiple comparisons.

Pretraining Phase
Overall, the pigeons showed differential sensitivity to the color of the response key, F(2, 28) = 15.37, p < 0.001, ηp 2 = 0.52, responded similarly in both groups, F(1, 14) = 0.19, p = 0.668, ηp 2 = 0.01, and the Group × Color interaction was not significant, F(2, 28) = 0.36, p = 0.701, ηp 2 = 0.02. A post-hoc pairwise comparison revealed that the pigeons pecked significantly more at the rewarded (yellow) CS than at the nonrewarded (blue and brown) CSs (yellow vs. blue: pcorr = 0.004; yellow vs. brown: pcorr = 0.006). In contrast, the pecking rates were similar between the two nonrewarded CSs (brown vs. blue: p corr = 0.960). The nonrewarded grey CS was not included in those analyses because it was only present for the pigeons from Group Uncertainty, and a paired-samples t-test indicated that responding to the grey CS did not significantly differ from that to the other nonrewarded CSs [grey vs. blue: t(7) = 0.17, p = 0.867; grey vs. brown: t(7) = 1.25, p = 0.252]. Thus, the pretraining results suggested that the two groups learned the appropriate associations and could start the next phase without behavioral biases.

Training Phase: Reward Uncertainty
The training phase aimed to determine how pigeons react to an initial-link CS predictive of 25% (uncertainty) vs. 100% (certainty) chance of reward under conditions known to generate no preference when presented simultaneously. The dynamics of responding within a trial was also examined in both groups. As the pigeons responded similarly to both initial-link stimuli (blue and brown), the stimuli were statistically merged and treated as if they were a single stimulus. Figure 2A represents the mean number of pecks in Groups Certainty and Uncertainty over the course of 12 acquisition sessions. We found significant overall effects of group, F(1, 14) = 5.07, p = 0.041, ηp 2 = 0.26, and of session, F(11, 154) = 20.97, p < 0.001, ηp 2 = 0.60, as well as a Group × Session interaction, F(11, 154) = 3.23, p = 0.028, ηp 2 = 0.19. A post-hoc pairwise comparison showed that the performance was similar in both groups on Session 1, pcorr = 0.421, and the pigeons learned the task in both groups. However, the effect of training on performance occurred faster in the certain pigeons (Sessions 1 vs. 2: pcorr = 0.024) than in the uncertain pigeons (Sessions 1 vs. 10: pcorr = 0.018). Responses in both groups showed significant differences on Sessions 2, 3, 4, 5, 6, 7, and 8, pcorrs ≤ 0.032, with the performance of the uncertain pigeons being consistently lower than that of the certain pigeons.
The dynamics of responses on each session was very similar in Group Certainty ( Figure 2B) and typically consisted of an initial burst of responses (Bin 1) after illumination of the key, followed by a stable response rate (Bins 2-6 or 2-7) and then a decrease in response rate close to the end of the trial (Bins 6-8 or 7-8). Concretely, a post-hoc comparison showed that all sessions contained a systematic significant decrease in responding from Bin 1 to Bin 2, pcorrs ≤ 0.05, in Group Certainty. This decrease was followed by a stable response rate until Bin 6 or Bin 7, pcorrs ≥ 0.085. At the end of the trial, the response rates dropped to a very low value from Bin 6-7 to Bin 8, pcorrs ≤ 0.015. In Group Uncertainty, the pigeons did not significantly decrease their responses from Bin 1 to Bin 2, except on Session 7, pcorr = 0.025). Then, their response rates were similar until Bins 6 or 7 and significantly decreased at the end of the trial on Sessions 7 and 12, pcorrs ≤ 0.028. The response rates remained low and stable on Session 2, likely because the learning of the task was very low at this stage.

Figure 2
Training Phase of Experiment 1 Note. (A) Key-pecking responses of Group Certaityn and Group Uncertainty pigeons across the 12 training sessions. (B) Second-bysecond detail of their responses on Sessions 2 (early), 7 (middle), and 12 (late). Error bars indicate standard error of the means. Legends: *p < 0.05, **p < 0.01.

Test Phase: Delay Extension
In the test phase, the duration of one of the initial-link CSs (blue or brown) was suddenly extended from 8 s to 24 s, while that of the other remained unchanged (8 s). Figure 3A depicts the mean number of pecks to the 8-s and 24-s CSs -divided by three to be compared to the 8-s CS -over the 6 test sessions. To begin with, the 8-s CS and the three 8-s "slices" (1-8, 9-16, and 17-24) of the 24-s CS on Session 1 of the test phase were compared with the 8-s CS on Session 12 of the training phase (not shown). This aimed to determine whether the sudden occurrence of a long CS presentation altered performance in some ways. None of these comparisons were significant in any group, pcorrs ≥ 0.940, indicating that pecking remained stable immediately after lengthening one of the two CSs.
Nevertheless, given that the pigeons were previously trained with 8-s CSs and showed a strong decrease in performance at 8 s, we should observe a trace of this anticipation at 8 s of Test Session 1 for the extended stimulus. We tried to identify a within-session depression in peck number during the first five trials as opposed to the last five trials at 8 s of Test Session 1 for the extended stimulus, in case pigeons would have quickly relearned the new condition. The results were nonsignificant both in Group Certainty [13.875  1.315 vs. 9.500  2.521; t(7) = 2.00, p = 0.085] and in Group Uncertainty [13.250  2.519 vs. 13.750  2.795; t(7) = 0.76, p = 0.470].
With respect to the 24-s CS, only the effect of the session was significant, F(5, 70) = 21.34, p < 0.001, ηp 2 = 0.60. In Group Certainty, there was a significant decrease in pecking rates between Sessions 1 and 4, pcorr = 0.003, followed by a slight, though nonsignificant, increase between Sessions 4 and 6. In contrast to the 8-s CS, performance remained lower on Session 6 than on Session 1, possibly because more sessions would have been necessary to reach the initial pecking rate. In Group Uncertainty, performance decreased significantly between Sessions 1 and 4, pcorr = 0.035, but did not increase between Sessions 4 and 6. The sudden occurrence of the 24-s CS was more effective for the Group Certainty than the Group Uncertainty pigeons because responding to the 8-s and the 24-s conditions was significantly different across sessions in Group Certainty, t(47) = -5.58, p < 0.001, but was not in Group Uncertainty, t(47) = 0.33, p = 0.746.  Figure 3B shows the mean number of pecks on the 8-s CS in each 1-s bin of its presentation. We represented the within-trial dynamics of responses on Session 1 (first occurrence of the 24-s extension) and Session 4 (maximal effect of the 24-s extension) of the test phase. Our objective was to determine whether the distribution of responses was different between these two sessions and also different in general configuration relative to the sessions analyzed in the training phase. An overall comparison between the two groups revealed only a bin effect on both Sessions 1 and 4, Fs(7, 98) ≥ 21.53, ps < 0.001, ηp 2 ≥ 0.61. In Group Certainty, there was an initial significant decrease in pecking rates from Session 1 (Bins 1 vs. 3: pcorr = 0.006). This was followed by a stable performance on Session 1 between Bins 3 and 7 and then by a significant drop at the end of the trial (Bins 7 vs. 8: pcorr < 0.001). On Session 4, the performance remained similar between the first 7 bins and significantly decreased at the end of the trial (Bins 7 vs. 8: pcorr = 0.049). In Group Uncertainty, the first significant decrease in pecking rate from Session 1 was observed on Bin 5 (Bins 1 vs 5: pcorr = 0.034), with a drop similar to Group Certainty on Bin 8 (Bins 7 vs. 8: pcorr = 0.023). On Session 4, the Group Uncertainty pigeons showed a stable performance until Bin 6, and the performance decreased significantly close to the end of the trial (Bins 6 vs. 8: pcorr = 0.026). Note that the same within-trial analyses were carried out with respect to Session 6. As the dynamics of responses was broadly similar to that found on Sessions 1 and 4, we did not report it in order to avoid unnecessary cumbersome.

Experiment 2
In Experiment 1, we showed that reward certainty stimulated sign-tracking more than reward uncertainty with an 8-s CS presentation (training) and that, after extending the presentation to 24 s, some pigeons decreased sign-tracking between Sessions 1 and 4 and increased it between Sessions 4 and 6 (test). The Group Certainty pigeons did not respond similarly to the Group Uncertainty pigeons across the 6 test sessions. Only the Group Certainty pigeons exhibited this pattern of responses independent of the CS duration, suggesting that they were more disturbed by delay extension than the Group Uncertainty pigeons. Experiment 2 aimed to determine how experienced (from Experiment 1) and naïve pigeons reacted to the presentation of a 24-s CS throughout training -no delay extension here. If a long CS presentation of 24 s is a source of frustration, like with rats (Fuentes-Verdugo et al., 2021), then key pecking should increase in the naïve pigeons relative to certainty training with the 8-s CS (see Figure  2A), owing to their inexperience of frustration. Also, given that the experienced pigeons should have recovered from frustration, their key-pecking performance in this new task should be both stable and similar to that of the certain pigeons in Session 6 of the test phase of Experiment 1. Depending on the results, other theoretical interpretations will be considered.

Method
We used 16 pigeons, out of which 8 were reused from Experiment 1. Half of them belonged to Group Certainty and the other half to Group Uncertainty. They were selected based on their moderate response rates in order to avoid any sampling bias, although variability in responding from pigeon to pigeon was not negligible. These pigeons formed the group of experienced pigeons because of their past experience with a variant of the task. The other 8 pigeons formed the group of naïve pigeons, which had never been trained in any task. They received some preliminary autoshaping sessions, as described in Experiment 1, before pretraining could start. The housing conditions and the apparatus were the same as in Experiment 1. Of note, one of the naïve pigeons became ill during training and was removed from the experiment, reducing the group size to n = 7 in this group. The data initially collected for that individual were not taken into account for analyses.

Procedure
Pretraining. All the pigeons were pretrained as the Group Certainty pigeons in Experiment 1. Other colors were used for the initial-link and the terminal-link stimuli to eliminate possible interference with the previous training/test conditions among the experienced pigeons. During the first 8 days, the pigeons were exposed to an orange terminal-link stimulus for 2 s and were rewarded with 4 food pellets after each stimulus presentation. For the next 4 days, the rewarded orange terminal-link stimulus was randomly intermixed with the non-rewarded 24-s aqua initial-link stimulus.
Training. The pigeons received one daily session (16 trials) for 18 days, in which the initial-link and the terminal-link stimuli were presented in succession. The aqua initial-link CS was shown on the response key for 24 s, and then the orange terminal-link stimulus was displayed for 2 s and was followed by the delivery of 4 food pellets.

Results
The naïve and the experienced pigeons were pretrained with the new stimuli. On Session 1, the experienced pigeons pecked significantly more at the orange stimulus -future terminal-link stimulus -than the naïve pigeons, pcorr = 0.027. Also, the experienced pigeons pecked significantly less at the aqua stimulusfuture initial-link CS -than at the orange stimulus on pretraining Session 12, pcorr < 0.001. Among the naïve pigeons, a t-test for paired samples showed that they pecked less at the aqua than at the orange stimulus during the last 4 pretraining sessions, t(31) = 4.23, p < 0.001. Post-hoc multiple comparisons indicated only a nonsignificant trend on Session 12, pcorr = 0.064.
The key-pecking responses of the naïve and the experienced pigeons over the course of 18 training sessions with a 24-s CS are depicted in Figure 4A. As the group of experienced pigeons contained 4 Group Certainty and 4 Group Uncertainty individuals from Experiment 1, the performance of each subgroup was analyzed (see insert for Figure 4A). Although the experienced Group Uncertainty pigeons responded more than the experienced Group Certainty pigeons on average, no significant effect of group was found, F(1, 6) = 1.24, p = 0.308, ηp 2 = 0.17, and there was no Group × Session interaction, F(17, 102) = 0.60, p = 0.882, ηp 2 = 0.09. However, the effect of session was significant, F(17, 102) = 1.96, p = 0.021, ηp 2 = 0.25. The experienced Group Certainty and Group Uncertainty pigeons did not differ in pecking rates on any session, and there were also no within-subject significant differences between sessions, pcorrs ≥ 0.141. Because of the absence of significant group difference, the two subgroups were merged to form the group of experienced pigeons.
As shown in Figure 4A, the experienced pigeons pecked significantly more than the naïve pigeons at the 24-s CS, F(1, 13) = 6.87, p = 0.021, ηp 2 = 0.34. There were also a significant effect of session and a significant Group × Session interaction [session: F(17, 221) = 1.71, p = 0.043, ηp 2 = 0.12; interaction: F(17, 221) = 1.86, p = 0.023, ηp 2 = 0.12]. Throughout the 18 sessions, performance increased among the experienced pigeons, F(17, 119) = 2.08, p = 0.012, but not among the naïve pigeons, F(17, 102) = 1.23, p = 0.230. A posthoc comparison revealed significant differences on more than half of the sessions, as indicated in Figure 4A, pcorrs ≤ 0.046. Of note, the burst in responding observed on Session 7 in the naïve group was due to only one individual (Pigeon 857) that pecked, for unknown reason, at a high rate on that session. Apart from that, they almost never responded to the 24-s CS in any of the 18 training sessions.
A within-trial analysis was carried out on Session 18, in which the behavioral effects of learning and motivation should be considered maximal in this experiment ( Figure 4B). The experienced pigeons responded significantly more than the naïve pigeons, F(1, 13) = 9.27, p = 0.009, ηp 2 = 0.42. The effect of bin was significant, F(23, 299) = 4.10, p < 0.001, ηp 2 = 0.24, as well as the Group × Bin interaction, F(23, 299) = 3.70, p < 0.001, ηp 2 = 0.22. In the experienced group, there was a significant decrease in key pecking from Bin 1 to Bin 2, F(1, 13) = 101.24, p < 0.001, followed by a period of stability in responding between Bins 3-12 and 13-22, F(1, 13) = 3.65, p = 0.078, and finally a significant decrease in key pecking from Bin 19 to Bin 24, F(1, 13) = 17.02, p = 0.001. Significant differences were obtained for Bins 1 versus 2, 1 versus 3, 1 versus 4, 1 versus 5, 1 versus 22, 1 versus 23, and 1 versus 24, pcorrs ≤ 0.006, whereas the naïve pigeons did not show any interaction with the CS during the 24 s of its presentation, pcorr = 1.000. As shown in Figure 4B, the experienced and naïve pigeons only gave a similar number of pecks on Bin 24, pcorr = 0.114. We compared performance on Session 6 of the test phase of Experiment 1 (naïve and experienced pigeons) with that of each of the 18 sessions of Experiment 2. The naïve pigeons responded significantly less than the pigeons at test on each of the 18 sessions (but Session 7 because of Pigeon 857), as they were almost unresponsive to the 24-s CS per se, pcorrs ≤ 0.001. In contrast, the experienced pigeons responded similarly to the pigeons at test on almost all sessions, pcorrs ≥ 0.051, but three produced higher response rates (Sessions 12, 14, and 18: pcorrs ≤ 0.047).

Discussion
The results from these two experiments can be summarized in four points. First, in the procedure used here, an uncertain CS was associated with lower pecking rates than a certain CS, although the pecking rates of pigeons became similar late in training. Second, a sudden extension of the CS duration caused a gradual decrease and then sometimes a gradual increase in pecking rates, an effect visible in the Group Certainty pigeons only. Third, a CS of long duration presented throughout training made naïve pigeons unresponsive, although experienced pigeons maintained the level of response previously acquired. Fourth, a noticeable decrease in response rates close to the end of a trial was often shown irrespective of reinforcement condition (certainty or uncertainty), of CS duration (8 s or 24 s), and of training amount (early or late). Together, these findings tell us something about the psychological processes that underpin the behavior of pigeons in Pavlovian autoshaping.

Reward Uncertainty Reduces Incentive Salience
The Group Uncertainty pigeons were less prone to peck at the initial-link CSs than the Group Certainty pigeons. The incentive salience hypothesis can explain this result. The food UCS has a motivational value, which is gradually transferred to the CS over repeated CS-UCS pairings (Bindra, 1978) and ends up making the CS attractive as if it was the food UCS itself (Berridge, 2007). Given the inconsistency of the pairings under uncertainty relative to certainty, less motivational value from the food UCS can be transferred to an uncertain as opposed to a certain CS -the former CS producing less responses than the latter. Of note, although a gap seems to exist between the initial-link CS and the food UCS, our design did not use a traditional trace conditioning procedure, in which an unfilled time interval occurs between the CS and the UCS. Each initiallink stimulus was followed by a terminal-link stimulus in all groups tested. While trace conditioning is ineffective in producing a conditioned response, filling the gap between the CS and the UCS is known to boost excitatory conditioning (Kaplan & Hearst, 1982;Rescorla, 1980). This finding contrasts with the well-established evidence that rats increase their response ratesmeasured as lever pressing and lever nibbling -to an uncertain CS relative to a certain CS (Anselme et al., 2013;Glueck et al., 2018;Hellberg et al., 2018;Robinson et al., 2014;Torres et al., 2016), even though this effect sometimes fails to occur (Boughner & Papini, 2008;Russell & Robinson, 2019). The procedure with rats is slightly different from that used here (only one CS is presented), and pigeons have been shown to increase their responses using the rat procedure (Gottlieb, 2004;Perkins et al., 1975). However, the occurrence probability of this effect might be lower in pigeons because their motivational and emotional reactivity appears to be poorer than that of rats in several reward-related tasks (Papini et al., 2019). In particular, here, the low performance under reward uncertainty precludes any interpretation of pecking rates in terms of incentive hope or frustration, two processes that are assumed to increase rather than decrease performance relative to reward certainty (Amsel, 1992;Anselme, 2015).
After 8 training sessions, however, key pecking reached similar levels in the two groups. Further training would have been necessary to see whether this represents asymptotic performance for both the Group Certainty and the Group Uncertainty pigeons or whether a longer training period is required for the Group Uncertainty pigeons to peck at a higher rate than the Group Certainty pigeons. In the former case, similar response rates mean that the Group Uncertainty pigeons have counterconditioned the aversive effects of uncertainty-associated non-rewards and come to respond like the Group Certainty pigeons (Anselme, 2016). The fact that performance follows a similar trajectory, with a small (nonsignificant) decrease in both groups between Sessions 10 and 12, may suggest that an asymptote has been reached. If this interpretation is correct, performance would remain similarly stable with further training, as already observed in pigeons with a simple CS-UCS association (Anselme, unpublished data). In the latter case, the Group Uncertainty pigeons would end up pecking the CS more than the Group Certainty pigeons. Early in training, a lower response rate in animals exposed to an uncertain stimulus has sometimes been reported before their performance overcomes that of animals exposed to a certain stimulus (e.g., Crawford et al., 1985;Gonzalez, 1973;Gottlieb, 2004;Wasserman et al., 1977). Regardless of counterconditioning, more time should be necessary for the uncertain-stimulusexposed individuals to properly learn the CS-UCS association -although the gap is usually closed within 2 or 3 sessions in the abovementioned studies. In our procedure involving a succession of two CSs, the Group Uncertainty pigeons were perhaps just very slow in acquiring this association and developing counterconditioning.

Surprising Delay Extension Induces Generalization, Extinction, and Reacquisition
The cSNC procedure is known to induce frustration in rats, which decreases their responses during the first downshifted session and then gradually recovers their initial performance over the next downshifted sessions (Cuenya et al., 2015;Flaherty, 1996;Pellegrini et al., 2004;Ruiz-Salas et al., 2020). This effect is typically stronger after certainty than uncertainty training (Pellegrini et al., 2004) because the contrast-induced violation of reward expectation is stronger for certain than for uncertain stimuli. In pigeons, the cSNC procedure has never provided convincing evidence for frustration (Papini et al., 2019). Contrary to rats, "[p]igeons also modify their behavior after downshifts in incentive magnitude, but their output depends on the magnitude experienced before the change, rather than on the ratio of the two magnitudes or on the postshift magnitude" (Pellegrini et al., 2008, p.186). Despite some similarities with the frustration patterns in rats, can frustration be used as an explanatory concept in our study?
Following the sudden (unexpected) extension of the CS duration, the pigeons from both groups showed a gradual decrease in key pecking. This effect was slower than the immediate decrease observed with rats exposed to a traditional cSNC procedure. Indeed, the pigeons responded similarly on Training Session 12 and on Test Session 1, irrespective of the CS and of its duration. No difference in responding could even be shown between the first and the last five trials of Session 1 at test. Thus, contrary to rats, pigeons show a stimulus generalization effect, whether it resulted from the high similarity between the two stimuli (which differed by a time interval only) or from a real failure of discrimination (e.g., Hearst et al., 1970;Lashley & Wade, 1946). These findings are consistent with the accumulated evidence that, contrary to rats and humans, pigeons tend to persevere in responding despite a stimulus change indicative of nonreward in serial and midsession reversal learning tasks (Rayburn-Reeves & Cook, 2016). Generalization or perseveration could explain why pigeons are slow at altering their responses after delay extension.
A second generalization effect was observed in the decrease in responding after delay extension for the extended but also for the unchanged CS, at least in Group Certainty. The absence of generalization to both CSs in Group Uncertainty is difficult to explain, given that both groups responded similarly on Session 12 (last preshifted session) in the training phase of Experiment 1. However, generalization cannot account for the observed subsequent changes in performance: After decreasing their responses to the unchanged and the extended CSs, pigeons trained under certainty showed an increase in responding to the unchanged but not the extended CS. (However, more test sessions might have revealed a significant increase for the extended CS as well.) One plausible explanation for the decrease and the increase that follows among the Group Certainty pigeons might be that, after the shift, the response to the 8-s CS extinguished while a new response to the 24s CS was acquired. Kehoe and Joscelyne (2005) studied the eye blink response of the nictitating membrane in rabbits with a random combination of two CS-UCS intervals, 200 and 1,200 ms. The rabbits developed a response peak at each interval. After retraining them with the 1,200-ms interval in a second stage, they observed extinction of the response to the shorter delay, despite having used the same CS and the same reinforcer for both delays (see also Joscelyne & Kehoe, 2007). Their methodology differed significantly from ours with respect to delay durations, the type of reinforcer, and the nature of the conditioned response. But the gradual (instead of abrupt) decrease in performance after delay extension (Test Sessions 1 to 4) in pigeons looked more compatible with extinction than with frustration. Also, the fact that the pigeons in Group Certainty extinguished their response faster than in Group Uncertainty may simply be a consequence of their higher reward expectation -they learned faster that reward would no longer be delivered.

13
The subsequent increase in performance sometimes observed (Test Sessions 4 to 6) could denote quick reacquisition of the response to the extended CS instead of recovery from frustration. This suggestion may appear counterintuitive given that in Experiment 2, the 24-s CS did not generate any response on its own after 18 training sessions in naïve pigeons. However, we showed that the pigeons that experienced a CS (8-s at training and 24-s at test) beforehand responded more, at least on average, to the new 24-s CS presented alone in Experiment 2. Thus, in Experiment 1, preexposure to the 8-s CS at training may have facilitated quick reacquisition of the response to the 24-s CS at test. Nevertheless, we have to be careful with this conclusion because, contrary to pigeons at test in Experiment 1, the experienced pigeons in Experiment 2 had already been exposed to a 24-s CS. Kehoe and Holt (1984) found that the conditioned nictitating membrane response of rabbits is acquired with a 400 ms CS-UCS interval but not with a 2,800 ms one. When the rabbits initially trained with 400 ms gradually reached the 2,800 ms, their responses decreased at the response level of rabbits initially trained with 2,800 ms. If we (conceptually) compare the 400 ms and 2,800 ms stimuli to the 8 s and 24 s stimuli in our experiments, respectively, we see that the longer delays (2,800 ms and 24 s) are unable to produce conditioning. But we also see an important difference: There was an adjustment (decrease) without subsequent change (increase) to the 2,800 ms in the rabbit experiment, while performance decreased and may increase after the 24-s CS extension. More research is necessary to clarify the role of preexposure to a shortduration stimulus in facilitating the response to a longer-duration stimulus.
It is worth noting that a reduction in the probability of reward and a longer CS-UCS interval duration increased spatial and temporal variability in conditioned responses (Schaal et al., 1998;Stahlman & Blaisdell, 2011;Stahlman et al., 2010). The high variability in the Group Uncertainty subgroup (insert of Figure 4A) and among the experienced pigeons relative to the naïve ones ( Figures 4A and 4B) could be a consequence of the previous training conditions. However, having reused all the pigeons previously trained under certainty would not have substantially changed our observations in Experiment 2. The differences between the experienced and the naïve pigeons would even have been stronger than they are.

Fall in Responding Near the End of a Trial
Frustration theory posits that, in the course of uncertainty training, frustration is learned and can be anticipated owing to the possible omission of reward on a trial. Anticipatory frustration is assumed to decrease responding close to the end of a trial (i.e., in temporal proximity to reward procurement) due to the aversive nonreward that could follow. This effect has been observed with uncertain (as opposed to certain) stimuli using a runway procedure (Amsel et al., 1964;Chen et al., 1980). The second-by-second analysis of the response dynamics carried out in our study revealed a decrease in responding at the end of a trial in pigeons as well. However, this effect was nonspecific; very often, it occurred independent of reinforcement (certainty and uncertainty), CS duration (short and long), and amount of training (early and late). In Experiment 1 ( Figure  3C), it was even obtained after CS extension in Group Certainty but not in Group Uncertainty. We have to conclude that this effect is unrelated to frustration in pigeons, a negative emotion predicted to occur under uncertainty only and whose anticipation is more likely with a long CS duration and late in training because that anticipation may take time to express in behavior (Fuentes-Verdugo et al., 2021).
Another interpretation of the present findings might be that the decrease in key pecking close to the end of a trial in pigeons reflects an effect called inhibition of delay (Pavlov, 1927). In other words, a long CS duration would reduce the salience of the CS and postpone or even cancel the occurrence of a conditioned response (Pavlov, 1927). In rats, inhibition of delay is consistent with the evidence that sign-tracking tends to decrease and goal-tracking to increase within a trial (e.g., Derman et al., 2018;Holland, 1977; for a recent model, see Iliescu et al., 2020). We could not collect information about goal-tracking in our experiments. But one interpretation of delay inhibition that fits quite well most of our within-trial results is that the CS decreases in salience due to reward timing. A number of findings indicate that animals are able to estimate the delay after which a reward is to be delivered, a process that plays a role in conditioning (e.g., Gallistel & Gibbon, 2000). Pigeons are very sensitive to the passage of time; they can anticipate the upcoming switch in a midsession reversal task (Rayburn-Reeves & Cook, 2016) and devaluate a delayed reward more quickly than rats and humans (Green et al., 2004;Mazur & Biondi, 2009;Tobin & Logue, 1994). Thus, it would be no surprise that pigeons are particularly good at assessing the duration of fixed delays as used in the present study. Such a sensitivity to reward timing could result in the expression of an attentional orienting response toward the food hopper that distracts the pigeons from the CS, decreasing key pecking, a phenomenon also shown in rats to a lesser extent (Fuentes-Verdugo et al., 2021). Contrary to other studies, the possible orienting response was unrelated to the reinforcement condition here (certainty vs. uncertainty; e.g., Swan & Pearce, 1987). The pigeons expressed this ability regardless of the conditions in which the CS occurred, suggesting that the process involved reflected a general effect in the task.
However, if the pigeons have learned to expect reward timing (as their rapid decrease at the end of the 8-s stimulus seemed to indicate), the absence of a fall in pecking rates at 8 s just after extending the CS duration to 24 s is hard to explain. They should have made a mistake in anticipating the end of the trial before being able to relearn the new duration. The immediate adjustment that we observed may suggest that the pigeons responded to a signal that the trial was ending rather than to a delay -a "Clever Hans effect", in which a horse was believed to do complex arithmetic operations until it was demonstrated by psychologist Oskar Pfungst in 1907 that the horse was just reacting to involuntary cues in the body language of its human trainer. Although we could not identify any auditory or visual cue that could have been used by the pigeons, this hypothesis cannot be discarded. Nevertheless, this hypothesis does not account for the decrease in pecking observed sometimes several seconds before the end of the trials, especially with a 24-s CS. One possibility that would deserve some investigation is that decreased performance is an effect of habituation. The initial-link CSs are never reinforced (meaningless), relatively long, and repeatedly presented within each session. After the modern characteristics of habituation (Rankin et al., 2009), the decreased pecking rates near the end of a trial could reflect habituation following persistent responding. Spontaneous recovery would occur at the start of each new trial. However, our experiments involved Pavlovian conditioning, an associative process, so habituation as a nonassociative process may play a role but is unlikely to fully explain decreased performance. Also, the decay pattern does not look like the traditional negative exponential to linear function of habituation (Rankin et al., 2009). Further experimental investigation is necessary to clarify this question.

Conclusion
Our results suggest that, in the procedure we used, reward uncertainty alone or a long delay alone decreases key pecking in pigeons, an effect that can result from a decrease in the attribution of incentive salience to a CS. When pigeons are exposed to the sudden (unexpected) extension of a CS duration, this causes a decrease and then sometimes an increase in key pecking. This transition was shown to induce stimulus generalization but also extinction of responding to the shorter CS duration and reacquisition of responding in some cases. The observed patterns are consistent with the well-known strong aversion of pigeons for delays, and there is no evidence that frustration played any role in their expression. The decrease in performance close to the end of the trial is also not a consequence of frustration in our experiments, but its very nature remains unclear currently. Further research is needed to go deeper with this analysis.