Workshop Effectiveness on Content Knowledge of Behavioral Observation Techniques for an Applied Animal Behavior Context

Comparative psychology has a long history of investigating topics that promote comparisons across disciplines, constructs, and species. One critical component of comparative analyses is to select the best data collection technique. Unfortunately, these observational skills are not always taught to individuals who need them the most: animal care professionals. To demonstrate the applicability of appropriate data collection techniques to this applied discipline, we conducted a multi-day workshop that provided attendees training and practice with several data collection techniques that could be used to evaluate animal behavior in both spontaneous and enrichment-provided settings. The program included (1) a presentation on different data collection techniques and the types of questions each technique can address, (2) two 20-minute sessions of observation practice at 2 different facilities, (3) a final summary presentation of the data collected, and (4) pre- and post-surveys conducted immediately before and at the end of the workshop. Out of 177 survey respondents, almost a third reported using behavioral data collection to manage animal behavior prior to the workshop. More than 90% of the respondents had heard of behavioral ethograms, and 68% of the respondents had used one previously. Many of the respondents reported familiarity with different observation techniques. Eighty-two individuals completed the majority of the survey with 81% expressing satisfaction with the initial workshop presentation. Respondents completing both surveys showed significant improvement in their knowledge of behavioral data collection techniques. Ultimately, the workshop introduced and clarified behavioral observation techniques and their applications in a variety of contexts. Respondents indicated that they could and would utilize knowledge gained from the workshop at their own facilities. These workshops illustrate the ability to teach technical content individuals in a short-term, targeted approach. The current paper reports on the results of a four-day workshop on comparative psychology principles and animal behavior measurement techniques given to a group of animal care and management professionals at a professional conference on animal behavior and management. During the workshop, the conference attendees had the opportunity to attend an informational lecture on relevant techniques, engage in hands-on experiences, and complete the conference with a summary presentation of the workshop participants’ efforts.

Comparative psychology has a long history of investigating topics that promote comparisons between human and nonhuman animals. Whether it is a convergence of clinical psychology and comparative psychology or psychology of learning and comparative psychology, the approach to behavioral comparisons enables different fields to complement each other while moving the field forward. Within comparative psychology, there are a number of techniques used to observe behavior. Different questions about behavior require different observational techniques. Altmann (1974) wrote a seminal paper on a variety of observational techniques that could be used to measure different types of behaviors in myriad contexts. Since the publication of Altmann (1974), a number of papers and books have been published that have expanded on these ideas with an emphasis on specific species or contexts (e.g., Altmann & Altmann, 2003;Mann, 1999;Martin et al., 1993).
One context in which comparative psychology has consistently contributed over its history is evaluating the cognitive abilities and behavioral patterns of animals in managed care. Wolfgang Köhler's work on the problem-solving abilities of chimpanzees led to important findings about insight (1925). Edward Thorndike developed a comparative method to study the influence of reinforcers and punishers during various problem-solving tasks with cats, which culminated in the Law of Effect (1927), the precursor to operant conditioning. Nikolaas Tinbergen expanded the comparative perspective to include ethology, which emphasized the importance of examining the proximate and ultimate mechanisms of behavior (1951,1953,1963).

2
Comparative psychology today may fall under many other disciplines, including zoology, behavioral neuroscience, learning, or cognition. One applied area that has benefited from comparative analysis techniques has been the creation of supportive habitats and social groupings for animals in human care. Terry Maple has spent the majority of his life shaping the thinking of zoological facilities to emphasize the importance of interactions that can improve animal welfare, whether at the individual or group level (Maple & Perdue, 2013;Maple & Segura, 2015). One key aspect of this process has been developing the knowledge of the humans who care for the animals.

Why Do Professionals Managing Animals Need To Know How To Collect Behavioral Data?
Learning the life history, environmental requirements, and dietary needs are all important in providing a supportive habitat and life for animals in human care. However, it is also important to be able to evaluate the impact of various environmental changes, social changes, or developmental changes systematically, rather than arbitrarily or anecdotally. Providing staff members, who are managing animals in their care, with the knowledge of how to assess current welfare systematically will enable them to determine accurately where welfare needs improvement (Maple & Perdue, 2013;McPhee & Carlstead, 2010).
One aspect of welfare is to assess the physiological needs of each individual animal through hormone profiles, diverse nutrition, and healthy body condition (Baird et al., 2016;Hill & Broom, 2009). Another aspect includes the social conditions and compatibility for living and breeding (Brando & Buchanan-Smith, 2018;Krebs et al., 2018). Animal care providers have a number of tools available to assess these different aspects of welfare. For physical health, tools include individual physical exams (e.g., ultrasounds, body conditions), bloodwork, excrement samples, saliva samples, and blow samples. In contrast, the behavioral and cognitive health of the animals must be assessed through behavior exhibited during social interactions, expression of species-typical behaviors (e.g., foraging, mating, offspring care), indicators of stable social structure, responses to enrichment, or presence of stereotypical or atypical behavior (Cameron et al., 2005;McPhee & Carlstead, 2010). Fernandez and Timberlake (2008) recognized that keepers tend to approach observational behavioral research to answer applied questions involving animal management (e.g., the animal itself, animal welfare, displaying animals to the public). These researchers suggested that collaborations with academics would facilitate knowledge about animal behavior from a functional point of view while offering insights about animal management to the zoological facilities (Whithman et al., 2013). Likewise, zoological facilities offer academics opportunities that may not be easily obtainable in field settings, creating an excellent synergistic collaboration (Bauer et al., 2010;Fernandez & Timberlake, 2008;Hopper, 2017). Although this perspective has been propagated over the years, collaborative research between zoological facilities and academic institutions has not been as common as envisioned despite clear exceptions (as reviewed by Maple & Perdue, 2013). In general, zoological facilities rely on internal staff to develop and conduct behavioral observations as needed for animal management (Anderson et al., 2010).

What Knowledge Do Animal Care Staff Have (or Not Have)?
While most animal care professionals are very good at determining when their animals are acting uncharacteristically, many are not trained in behavioral assessment techniques (Clegg & Delfour, 2018). Depending on the facility, animal care providers may or may not have a college degree, with a smaller percentage having master's degrees (Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, Animal Care and Service Workers; https://www.bls.gov/ooh/personal-care-andservice/animal-care-and-service-workers.htm). Myriad educational backgrounds and personal interest in animal behavior create a varied set of experiences regarding behavioral assessment.

3
In general, unless staff are encouraged by zoological management or are intrinsically motivated to learn more, most animal care professionals have never or have rarely conducted standardized behavioral observations. In the early 2000s, animal care professionals generally reported having a lack of research experience, knowledge, or expertise in their field (Anderson et al., 2010). Animal care professionals reported wanting to have (1) the support of the zoo director, (2) staff with effort dedicated to conduct scientific programs, (3) well-defined research goals/objectives with a plan, and (4) a strategic plan supported by the facility. The animal care professionals suggested several paths for addressing this knowledge gap: (1) workshops at organized conferences to learn relevant techniques, or (2) collaborations with universities to acquire knowledge and assistance that can be shared with the public (Anderson et al., 2010). This belief was reiterated repeatedly by Maple and Segura (2015) in a review about advancing behavioral analyses for health and welfare of animals and the importance of collaborations with universities. Although it is unclear if these needs have been addressed fully in the last two decades, increases in attendance at discipline specific conferences and collaborations between zoological facilities and universities have been observed.

What Limitations Exist for Collecting Data?
Systematic behavioral research requires a few conditions: (1) knowledge of formulating a question and research design, (2) time to complete data collection, (3) proper recording tools, and (4) data analysis experience. Obstacles experienced by many animal care professionals include limited knowledge or experience, time to collect data consistently, and time or knowledge to analyze the collected data (Anderson et al., 2010). For example, behavioral and cognitive research is not incorporated as often into zoological facilities because it can be challenging and there are limits in costs and time (Hopper, 2017). For some facilities, recording tools may be limited, although a paper and pencil model can work effectively as a data collection tool anywhere. However, designing an appropriate behavioral checklist (i.e., ethogram) that can be used to collect and process data more efficiently requires technical data collection knowledge, such as the most appropriate sampling techniques.
These limitations can be reduced in a number of ways such as collaborating with researchers, training within the facility, or attending sessions/workshops that focus on techniques for systematically creating and collecting the behaviors of the animals. Knowing techniques of how to collect the data is critical to address the reasons for the data collection, (e.g., abnormal behavior, introductions, social hierarchy, enrichment impact, behavioral and social management, breeding partner compatibility, typical developmental patterns, use of environment, overall welfare).

What Information Should be Known by Animal Care Professionals?
The research question will dictate the type of data measurement and the specific methodology. The methodology utilized depends on the chosen behaviors and recording techniques, which include sampling and recording rules. Sampling rules include the approach to the data collection, such as which animals will be studied (e.g., single, dyad, group), where the animals will be studied (e.g., field, enrichment environment), and when the animals will be studied (e.g., time of day, behavioral state) (Altmann, 1974;Martin et al., 1993). Recording rules include the specifics about how the data will be collected (e.g., each minute, continuously, presence/absence) (Altmann, 1974;Martin et al., 1993).
Developmental questions require long-term, individual-based data collection as opposed to specific behavioral issues (e.g., regurgitation, stereotypic behavior, social compatibility) that can be conducted in a limited time frame. Long-term observations enable the evaluation of seasonal and time of day behavioral changes and the development of social interactions. For example, understanding the diurnal and nocturnal patterns of behavior are critical for welfare and could be facilitated by collaborations with universities (Walker et al., 2017). Brando et al. (2018) has emphasized the need to assess animal activities or welfare for 24 hours a day, all days of a week. These around-the-clock studies are time and effort intensive and likely difficult to be 4 completed with consistency at most zoological facilities for a variety of reasons (e.g., streamlined number of employees, limited funding, unclear objectives). Most facilities have recording observation protocols in place for behavioral observations during health concerns or the presence of new offspring. However, these observations are often only used for the immediate purpose of informing the care of the individual animal and usually are not extended beyond the original purpose (e.g., for publication).
One reason for the failure to extend data collection beyond the immediate purpose is the lack of formal knowledge and experience of conducting behavioral observations systematically. Baseline data are necessary to facilitate conclusions about possible relationships or causes related to changes in an environment and subsequent behavioral responses. Many times, the development of a behavioral project may have been driven by a specific reason, but, if valid and reliable observations are not made, then any results from this study are limited and cannot be generalized to other contexts or facilities.

How Can the Knowledge be Acquired?
Research examining the impact of short-term workshops has demonstrated the effectiveness of training specific techniques or more general concepts for professionals whether they are in an applied or academic setting. Womble et al. (2013) conducted a short-term workshop at an international meeting regarding marine mammal free-ranging behavioral data management and analysis. The workshop was conducted by invited speakers who presented on specific topics of behavioral data management and processing. Due to the technical nature of the statistical content, workshop participants were assumed to have some knowledge about the topic (e.g., how to identify behavioral units or analytical approaches that could be used to classify behavior) and were expected to be able to follow along with the various demonstrations and discussions. The workshop was effective, as workshop participants reported advancement in understanding the usefulness and future applications of the electronic device discussed, including standardization of the data collection process.
Compared to short-term workshops, longer length workshops (e.g., 1-3 weeks) facilitate more in-depth learning over selected topics due to the repeated practice and applications of the material. One study examined the effectiveness of a one-week workshop on content knowledge and activities-based learning involving three instructional methods -field trips, direct instruction, and lesson plans at a teacher's professional development workshop. This workshop was developed in collaboration with a zoological facility. The results suggested that content knowledge improved significantly on both standardized and self-reported assessments. Although improvements were observed, workshop facilitators expected greater increases than what actually occurred, which may have been due to the complexity of content. Feedback from the teachers indicated that they did apply the lesson plans and knowledge acquired from the one-week workshop in their classes following the workshop (Pecore et al., 2013).
These workshops illustrate the ability to teach technical content to individuals in a short-term, targeted approach. The current paper reports on the results of a four-day workshop on comparative psychology principles and animal behavior measurement techniques given to a group of animal care and management professionals at a professional conference on animal behavior and management. During the workshop, the conference attendees had the opportunity to attend an informational lecture on relevant techniques, engage in hands-on experiences, and complete the conference with a summary presentation of the workshop participants' efforts.

Purpose
The professional experience of the authors includes teaching behavioral observation techniques to college students. Recognizing the real-world implications that could assist those working on animal welfare, the authors wished to share this knowledge with animal care professionals. The primary goal of the paper is to illustrate the effectiveness of the workshop activities on developing or refining knowledge about appropriate 5 types of data collection techniques and their applications to different contexts. Conference attendees were asked to complete a pre-workshop survey, which assessed their existing knowledge of data collection techniques and previous experience with specific techniques. Following the opening informational lecture, attendees were then randomly assigned to different times, animals, and data collection techniques for a 20-min periods at each zoological facility visited across a two-day span. The conference attendees collected various types of data on different species, which were then used to demonstrate the usefulness of those data collection methods for various questions of interest. In addition to sharing some of the results, the final portion of the multi-day workshop allowed the attendees to share their experiences with the different techniques utilized across the two facilities on a post-workshop survey.

Specific Hypotheses
Using pre-and post-workshop surveys, attendee knowledge was assessed regarding behavioral observation techniques. The previous experiences and knowledge with behavioral observation techniques was also examined. It was expected that attendees would improve their knowledge regarding behavioral techniques and would report that they were willing to incorporate this knowledge in the future.

Workshop Survey Respondents
All participants were attending an annual conference on animal behavior and management, sponsored by the Animal Behavior and Management Alliance in March 2018. Conference attendees were associated with facilities caring for animals, including large and small aquariums, sanctuaries, and zoological establishments. The five-day conference was structured as a single session, with no concurrent sessions; thus, all attendees for the day were expected to be present for each presentation or session. During check-in, the attendees were provided a schedule for their observational experiences, which assigned the location, the animal, the time of day, and the type of ethogram for the two different zoological facilities they would visit during the conference. However, conference attendees could come and go as needed and were able to attend any conference day of their choosing. Thus, not all attendees were present for all four activities sponsored by the workshop on behavioral observation techniques. However, as the primary workshop sponsored by the organization, all conference attendees were given the opportunity to participate in all aspects of this four-day workshop. A pre-workshop survey was completed on the first day of the conference before the workshop, and a post-workshop survey was completed on the last day of the conference after the closing workshop presentation. Out of 177 pre-workshop survey respondents and 73 post-workshop survey respondents, 60 respondents completed both surveys. To maintain anonymity, demographic information on the pre-workshop survey only included previous experience directly related to behavioral observations. We did not request any other demographic information as we were only interested in experience related to the workshop content.
The pre-workshop survey was completed by 177 participants, with some participants not answering specific questions. Out of 174 people responding to whether or not they had previous experience with systematic behavioral observations, 62% (n = 108) had conducted systematic behavioral observations previously. Out of those who responded to having previous experience (n = 107), 42% (n = 45) had completed one project, 22% (n = 24) had completed two projects, 15% (n = 16) had completed three projects, 1% (n = 1) had completed four projects, and 20% (n = 21) had completed five or more projects. Also of interest was previous knowledge of and experience with ethograms. Out of 173 people responding to whether or not they had heard of an ethogram, 82% (n = 142) had heard of this tool, 9% (n = 16) thought they had heard of this tool but were not sure, and 9% (n = 15) had never heard of this tool. Also out of 173 people responding whether they had used an ethogram, 69% of the respondents indicated that they had used an ethogram before (n = 119); the remaining 31% (n = 54) had never used an ethogram before.

Sample
For 60 respondents who completed both the pre-workshop and post-workshop surveys, whether or not they had previous experience with systematic behavioral observations, 65% had conducted systematic behavioral observations previously. Out of those who responded to having previous experience (n = 39), 46% (n = 18) had completed one project, 18% (n = 7) had completed two projects, 21% (n = 8) had completed three projects, and 15% (n = 6) had completed five or more projects. Also, of interest was previous knowledge and experience of ethograms. Out of 60 people responding to whether or not they had heard of an ethogram, 85% (n = 51) had heard of this tool, 10% (n = 6) thought they had heard of this tool but were not sure, and 5% (n = 3) had never heard of this tool. Also, out of 60 people responding whether they had used an ethogram, 72% of the respondents indicated that they had used an ethogram before (n = 43), and the remaining 28% (n = 17) had never used an ethogram before.

Materials
As part of the four-day workshop presented to conference attendees, a lecture on different behavioral observation approaches was presented on the first day of the conference. Behavioral observation practice with an ethogram occurred at two different zoological facilities on the second and fourth days of the conference, and a closing workshop presentation occurred on the fifth and last day. Surveys were given immediately before the first workshop lecture and immediately at the conclusion of the closing workshop presentation.

Pre-workshop Lecture Content
The initial workshop lecture included content on three different approaches, the purpose for each approach, and time frames for data collection. Workshop attendees learned about continuous behavioral sampling, instantaneous sampling, and one-zero behavioral sampling techniques. These approaches were selected, based on practical experiences with our own comparative psychology classroom exercises, for greatest application to the audiences' needs, and to minimize techniques to be taught while maximizing the opportunities for comparisons. For each of the approaches, content was provided on how the sampling technique would be used in certain situations (e.g., for continuous sampling, we discussed activity budgets as they related to duration and frequency data; for instantaneous sampling, we discussed scanning across time intervals and indicating the behavior of a single animal at each interval, which allows for estimating the approximate time spent in certain activities; for one-zero behavioral sampling techniques, we discussed how many behaviors were exhibited in one time interval and its use in recording diverseness of behaviors, without consideration of rates or durations).
Attendees were provided an example of the ethogram, viewed a video to demonstrate each technique, and were provided some practice for each approach. For continuous behavioral sampling, the attendees viewed a video on beluga behavior and were asked to record the beginning and ending time of each behavior, given as a possible behavior from a shortened ethogram. For the instantaneous sampling, attendees viewed a video on rhesus monkeys engaging in different individual behaviors and were asked to record the behavior of the individual at the beginning of each 1-min interval. For the one-zero sampling, attendees viewed a video on baboons socially interacting and engaging in a variety of different behaviors that could have been observed and were asked to place a mark for each behavior that occurred in each 1-min period.
In the lecture, attendees also viewed examples of completed ethograms that they would be completing as part of the handson experience during their visits to the two zoological facilities. In addition to viewing animal behavior at the two zoological facilities, some attendees were also asked to record the arrival and departure times of specific visitors and the total number of visitors at 1-min intervals. To facilitate this data collection practice, two different ethograms were prepared and demonstrated to the workshop attendees. One ethogram was used to record the arrival and departure times of individual guests at exhibits. A second ethogram was used to record the number of guests that were present at the beginning of each 1-min interval.

Pre-and Post-workshop Surveys
Two surveys were constructed for the pre-and post-workshop surveys (available in Appendix A). IRB approval for the surveys was granted by the University of the Incarnate Word. Participants were able to access the study from any electronic device with internet access. Hard copy surveys were also available for individuals who did not have an electronic device or internet access. Individuals who chose to participate in the study were provided with a bar code or a website to access the study. This barcode and URL directed participants to the study administered via Qualtrics (Qualtrics, Provo, UT), a web-based software platform that enables researchers to conduct data collection online. Participants completed an informed consent for each survey and were provided with a debriefing statement at the end of each survey. A unique code was requested from each participant to allow for the matching of both surveys. The pre-workshop survey contained 27 questions, including forced choice responses to questions about sampling rules, recording rules, ethograms, and open-ended questions regarding previous research experience in general and specifically about data collection techniques. The post-workshop survey had 16 questions from the pre-study that tested knowledge related to ethograms and sampling and recording rules, as well as 2 open-ended questions about participants' experience with the workshop.

Ethogram Data Sheets
Five different ethogram data sheets (i.e., continuous animal behavioral sampling, 1-min instantaneous animal behavioral sampling, one-zero 1-min animal behavior sampling, continuous visitor presence duration, instantaneous sampling of number of visitors per 1-min intervals) were created for consistent data collection at animal habitats selected for hands-on practices at each zoological facility. The behaviors selected for ethogram datasheets were the same across the animal habitats to facilitate the comparison of behaviors with the different techniques (Appendix B). All behavioral observations were 10 min in duration.

Closing Post-workshop Lecture Content
On the final day of the workshop, attendees heard about the findings of the data collected during the week. In this closing presentation, examples of data analysis were provided for each sampling technique utilized from the hands-on experiences. In an open forum, attendees were asked to share their impressions of each type of sampling technique and efficiency and functionality of data collected. The advantages and disadvantages of different time frames and the importance of knowing one's purpose for the behavioral observations were highlighted. We also discussed the issue of missing data, its impact on behavioral analysis, and how to address it with future studies. Finally, we discussed some of the confusion with recording the data properly, the influence of limited practice, and the need for inter-observer reliability.

Procedure
Conference attendees were informed of the conference-sponsored workshop on behavioral data collection techniques during registration. As attendees registered, they received two stickers, which indicated the time, animal, and two types of ethograms to be used at each facility that they would visit during the week of the conference. All registered attendees were randomly assigned these stickers, but some attendees asked for selected times to accommodate individual schedule of activities for each facility. The color of the sticker (e.g., green) indicated the time of day for data collection (e.g., 10:30 am), the first number (1-4) indicated the animal habitat to observe (e.g., mongoose at the zoo, sea lions at the aquarium), and the second number indicated the two types of observational techniques (e.g., continuous sampling/one-zero behavioral sampling or one-zero behavioral sampling/instantaneous sampling).
Immediately before the first-day presentation, workshop attendees were asked to complete a survey in which they indicated the techniques they were accustomed to and had utilized at their facilities. During the workshop, attendees practiced each technique with video clips provided by the presenters following instruction on each technique. On the second and fourth days of the five-day conference, attendees participated in data collection at two different facilities, an oceanarium and a zoological facility located in San Antonio, Texas, USA. Each participant spent 20 min at the randomly assigned location collecting data. The 20-min period was split into two 10-min data-collection periods in which attendees actively collected data using two different techniques. The techniques were randomized so that every person had a chance to practice different techniques across the two facilities. Each facility had four habitats that were pre-selected for observations across the day. Some observations were of natural, spontaneous behavior of the animals, some occurred during feeding times by the public (if applicable), some during feeding times by the facility staff members, some occurred with different forms of enrichment (predetermined) provided by the facility, and some involved watching human guests while at the exhibit. The data were collected on paper ethogram datasheets, which were provided to each attendee at the observation location by research assistants. Attendees returned the datasheets to the research assistant at the completion of the 20-min data collection period. The trained research assistant coordinated the start and end times for each 10-min observation period and answered any questions regarding data collection techniques. The data were entered into a database for each habitat by the workshop leaders for use during the closing workshop. The closing workshop presentation presented an overview of the two days of data collection, including data about the animals observed, the quality of the data collection, and the disadvantages and advantages for each technique practiced. Immediately after the presentation, attendees were given the opportunity to complete the post-survey.
Following the completion of the conference, the data from the pre-and post-surveys were processed to evaluate the effectiveness of the workshop experiences. All responses on both surveys were initially processed. Data that were incomplete were excluded from being processed. Using the unique code provided by the respondents, the pre-and post-surveys were matched for data analysis to evaluate the effectiveness of the workshop. All open-ended questions were split and scored by two raters (RW/MG) and confirmed by a third rater (HH). Tables 1-5 summarizes the operational definitions of all themes developed for open-ended questions from the surveys. 18 "Bear pacing observations."

= Moderate Explanation
Short summary of their use of an ethogram on a project.
36 "Conducted wolf observations to see how active they are throughout the day." "Jaw clapping with a harbor seal. Note when behavior occurred and staff activity. 99% of the time she jaw clapped people were present."

= Detailed Response
Detailed summary regarding the type of ethogram or layout of the project.
24 "I took samples using both scan and behavioral depending on the animal group requested in our small animal building during night events. We were able to determine enough stress responses." "Grey squirrels -scanned for behavior every 30 sec, categorized behavior as foraging, social, anti-predator or other. Compared urban to forest squirrels. Humans -recorded every instance of greeting behavior and group demographics at a local coffee shop for 2 hours. Compared demographics. Snakes -continual observation for 20 min after enrichment placed, recorded every behavior, compared to snakes whose enclosure was opened but no enrichment placed." Note. From the pre-survey: "Briefly describe one project you conducted with each method." The respondent mentions having some type of difficulty when reflecting on collecting data for behavioral management.
37 "I currently have a team of keepers collecting data on one animal and they are using a photo scoring system. We started by having everyone record and enter their data when they worked the area but found it probable that even with photos people were choosing different scores. So now just one person uses the scoring system." "Continuous Cindi collecting on a group of 6 primates for 15 minutes was hard." Theme 2: Just Documenting Behavior/Experience (Neutral) The respondent does not mention any difficulties or pleasantries experienced during their time collecting data for behavioral management.
12 "Noticing pacing happens long after keepers walked away." "Many years ago I watched a gorilla and recorded when he scratched at a scab." Theme 3: Facilitating Change (Successful or Unsuccessful Attempt) The respondent mentions trying to bring about changes for respective settings or practices. This can be an acknowledgment of bringing change or an actual attempt that aimed to do so, whether successful or not.
9 "We were able to provide enough evidence to close the small animal building for certain night events." Theme 4: Useful/ Good Experience/ Insightful The respondent notes that collecting data for behavioral management was beneficial or helpful in some regard.
22 "Not much experience, but it went well." "I did a 30-hour behavioral study on a ocelot. The time itself felt really long. The good thing was watching the exotic cat interact with enrichment and trainers. The ugly was watching the animal straining and pacing." Theme 5: Not Very Useful or Insightful Experience The respondent states that collecting data for behavioral management was not helpful.
2 "Our ethogram of regurgitation did not result in any great answers on how we could reduce this undesirable behavior." "Good-common goal/information for behavioral modification; Bad-time; Ugly-not getting any correlating data"

Theme 6: N/A or No Previous Experience
The respondent does not have any prior experience collecting data for behavioral management.
13 "I can't think of one." "I do not have much experience in this." Note. From the pre-survey: "Please describe a specific experience with collecting data for behavioral management. What was the good, the bad, and the ugly during this experience?" Some responses were scored for multiple themes due to the level of detail provided. 39 "A behavioral ethogram is the use of one of many sampling charts to monitor one or many behaviors over a certain amount of time. Can be averaged over many observations." The respondent mentions that they learned something that they did not know before.
9 "I learned that it takes a lot of practice if you want the data to be accurate. I also learned more about how to use the different kinds of observation in specific situations. I'll defiantly try to practice more and use these skills in the future. Thank you very much!" "I liked learning about the techniques and then trying them in different situations, some situations when the data collection process appeared smooth and other situations where it appeared chaotic" Theme 2: Better Understanding The respondent states that a better understanding on the methods or applications of the workshop were acquired.
16 "I walked away with better understanding of the technical terms. I already use continuous focal follows. Would be interesting to try other focal follows to see if the data is similar to possible cut down on coding time when we all struggle with time." "I have learned pros and cons to different ethograms. I may share this information with co-workers to learn more about the animals we work with." Theme 3: Ethogram Practicality The respondent mentions that the workshop showed that an ethogram was practical.
16 "It was a good refresher on the best types we can use for obs. Also being able to see the different types results side by side to see what you end up missing with the different types." "We need to do more of this in our zoos and aquariums to not only improve the welfare of our animals but also prove to people that we do take care of our animals with the best practices." Theme 4:

Refresher/ Reminder
The workshop helped the respondent remember something they may have forgotten.
4 "I was reminded of ethograms and learned more about different data collection techniques. I definitely want to be able to use these techniques in the future." "It was a good refresher on the best types we can use for obs. Also being able to see the different types results side by side to see what you end up missing with the different types." Theme 5: Stressful The respondent states that the workshop impacted them in a negative manner.
4 "This is all still very not intuitive to me and is quite stress inducing. I'll still need someone to explain it 4 more times to use it." "Kinda of interesting, kind of confusing. Might use some of it in the future." Theme 6: Not Helpful The respondent states that the workshop was not very useful.
1 "It was an ok introduction if a facility doesn't have previous experience with data collection but didn't get enough information to change current observation procedures or how to evaluate data. Maybe setting this type of presentation up as the workshop day would be more effective. It was hard to work into the field trip days between tours based on location and times." Theme 7: No Response 3 "Yes." Note. Some responses were scored for multiple themes due to the level of detail provided. The respondent states that the practicality of an ethogram would be useful for documenting nondaily or irregular behaviors of the animals.
13 "Look at stress in kennel."

3: Enrichment/Welfare of Programs/Animals
The respondent states that the practicality of an ethogram would be able to serve as a way to improve or enhance a program for animal's wellbeing.
29 "Figuring out how to make the best user of animal space and enrichment"

4: Unsure
The respondent is not sure how an ethogram would be practical.
1 "I am not sure, it is not anything I have thought of doing. "

5: Answer Doesn't Make Sense
The respondent's answer is not clear or specific.

"Definitely!!"
Note. From the post-survey: "How could a behavioral ethogram be useful for your work?" Some responses were scored for multiple themes due to the level of detail provided.

Statistical Analyses
To illustrate changes in knowledge as associated with engagement in the workshop, quantitative analyses utilized matched pairs data only. Paired sample t-tests and chi-squared tests for distribution of responses were calculated from pre-to post-survey responses. Qualitative analyses included all possible data to evaluate different aspects of the workshop. Descriptive statistics are provided as support for the effectiveness of the workshop for both knowledge content and applicability of acquired knowledge.
To evaluate the accuracy of responses from experiences prior to the workshop (e.g., "In your experience, which sampling rule would allow you to focus on a single or pair of animals?") and compare it to accuracy from experiences acquired throughout the workshop (cf. pre-survey question), the data from questions involving sampling rules were first identified as correct or incorrect and converted to a percentage accurate. The percentages from each sampling rule question were then averaged to create a composite accuracy score for sampling rules based on experience before and after the workshop. The same process was conducted to calculate the accuracy scores for recording rules based on previous experience before (e.g., "In your experience, which recording rule would allow you estimate the percentage of time in an activity?") and after the workshop (cf. pre-survey question). The same process was utilized for questions regarding definitional knowledge of sampling rules and recording rules (e.g., "Identify which sampling rule would be used to observe behavior spontaneously and intermittently."; "Identify which recording rule would be used if a behavior occurred at an interval indicator (beep)").

Sampling Rule Familiarity
Respondents demonstrated a significant increase in understanding of the focal sampling rule definition from pre-to post-survey. The majority of respondents continued to correctly identify scan sampling rule definitions and behavioral sampling rules from pre-to post-workshop survey, indicating that this knowledge remained stable. Interestingly, while the majority of the respondents correctly identified ad libitum sampling as the response to observing behavior spontaneously and intermittently, before the workshop, a smaller percentage correctly responded to the question on the post-survey, appearing to confuse ad libitum sampling with behavioral sampling or focal sampling as shown by nonsignificant increases in number of participants selecting these techniques. Table 6 summarizes the specific descriptive and chi-squared statistics for each type of sampling rule. The figures illustrating the outcomes of each question may be found in Appendix C.

Recording Rule Familiarity
Almost all of the respondents were able to correctly identify the definition of continuous behavioral recording both before and after the workshop, demonstrating that this knowledge was maintained by the workshop experiences. Encouragingly, significantly more respondents correctly identified instantaneous sampling recording rule following the workshop as compared to the beginning of the workshop, in which approximately half of the sample were able to correctly identify the definition. Similarly, the number of respondents doubled from pre-to post-workshop on their ability to correctly identify the one-zero recording rule definition. Table 6 summarizes the specific descriptive and chi-squared statistics for each type of recording rule. The figures illustrating the outcomes of each question may be found in Appendix C.

Application of Sampling and Recording Rule Knowledge
When respondents were asked to apply their knowledge to different situations in which the various sampling and recording rules could be applied, a growth in knowledge was observed. First, the number of respondents who correctly identified that the one-zero recording rule would be used to indicate whether various behavior occurred within an interval almost doubled from pre-to post, representing a significant change in distribution. Second, significantly more respondents correctly indicated that the continuous recording rule is best when attempting to capture rare events. The remaining rules were already well identified with their knowledge remaining stable from before to after the workshop. Knowledge about utilizing scan sampling is best applied to a group of animals increased from pre-to post-workshop survey, but this statistic did not reach a probability of .05. Knowledge about behavioral sampling being applied to examine a specific behavior also increased in number of respondents, but it did not represent a significant increase in distribution. Finally, the recording rule that should be applied to assess behavioral states did not change significantly from pre-to postsurvey, as the majority of respondents continued to endorse that continuous (all occurrence) sampling was the most accurate representation of behavioral states. Despite this lack of statistical significance, more respondents endorsed this recording rule following the workshop experience. Table 7 summarizes the specific descriptive and chi-squared statistics for question regarding application of sampling and recording rules. The figures illustrating the outcomes of each question may be found in Appendix C. Figure  Application  To evaluate the existing knowledge from previous experiences with behavioral observations, the accuracy scores on sampling rules and recording rules were compared from before to after the workshop experiences, respectively. The results of two paired sample t-tests indicated that participants were significantly more accurate in their averaged responses to sampling rule questions, t(59) = -4.70, p < .001, and to recording rule questions, t(59) = -2.94, p = .005 following their week-long experiences. Table 8 summarizes the descriptive statistics for each analysis.

Application of Sampling and Application Recording Rule Knowledge Pre-and Post-Test
To evaluate the definitional knowledge of behavioral observation techniques, accuracy scores on questions about sampling rules or recording rules were compared from before to after the workshop experiences, respectively. The results of two paired sample t-tests indicated that participants were significantly more accurate in their averaged responses to recording rule questions, t(59) = -4.20, p < .001, but were the same for sampling rule questions, t(59) = -0.26, p > .05 following their week-long experiences. Table 8 summarizes the descriptive statistics for each analysis.

Previous Experience with Behavioral Data Collection
Out of the original 177 respondents, 95 described a previous project using ethograms with varying details (Table 1). The majority of these responses (86%, n = 82) included descriptions of projects conducted at various facilities for a variety of purposes. In general, respondents provided moderate levels of description (38%) regarding specific projects followed by detailed levels of description (25%) and then simple descriptions (19%) ( Table 1). Some respondents indicated their experience was from a class project (9%), and the remaining did not have experience with behavioral observations (8%). Some respondents then expanded upon specific experiences in which the outcomes of behavioral observations and/or techniques were described. Of these shared experiences, five themes emerged, excluding those without experiences (14%). As summarized in Table 2, the most frequently occurring theme was reported by respondents as having issues with knowing how to record the data (39%). Following this theme, 23% of respondents reported the experience as useful or insightful, 13% of respondents reported examples of behavior documentation but did not indicate usefulness in comment, 9% reported using behavioral observations to facilitate change, and 2% reported the experience as not useful or insightful.

Ethogram Accuracy
On the post-workshop survey, the majority of respondents (62%) accurately defined an ethogram, with another third (37%) identifying some key elements in their description (e.g., time frame, behavioral categories, method of data collection). Table 3 provides examples of each category of accuracy along with frequencies of respondents.

Knowledge Gained from Workshop
Several themes emerged from participant responses to a post-workshop survey question regarding their knowledge gained from the workshop experiences (Table 4). The two most frequently occurring themes were a better understanding (30%) and ethogram practicality (30%). Learning something new was also endorsed by 17% of all respondents. Finally, 8% of respondents reported the workshop experiences as a refresher or reminder, and 8% of respondents found the workshop experiences as stressful.

Perceived Usefulness of Behavioral Observations in the Future
From respondents who provided a response about the perceived usefulness of behavioral observations (i.e., ethogram) in the future (n = 53), 43% (n = 23) provided information about whether they would use an ethogram in the future. Of those 23 individuals, more than half of the respondents (57%, n = 13) intended to use an ethogram in the future following their workshop experience, with about a third (35%, n = 8) indicating they might use an ethogram, and 9% indicating that they would not use an ethogram in the future (n = 2).
Almost half of the respondents (43%, n = 34) indicated that they would use behavioral ethograms in the future to identify normal behavior, and another third (37%, n = 29) indicated that would use behavioral ethograms to examine enrichment or welfare of animals. A smaller number of individuals (16%, n = 13) reported that they would use ethograms to identify abnormal behavior with the remaining responses indicating no response or unsure ( Table 5).

Effectiveness of Workshop Experiences from All Attendees Post-Survey Responses
Attendees were asked to rate their satisfaction on the workshop presentation that they attended at the beginning of the week. Of the 82 attendees who completed the post-workshop survey, the majority of the attendees indicated that the presentation was satisfactory (n = 66, 81%). Attendees also provided information regarding their understanding of the tools used for data collection and their enjoyment of collecting data at each facility. For both facility experiences, the majority of attendees expressed that they understood how to use the tool (Facility 1: N = 80, n = 71, 89%; Facility 2; N = 75, n = 69, 92%). Attendees also found their experience of collecting data during the week-long workshop enjoyable (Facility 1: N = 79, n = 52, 90%; Facility 2: N = 73, n = 47, 65%). Overall, post-survey respondents (N = 81) indicated that they would use the behavioral ethogram in the future (n = 64, 79%).

Did the Workshop Improve Knowledge?
Utilizing knowledge derived from comparative psychology and behavioral data collection techniques, we attempted to demonstrate knowledge about techniques to collect behavioral data in zoological facilities under different circumstances (e.g., time frames, number of animals, types of behaviors, overall purpose). Using a multifaceted approach to deliver the desired knowledge, attendees experienced informational presentations with interactive demonstrations and hands-on experiences in which the knowledge was practiced. After the one-week conference with four days of instruction and interaction, we found that knowledge did improve across multiple measures as expected for at least the duration of the conference.
About 80% of the conference attendee respondents reported being familiar with the various techniques presented in the workshop, with the majority of respondents having some prior experience. Of the individuals who reported having previous experience with conducting behavioral observations, many of the experiences were associated with health concerns, atypical behaviors, and response to enrichments. In terms of the technical knowledge of the respondents who completed both the pre-and post-surveys, the majority of the respondents were able to choose the correct answer for the different sampling rules on the pre-workshops surveys, including focal follows, scan samples, behavioral sampling, and ad libitum sampling. The responses continued to be consistent on the post-workshop survey with a couple of caveats. After the workshop experiences, respondents appeared to confuse ad libitum observations with other types of sampling rule techniques, such as focal follow or behavioral sampling, although the majority still correctly identified the appropriate technique. Additionally, behavioral sampling was confused with focal follows, scan samples, and ad libitum more after the workshop than before, although behavioral sampling was still identified properly in most cases. These small changes most likely occurred due to a lack of emphasis in the workshop on these two sampling rules. Overall, respondents completing both surveys significantly improved their knowledge, and more participants responded correctly when asked to pair definitions with a specific sampling rule (Figures 1-4).
Similar increases in overall accuracy were also observed for recording rules, especially for instantaneous sampling and one-zero sampling. Almost all had heard of continuous or all occurrences sampling prior to the workshop, whereas almost no one had heard of instantaneous or one-zero sampling prior to the workshop (Figures 5-7). However, by the end of the week-long workshop, there was a significant change in the number of correct matches for the two recording rules that were unknown prior to the workshop.
Application of sampling rule responses remained consistent in accuracy from pre-to post-workshop responses (Figures 8-10), suggesting that respondents were knowledgeable of the different sampling rules or were able to discern the correct answer from the names of the technique. In contrast, recording rule knowledge increased from pre-to post-workshop surveys (Figures 11-13). Respondents were more accurate in determining which recording rule should be used for specific types of questions following the week-long experience. The most accurate responses included continuous recording or all occurrence sampling and one-zero sampling. However, respondents appeared to have some confusion about the use of an instantaneous rule, which would provide an estimate of time in an activity. The respondents' accuracy on both sampling and recording rules were further evaluated with questions regarding previous experience (e.g., previous familiarity with the application of the techniques) and definitional knowledge (e.g., factual information). When previous experience was considered, the respondents significantly improved on accuracy in knowledge of both sampling and recording rules. In contrast, when definitional knowledge was examined, the respondents significantly improved their definitional knowledge only for recording rules; sampling rules accuracy remained stable across the week with the majority demonstrating a relatively high level of knowledge already. Finally, respondents were more accurate about describing an ethogram following their experience with the workshop.
If this workshop was successful, we would expect participants to use the knowledge to address different issues at their home facility. The overall results indicated that knowledge about data collection procedures increased from before to after the workshop and practical applications, supporting our desired outcome. The participants generally enjoyed the experience and were satisfied with the topic and practice. Over half of the respondents who completed both surveys indicated that they planned to use a technique in the future. Respondents who indicated that ethograms would be used in the future reported that it would focus on normal behavioral patterns or enrichment and welfare for the animals, topics that are of critical importance to animals in managed care (Maple & Perdue, 2013). In summary, the week-long workshop was considered a good refresher for many attendees who had used these comparative techniques previously (Fernandez & Timberlake, 2008). The workshop was also successful in providing knowledge of new techniques for attendees who were not previously familiar with these data collection options, much like previous studies (Pecore et al., 2013;Womble et al., 2013). Lastly, many individuals indicated that the opening presentation was helpful, the hands-on practice was a good experience, and the closing summary was helpful but overwhelming for some respondents.

Limitations
As a four-day workshop conducted within a week-long professional conference, several lessons were learned through the process. First, the goal to randomly assign attendees to specific times, locations, and observation techniques was difficult to achieve due to competing scheduling issues and partner selection. Second, many of the attendees who were present on the first day and completed the pre-survey did not necessarily attend the two days of hands-on activities and/or the final presentation day and, thus, did not complete the post-survey. Third, attrition occurred due to overlap in unique identifiers, which resulted in the exclusion of duplicated codes. Finally, it was clear from the knowledge-content responses that some of the techniques (behavioral sampling, ad libitum) were not sufficiently emphasized, primarily due to time. Despite these limitations, the data gathered offer a valuable contribution to evaluating the immediate, if not long-term, impact of educational content to professionals working within an applied field of animal behavior or comparative psychology. The positive outcome of this workshop was due to the collaborative efforts of the professional organization (Animal Behavior and Management Alliance), two zoological facilities, and two academic-based comparative psychologists.

Future Applications and Suggestions
The purpose of the four-day workshop at a conference for a professional organization was to both educate professionals working in applied animal behavior while also illustrating the possible collaborations between professional organizations, zoological facilities, and academic institutions (Fernandez & Timberlake, 2008;Maple & Perdue 2013). In addition to providing educational content, another goal of the workshop was to illustrate the steps that were needed to extend data collection from an immediate purpose (e.g., a temporary health issue) to a long-term opportunity to share information with guests, other professional organizations, and the scientific community. In order to achieve these long-term objectives, data must be collected accurately and consistently. Having the technical knowledge and tools is necessary to develop these objectives. This knowledge can be acquired, as described previously, through similar types of workshops, a single day meeting, monthly brown bag presentations, one-on-one interactions, or self-learning using online resources (e.g., "Wild discoveries" series NSF/UF/Santa Fe College, https://youtu.be/KAYE-7L77Oc). There are several limitations to many of these suggestions. For example, the experts who are sharing knowledge are generally doing so voluntarily, the animal care professionals are limited in availability and resources, and accessibility to zoological facilities may be difficult to obtain.
The most important aspect though is that the exposure to content knowledge should not be a one-time experience but rather reinforced periodically with refreshers, such as small-group discussions or master lectures at a regional conference. A follow-up survey investigating the effect of the knowledge acquired from this workshop would be highly beneficial. All technical skills need to be practiced in different contexts to be maintained, which would suggest that animal behavior data collection should become a priority for facilities and consistently practiced by animal care professionals (Anderson et al., 2010). These skills could also be part of an outreach education program sponsored by the facility and/or academic experts to develop citizen scientists who wish to assist in data collection. The bottom line is to excite animal care professionals, academics, and the general public to learn more about the animals they care for, study, or observe, respectively. By creating a priority to study animal behavior systematically, the welfare of animals housed in zoological settings may be directly impacted and improved with more knowledge (Anderson et al., 2010;Fernandez & Timberlake, 2008;Maple & Perdue, 2013). Ultimately though, this knowledge should be utilized beyond the facility itself and shared with the larger communities -both scientific and general.