School-Based Observation -- Briesch, Volpe, & Floyd

Types of Observations

Indirect Forms of Assessment: e.g., interviews and rating scales; easier to make but less reliable because they rely on people's memories.

Anecdotal Observation: entering the observation setting (e.g., classroom) and recording behaviors as they occur. (Naturalistic settings: conditions we do not manipulate or control.) Limitations: produce only qualitative data; this method is somewhat subjective; prone to confirmatory bias (i.e., we are more likely to pay attention to and record behaviors that confirm our initial thoughts). But anecdotal observations "can provide a broad picture of both student functioning and the naturalistic environment and can inform the selection of more systematic assessment methods." See the Hybrid Observation Form (Appendix C.9).

Systematic Direction Observation: the use of specific observation criteria to quantify a small number of target behaviors or behavior categories. Difference between specific behaviors (e.g., calling out) and behavior categories (e.g., disruptive behavior). Must first operationally define the target behaviors or behavior categories. An operational definition is clear (i.e., unambiguous), complete (the boundaries are explained, example and nonexamples given), and standardized (the same procedures and decision rules are used to code behavior under similar conditions). Limitations of systematic direct observation: we need more than just numbers, may miss other useful information.

* * * * *

Methods of Systematic Direction Observation

Recording Event Behaviors. Events are discrete behaviors that are short in duration. Discrete behaviors are those with a clear beginning, middle, and end (e.g., hitting). With discrete behaviors, the assessor is typically interested in how many times the behaviors occur or how long a behavior occurs or how much time elapses before the behavior occurs.

Event Recording (Reproducible C.1). Put a check mark each time student engages in behavior. Don't record duration, just if student engaged in behavior. Good for recording discrete behaviors that occur at low rates and for a fairly consistent duration -- e.g., would not make sense if there were three instances of a behavior and first instance lasted for 1 second and second instance lasted for 3 minutes.

Duration Recording (C.2). Use if duration of behavior is relevant. This involves noting the length of time a behavior is present for each occurrence of the behavior. Use a stopwatch.

Latency Recording (C.3). Use to determine the length of time between a prompt and the initiation of the behavior -- e.g., how long it takes student to comply with teacher's request.

Recording State Behaviors. State behaviors are longer in duration than event behaviors and do not always have a clear beginning or end. With state behaviors, we're less interested in the number of times the behavior occurs than in the proportion of time during a session that a behavior occurs.

Interval Recording (C.4). Dividing the observation session into smaller units or intervals of time -- e.g., a 10-minute observation is divided into 40 15-second intervals. The observer records the presence of the behavior of interest for each interval. Great to use when exact quantification not necessary and when it is disirable to measure than one behavior. Three types of interval recording: (1) partial interval, (2) whole interval, and (3) momentary time sampling.

1. Partial Interval Method: record the presence of a behavior if it was present at any time during interval; good for recording behavior that occur at a relatively low rate; this method tends to overestimate the duration the behavior was present. 
2. Whole Interval Method: record the presence of a behavior only if it is present over the entire length of the interval; this method tends to underestimate the proportion of time behaviors are present. 
3. Momentary Time-Sampling Method: the observer records whether a specific behavior is present at a particular instant (e.g., the instant the interval begins or ends). Research has shown that this method is more accurate than partial interval and whole interval methods. 

Considerations for Selecting Interval Recording Methods. Don't use if interested in the frequency of a behavior. Best suited for quantifying behaviors of longer durations (e.g., on-task behavior). The shorter the interval, the more accurate the estimate of the behavior; recommendation is 10-15 second intervals.  

Collecting Peer Comparison Data. Should select 3-4 peers for comparison. Should select peers of the same age, race, and gender. Should select peers who are not sitting near the target student (because the target student can disruptive nearby peers). 

Behavioral Observation of Students in Schools (BOSS): comparison peers are observed every fifth interval. Direct Observation Form: target students and peer are observed in alternating 10-minute sessions. Cooperative Learning Observation Code for Kids (CLOCK): target students are observed for 3 minutes and then a comparison peer is observed for 1 minute. See Appendices C.6 and C.7. 

How to Decide When to Observe Student. Give teacher scatterplot (C.8). Each row should represent no more than 30 minutes. Teacher places a mark in the box each time behavior is observed. If you want data on multiple target behaviors, use a different symbol for each -- e.g., +, O, X.

Decision Tree for Selecting or Creating an Observational Assessment System. Interview and/or Anecdotal Observation >> Operationalize Behavior >> If a Code exists, use it >> If a Code does not exist, create a New Code.

* * * * * 

The Standards for Educational and Psychological Testing outline what sound assessment practices look like. 

• Accuracy: the ability to draw a picture of a student that truly reflects what he/she is doing or knows. 
• Reliability: the consistency of a measurement. Test-retest reliability: If I administer a test 10 times, I will get roughly the same result each time. Inter-rater (or inter-observer) reliability: If different people use a measure to observe a student, they should both record roughly the same results. 
• Validity: the test is actually measuring the thing we intend to measure. 

* * * * * 

Classroom Observation Codes

Classroom Learning Assessment Scoring System (CLASS). Assesses classroom environment -- e.g., positive climate, teacher sensitivity, behavior management, quality of feedback. 

* * * * * 

Non-Classroom Observations

* * * * * 

Interpreting and Sharing Observation Results

Summarizing Anecdotal Observations. Avoid play-by-play descriptions. 

Summarizing Systematic Direct Observation Data. Try to use tables and/or graphs. 

Tables. Tables are useful when there are several quantitative values to report, but they are not necessarily being compared to one another. Example: a single observation conducted using a complex code like the BASC-3 SOS (Figure 10.1).

Selecting Chart or Graph:

• Line Graphs: most appropriate when depicting change over time. Assumption of line graphs is that data have been collected under similar conditions (e.g., same setting, same time of day). 
• Bar Graphs: best used for comparing data across qualitative categories. Example: on-task behavior in math (bar #1), language arts (bar #2), and social students (bar #3). Example: comparing one student to another. 
• Pie Charts: parts add up to 100. Example: calling out resulted in attention from peers 50% of the time, attention from teacher 30% of the time, etc. 

Analyzing and Interpreting Systematic Direction Observation Data. Two levels of analysis: interindividual analysis (what student's behavior looks like in comparison to peers) and intraindividual analysis (more helpful in information intervention steps). Intraindividual analysis: must consider whether behaviors were observed in multiple settings; if not, you shouldn't average. 

Gap Analysis. We can determine the extent of a problem by comparing the level at which a student is performing to an external standard (e.g.,, how a peer performs). Your goal is to make the gap smaller. 

Visual Analysis. This involves looking at a graph to determine whether the behavior has improved from baseline to intervention. Can look at level (represented by the mean value). Can look at tend or slope. 

Reports. Must provide enough detail so another individual could conduct the observation in the same manner. Must include the recording process used -- e.g., 

A partial interval time-sampling procedure was used to record instances of disruptive behavior. That is, if any instance of disruptive behavior were observed during each 15-second interval, the behavior was recorded as present. For the purpose of these observations, disruptive behavior was defined as any student action that...

Threats to validity: reactivity (ie., when the student's or teacher's behavior changes in the presence of an observer).