Beware the causes!

I have read two particular studies in the past few days, one for this class and another for a Guided Study in International Human Resources, in which the authors "point blank" warned against making assumptions about causality based on the data they collected. Both were designed to assess correlations between variables so I’ll first discuss the dangers of implied causation in correlational studies as they were highlighted by our textbook author. First, Dr. McMillan points out that even if a positive or negative or predictive relationship between two or more variables is proved, the causality of that relationship should not be assumed because a correlation is not designed to provide evidence about causal direction.

This particular problem was illustrated in an article about the factors that influence the work role adjustment of an expatriate manager in Japan. In his study, Black (1988) hypothesized that the level of family’s adjustment is correlated to the manager’s adjustment but he backed away from trying to assess causal direction because the family’s adjustment (or lack thereof) could be caused by the manager’s adjustment (or lack thereof) just as easily as the manager’s adjustment could be caused by the family’s adjustment.

The second study that mentioned causality as a limitation is one that I am reading for my research proposal, about the relationships between several individual and organizational characteristics and organizational citizenship behaviors (OCB) in schools. At the end of the article, the authors (Somech & Ron, 2007) suggest that since they used a cross-sectional survey for their study, they were not able to assess the extent of a causal relationship between the variables and the OCB that they otherwise were able to show were correlated.

The other warning that Dr. McMillan issues about inferring causation from correlational studies is more so a warning about taking actions based on observed correlations. Again, going back to the studies I’ve picked as examples, if one were to deploy 100% of available resources to address and improve the family’s adjustment to the international assignment when it is actually the manager’s issues that are causing the family’s disorder then the organization is misallocating its resources. And based on the OCB study, if it were determined that OCB precede the evidence of organizational characteristics, policies that focus resources on replicating the related organizational characteristics will be misaligned.

This assumption of causation is equally a problem in comparative studies. Even if a study shows that there are significant similarities or differences between groups when certain variables are assessed, it does not mean that the groups are similar or different because of the independent variable. To illustrate this, I’ll use an aspect of my own planned research proposal. If I were to execute a comparative study only and I determined that the OCB reported by teachers who receive merit pay incentives is lower than those teachers who do not receive merit pay, it would be very unwise to stop there with a conclusion that merit pay reduces teacher OCB. The lower OCB observation could very well be related to a different characteristic or variable that I have not even included in my study. My conclusion then should be to propose research that digs deeper into other causes or implications of OCB and how those might also interact with merit incentives.

Citations for the studies used as examples above are:

Black, J. S. (1988). Work role transitions: A study of American expatriate managers in Japan. Journal of International Business Studies, 19(2), pp. 277-294.

Somech, A., & Ifat, R. (2007). Promoting organizational citizenship behavior in schools: The impact of individual and organizational characteristics. Educational Administration Quarterly, 43(1), 38-66. doi:10.1177/0013161X06291254

Primum non nocere – First, do no harm.

I’ve wondered before about this adage because I naturally tend to assume that only a sociopath would ever enter a situation planning to harm anther being. Yet this protective principle is found front and center in the code of ethics of most professions, as it is in James McMillan’s “top ten” aspects of ethical concerns for educational researchers. I suggest that it is absolutely appropriate for professions and organizations to keep it in focus, not because it is a natural approach (at least for the psychologically stable) but because it is a universal principle whose operational definition is easily comprehended.

So what does it mean for us as educational researchers to 'do no harm'? To me, it means that we take utmost care to identify the implicit risks, the explicit risks and all of our stakeholders prior to conducting and designing research. Explicit risks are the obvious physical threats or psychological dangers that might be posed by our presence, by coercive techniques, by our withholding of important truths, or simply put, our negligence. The implicit risks are the “harm” that might be caused by our lack of planning or forethought, or our failure to make contingency plans. And beyond the need to manage risks, we have to exhaustively discover all of our stakeholders which I think is harder and takes longer than expected.

Using my research problem to illustrate this responsibility, I have to start first with the participants that I sample, namely the teachers who are compensated via performance incentives as well as those in the control group who are paid straight salary. In addition to the participants directly measured, I will require resources in the form of both data and time from their administrators, from their Human Resources support staff and from their district leadership. And beyond them, I can identify the other stakeholders as the students in their classrooms, their families, their communities, and the school budgets that pay them. All of these parties will be directly impacted by my research and certainly by any incremental change in performance or behavior that results from my presence or the conclusions that my research supports. Unless I take great care to identify all of the stakeholders involved in the system I intend to research, I cannot ever fully manage the risks of my methods.

Another ethical responsibility that McMillan mentions and about which I feel strongly is the role of the investigator in maintaining the confidentiality of the participants and the propriety of the study. To me, this means ensuring that data and records are protected and planning for all contingencies necessary to safeguard them, regardless of the financial and physical resources required. The researcher is also responsible for ensuring the work is all unique or is properly cited if not. As well, in the end, she must guarantee that the design, the data, the methods, the results and the conclusions are fully transparent and fully explained.

In this, I arrive full circle at the original, overriding guideline and to share a concluding operational definition. ‘To do no harm’ means to always leave the participants, their communities, and fellow researchers in a better place than where you found them. Even if my data reveals that performance pay does not impact the organizational citizenship behaviors of teachers, this conclusion can still in some way benefit those leaders who make educational policy on behalf of our children.

Sampling

My review of literature relating to merit pay in education is only a fraction complete. I’ve already taken a few wrong turns on my way to different grocery stores, all in the name of looking for the right fruit.

I am settling now on the concept of teacher organizational citizenship behavior (OCB) and how it is impacted by merit pay compensation. I intend to look further at research already available regarding the components, causes and moderators of OCB, in teachers and other professionals. I also will review the efficacy of using existing instruments to measure OCB in teachers. At this point though, I hypothesize that merit pay compensation programs will moderate the OCB of public high-school teachers in the United States.

I believe that the interaction between merit pay and teacher OCB will eventually be best explored through a mixed-methods analysis taking both a quantitative and qualitative approach to the problem. However, I propose to start with a quantitative study that compares the level of OCB in teachers who are compensated via a merit pay system to teachers who are compensated via a fixed salary schedule. I will discuss and debate the various sampling procedures here as if I am undertaking that particular research. In this case, my sampling frame will need to be much narrower than my target population of all public U.S. high school teachers because I must account for other environmental and organizational culture factors that may moderate OCB in the experimental and the control groups. Also, specifically for the experimental group, I will need to evaluate the design of the compensation plan to determine whether any extra-role behaviors actually become in-role behaviors according to the performance targets established by the merit pay plan.

The sampling procedure that I use will depend largely on the availability and size of a sampling frame to which I can gain access. If I can find a sizable school district that has a large enough population of high school teachers who are compensated via merit pay as well as a population of teachers in the same schools who are not being compensated with performance incentives, I may be able to utilize some probability sampling procedures. Assuming such a survey population is available, I could use a simple random or systematic procedure to randomly select subjects for the sample. The risk in taking these approaches is that I may not get a representative sample, i.e. male and female teachers or teachers with varying experience or education, which will bias the measurement OCB and thus, my results. To solve for this, again assuming that the sample frame is large enough, I would first identify and divide the population into different subgroups related to the dependent variable – OCB. For example, I would identify subgroups by gender, years of education and/or experience, subject taught, school taught in, etc. for each of the experimental and control populations. I could then take a proportional stratified sample from these samples or if I don’t have enough participants represented in each subgroup, I could take a disproportional stratified sample and then weight the results to reflect the whole population.

Using a cluster sample from within the district might be more appropriate, that is I could find one particular high school within the district that has teachers who are compensated with merit pay and those who are not. Although limiting the sample frame in this way may reduce the validity of the results, it may help to control for some of the environmental and organizational culture factors that moderate OCB.

In the case that I am not able to gain access to a school district large enough to support probability sampling, I will need to rely on nonprobability sampling. Using a convenience sample procedure, I would be identify a particular high school which has instituted a merit pay plan for some but not all of its teachers and then measure and analyze the OCB of teachers in that particular school. In the unexpected case that I find one such high school that meets these criteria yet is also big enough to afford me the chance to sample certain subgroups, I could effect a quota sample and select participants who represent the subgroups mentioned earlier, like gender and teaching experience.

If I were to pursue a mixed-methods analysis of merit pay and teacher OCB, I could proceed to use some purposeful sampling procedures like typical or extreme case sampling. In each of these, I would define and identify the “average” (for typical case) or “atypical” (for extreme case) high school teacher from a particular school division or high school and measure the OCB exhibited by those who are and are not compensated with merit pay. If I find both the “average” and the “atypical” teacher, I can effect a maximum variation sample.

In the case of this research proposal, a snowball sample seems inappropriate. I believe that there would be too many different cultural and environmental factors at play in such a network sample which would render the OCB measurements and conclusions invalid. In the same way though, I think that to control for these same factors, an element of critical case sampling will be valuable in this study. That is, I think the study will ultimately need to identify one particular school district or even one particular high school from which an experimental and a control group of teachers can be drawn.

If I use nonprobability sampling procedures and an element of critical case sampling as currently planned, I know that I won’t be able to generalize any discoveries about the relationship between teacher OCB and merit pay to the larger US high school teacher population. However, I still think that the results will be significant for the school district that is sampled and that the data obtained will inform us about how merit pay and teacher OCB interact. At the least, it will suggest complementary future studies.

Way Too Much Fruit

I am drowning in fruit now... aargghh!

Making Fruit Salad

As my 7-year old daughter was getting ready for school this morning, she asked me what I was doing today since Fridays are known to be one of Mommy’s important study days. I told her that I needed to write about fruit salad for a class. I knew this revelation would totally turn her on because she has been the self-appointed Iron Chef of fruit salads in our house for a number of years. So, Lindsay wanted to make sure I reported that “fruit salads are good and that if you use frozen fruit, they are cold also”!

Typology of fruit salad

When I reflect about the fruit salads we make at our house, I can actually categorize them in two different ways. First, there is the fruit salad that is thrown together when there is a ripe banana on the counter and you realize that there also happens to be more than one other kind of fruit in the fruit bowl that day. For the record, the critical threshold for a spur-of-the-moment fruit salad is three different types of fruit. Second, there is the fruit salad that you actually list on your weekly menu or for which you purposely shop because you are making dinner for guests or because you signed up to bring fruit salad for a potluck dinner. In the case of the second type, the ingredients are usually more numerous and exotic. I suggest that at least five different types of fruit are critical and that raspberries, kiwis or pineapple must be considered for fruit salads meant to be consumed by others. As for a juice or syrup to toss with the fruit, for both types of salads, the fruit:sauce ratio is very important since it is always more appealing when the fruits you eat are recognizable.

So what does a fruit salad have to do with a literature review?

But, how does this brief personal reflection about fruit salad add value to my goal to describe a literature review? As suggested in class, the fruit salad and its preparation, is a metaphor for the criticality of knowing why and for whom you are reviewing literature. Just as the quantity, type, and quality of fruit and the intricacies of any sauce depend on whether you happen into or plan in advance for making fruit salad, the quantity, type, quality, and “sauce” of your literature review will depend on your research problem, your audience, and the context. And just as how the taste of the dish is dependent on how you choose, prepare and put the fruit together, so does the significance and quality of your research depend on your efforts to find and analyze your sources.

Are these bananas appropriate for a fruit salad or are they best baked into a quick bread for breakfast?

Sometimes, before you even start your literature review, you have “fruit” to sort. No sooner had I selected the topic of merit pay in K-12 education for this class project than I started recognizing the topic everywhere. I read about it in popular magazine articles and newspapers, and I listened to it in a Planet Money podcast. I also happened to mention the topic to my Labor Economics professor and sure enough, she started sending me abstracts from economics journals. My point is that even though these resources were not found in the course of my formal, step-by-step literature review, they are fruit to sort through nonetheless. And importantly, even if they don’t end up as references in my final project, they have suggested other ingredients to consider. For example, the economist interviewed by the Planet Money team has published a study on the economic value of good teachers versus bad teachers – I can look for his name and his study when I shop in peer-reviewed journals. And via the suggestions sent by my professor, I can find other studies as well as journals that have reported on the topic. One of the abstracts she sent appears to be written as a book chapter which suggests that it is a secondary source and may be good map of other literature already out there.

So, I know that fruit exists but I still have to go shopping!

I despise going to the grocery store without a list. My list keeps me focused on the menu at hand; so that I buy what I need and I don’t end up with the stuff that just looks yummy. For this literature review, my “list” contains my main topic, merit pay in K-12 education, and related terms such as pay for performance, incentive pay, teacher morale, extra-role behaviors, and teacher performance management.

Before I get in my car to go shopping, I first check to make sure my grocery bags are in the trunk. That is to say, I need to make sure I have something to bring my fruit home in once I buy it. So I create an Excel spreadsheet to track author name, article/chapter title, and the title of the journal/book. I also decide to set up a way to keep track of my purchases in case I need to tell my dinner guests where I found such delicious fruit later on. So, I also take some time to recall my RefWorks password and I create a new folder there called EDUS 660.

With my bags, list and car keys in hand, I stop to realize that I still don’t know where I am going. I need to find out which grocery store is going to have the type and quality of fruit that I want. To accomplish this for the purposes of my literature review, I identify and save a set of databases to begin my search with and then finally, I am off to shop. Even with my list, I foresee the temptation to buy other items that are strategically placed near the fruit I am looking for. For example, if “teacher engagement” wasn’t on my original list but I notice that it is a keyword in the first sources I find, I’ll probably add it to the list. On the other hand, if the serpent is hocking the caramel dip next to the apples or even just distracting exposes of No Child Left Behind legislation, I will stay focused! I must remain in “Fruit & Produce”. I dare not wander to “Frozen Foods”!

Just as I do when I shop using a list, I may make substitutions based on what is ripe or in season. For example, I may realize that one of my keywords is consistently directing me to non-peer reviewed journals or popular magazines. I’ll substitute different terms then. Also, within the records that are retrieved based on keywords or topics, I will look through the selections and pick those that appear most relevant to my topic. Before heading to the checkout line, I will also make sure that I buy enough to feed everyone at the table and that the fruit I’ve selected isn’t too old or too mushy. Similarly, my literature review must be comprehensive enough but it must not include studies that are out of date or irrelevant based on current circumstances.

Is it a Berry Medley or a Tropical Fruit Salad?

Until I get home with my groceries and start cutting up and tasting the fruit, I might not know what type of salad will actually be served at dinner. In this same way, I look forward to what my literature review will produce and how my research proposal may change based on the sources I find. Secondary sources may turn me on to a whole group of studies that make my intended proposal less relevant. I may find primary sources which suggest an area for further research that is more significant than the one I originally identified. As I organize, summarize and relate the studies to each other, I could also end up with a whole new framework for the research that I propose. And then, who knows but I may even need to run back to the store for more fruit!