Counting feelings ... a Pirate Honor Code

A few points from the last class session really struck me so I want to memorialize them in this blog. (Or perhaps I am just procrastinating on my research proposal?)

First, we just happened to wander onto the Count's video and it worked great as an illustration of the importance of counting in meaning making. But, his song about counting feelings is also an apt reminder of the need to strike balance between quantitative and qualitative methods. It also aligned with my last blog about being a pirate and ninja, in equal measure. A “feeling” is an important indicator of a participant’s reality. All of the participant’s emotions and reactions should be identified and added up. It is also critical for the researcher to acknowledge and understand her own feelings as she stays attentive to bias and open to new meanings.

Second, this leads to the last few slides we were shown from the Class 9 deck on qualitative analysis. I’ve summarized the points here as a sort of “Pirate Honor Code”:

On my honor, as a Pirate, I will strive to be a good writer and fair observer. I will be respectful, reliable and trustworthy. I will always endeavor to remain open and honest about my biases, and to look for new meaning and patterns in every observation that I make.

The Count said it well… “Feeling all these feelings makes me feel exhausted!” On his advice, I am going to go catch 40 winks.

Pirate and ninja, in equal measure

Prior to the beginning of this course in January, I certainly did not appreciate the concepts of pirates and ninjas as I do now. I also certainly wouldn’t been as tickled as I was to have the following conversation with my 10-year old daughter yesterday:

K: “Mom, I have to find a ninja school to enroll in!”

Me: “But, K., ninjas aren’t seen or heard so you’ll have to be really quiet. Are you sure you’ll enjoy being a ninja?”

K: “I know they’re quiet but they also climb walls. I have to go to ninja school to learn to do both!”

Anticipating the end of this course and indeed the end of my graduate coursework, I would say that I have learned the value of being a pirate and a ninja, in equal measure. I have learned to be a stealthy and quiet ninja at times, and an inquisitive and reflective pirate at others. I have filled my tool belt with both quantitative and qualitative research methods, designs, techniques and procedures. I understand the difference between positing a hypothesis and identifying a foreshadowed problem and how to proceed when researching each one. I know now how to find giants and how to stand on their shoulders when they have already done the hard work for me. I also know how to tell when they really aren’t giants at all.

After two and a half years of exploring this curiosity and that whim as a way to pay some of the household and tuition bills, I have just accepted a full-time job back in my original profession of Human Resource Management. Since I am thinking constantly anyway about this new role, I’d like to use this blog post to reflect on how I can use this course’s lessons to improve my performance at this organization and how I can translate my new-found skills and abilities into success for the business.

Two weeks from now I will begin working as an HR manager for a relatively young organization which contracts with government agencies to provide hardware installation and support. The company has doubled in size in the past year to about 70 employees and anticipates needing to double again within six months. I will be HR person #2; hired to develop a competitive and sustainable employee benefits program as well as to build a full corporate curriculum covering customer service, organizational policies and procedures, leadership development, and performance management.

In the past, I have tended to approach new jobs and business relationships where I am expected to play an expert or advisory role, as a ninja. I naturally prefer to provide proof for my recommendations through credentials, past work samples, and statistical data. Now, I can add to this tendency, my new skills in finding quantitative research in peer-reviewed journals and in assessing the validity of studies, to develop my recommendations. As this company is in start-up mode, I foresee a specific need to research and clearly communicate study results to management, about best practices for emerging companies around benefit plan design (what types of coverage, plan designs, and employer/employee cost sharing percentages), management:staff ratios, and employee training among other things. I’ll also use these studies to encourage data-driven reflection by the leaders prior to developing and implementing our new structures and policies. I hope to also propose and test hypotheses about what will happen in this particular company with variables like expected enrollment levels, productivity standards, and transfer of learning rates from classroom to desktop. Even though I will not (purposely) undertake experimental research in my new role, I will still be able to quantitatively “study” the impacts of new programs, policies and procedures even without any formal control groups.

But I am most excited about altering my approach to be more like a pirate. I can consider this new job as an opportunity to conduct an ethnography. In the coming months and hopefully, for even longer, I will be able to live within this organization and culture, both as an observer and a participant. As I advise the company’s leadership about ways to develop and grow wisely, I will likely need to serve as a mirror in some cases and to be an interviewer in others. I anticipate formulating grounded theories over time about what works and doesn’t work, and what motivates or discourages. I also hope that there will be one or more employees, managers, or departments who will be great case studies in superior customer service, effective process improvement, innovation, or cross-border teamwork. I also look forward to using qualitative methods to develop a new performance management system. I know already to begin with dialogue and interviews about what performance is and should be and what the culture can possibly support in the way of tools, forms and procedures.

Regardless of the design, a constant respect for ethical behavior will be necessary. I will apply the required researcher behaviors that we identified via our grounded theory activity to my role. Obviously, I will need to ensure confidentiality for all employees and managers, as well as to manage the risks of any “research” I conduct. I will also ensure that when I leave the organization, the leaders, the employees and the business will be in a better place and a stronger position than where I find them.

Undoubtedly, a mixed-methods approach will best serve me and this organization. I look forward to being a pirate and ninja, in equal measure!

Skip to the End!

This is one of my favorite lines from my favorite movie of all time, The Princess Bride. Prince Humperdink was trying to make sure the “mahwage” ceremony was finalized before Buttercup was rescued by the Dread Pirate Roberts.

I was reminded of it when Professor Croasdaile used a similar line in response to my plea for help on the Introduction/literature review section of the proposal. Prior to class last Thursday, I had spent all of Wednesday and the better part of the prior weekend trying to get started on filling in my outline of major concepts and topics. This first section was clearly the biggest challenge of the project so far for me.

But, the advice was sound. When I sat down to get started again on Friday morning, I found that putting together the pieces and parts of the sample, procedure and measures was much easier than trying to first be eloquent about concepts and theories. I was able to address the nitty-gritty details (who/what/when/how) of the question I wanted to answer which in turn guided me back to a review of the literature that had piqued my interest in the first place.

After this rough start, I feel pretty good about the draft I’ve just turned in. I do hope to receive feedback on some additional concerns though. I worry about the measure that I’ve created in Appendices A & B. It is adapted from other measures that have been validated separately but never as a whole. How do I express the need to validate the instrument and how specific do I need to be about how that will be done?

I am unsure if I’ve transferred everything that is in my head down onto the paper. Have I fully defined all of the variables for the reader or have I left out points because I just naturally understand them after reading for the past two and a half months? Really, I am having trouble reading it objectively at this point.

My task ahead: let this sit for a while. Then, with a refreshed mind, I’ll hope to be able to process the feedback and finish this proposal in time for the April 26 deadline.

As a side comment, I think the advice to "Skip to the end" is valuable in other settings as well. Sometimes, giving ourselves permission to move on to the people, the details, the process, and the execution lends great clarity to the end goal and desire. Sometimes, not having every idea researched, source uncovered, and thought written out perfectly is the way forward.

Say phenomenological three times... fast!

Hypothetically speaking of course, if I were to decide to pursue a qualitative study of teacher organizational citizenship behaviors and pay-for-performance incentives, I could do either a phenomenological or an ethnographic study. The chosen approach would depend on whether I focused on just the identification and existence of extra-role behaviors and a shared cooperative culture of teachers (ethnography) or on their experiences as they relate and react to the implementation of an incentive pay system (phenomenological study ). Or perhaps I could accomplish both!

In the case of an ethnography, I would start with a foreshadowed problem generated from the active debate within existing research and theory about the existence and definition of teacher organizational citizenship behaviors. I might choose to focus my study on high school teachers and then enter into a relationship with a particular high school in a particular district for the express purpose of sampling, observing, obtaining and analyzing data about the behaviors outside of those required by formal teacher job descriptions. I would seek to identify a pattern from these observations and data about why, how, when and for whom teachers express OCB.

Backing up to review individual aspects of an ethnography, in that I cannot possibly study the extra-role behaviors of a comprehensive sample of high school teachers across the U.S., or even likely all teachers in a particular district, I would have to engage with a purposeful sample by case of teachers in one or two particular high schools. To protect validity and support any efforts at generalizability , I would need to identify all of the subject/selection threats that were inherent in the demographics of the district and school with which I chose to engage.

To obtain data, I would work as a moderate participant to record comprehensive observations, both structured and unstructured. I would also arrange to observe applicable committee and team meetings as well as ask questions about behaviors that I do observe. Interviews would also be very important to my study so that I could engage individual teachers in conversation about their behaviors and their own observations of others’ behavior. I foresee that document reviews would be valuable in that I could review the job descriptions and performance evaluations of the teachers to ensure that I could accurately differentiate what was in-role versus extra-role behavior and expectations.

A separate, additional phenomenological study might also be intriguing in that I could then build a further understanding of teacher OCB as it is influenced by the implementation or existence of incentive pay schemes. It would be very cool to query teachers through in-depth interviews or focus groups about their observations, reactions, and feelings (drama!) about pay-for-performance. If I was then able to interpret the data with any significant measure of reliability, I could hope to build new understanding about the use of incentive pay for teachers. Insight on how teachers experience pay –for-performance vis a vis the behaviors that they report being incentivized or discouraged would be very valuable feedback to have when considering the design of future teacher compensation systems.

Outside of this particular research problem relating to teacher behavior, I think a phenomenological study of pay-for-performance in any corporate environment could be extremely meaningful. There are a plethora of reports and studies that pay-for-performance is ineffective, broken, and not worthwhile but I don’t recall seeing any that try to derive meaning about pay-for-performance systems based on a qualitative investigation of participant experiences with it.

A sample of my literature review

Study #1: Extra-role behaviors: job satisfaction, efficacy and multiple dimensions

The first study was by researchers in Israel who looked at the components of extra-role behavior (ERB) and the correlational relationship between ERB and three variables: job satisfaction, self-efficacy, and collective efficacy. The researchers defined extra-role behavior for the purposes of their study as,
"those behaviors that go beyond specified role requirements, and
are directed towards the individual, the group, or the organization as a unit,
in order to promote organizational goals." (Somech & Zahavy, 2000, p.650)

The sampling frame was elementary school teachers in northern Israel. Questionnaires were distributed to 375 teachers from 13 different elementary schools, of which 251 were returned to yield a response rate of 67%.

To develop a measure of extra-role behavior, they conducted a semi-structured interview of five principals and 25 teachers who were asked to list behaviors, outside of formal role requirements, that teachers exhibit to benefit of the student, their team or the school unit. The 60 items produced from this process were validated by teachers who were involved in a management-training program. The list was reduced in this way and then through factor analysis, it was organized into three subscales of eight items each, which measured ERB toward the student, towards the team, and towards the organization.

To measure the variables of job satisfaction, self-efficacy, and collective efficacy, three additional scales based on prior research were adapted for use with teachers. All three of these measures had respondents indicating their agreement or disagreement with the test items using a five- or seven-point Likert scale.

To test the first hypothesis regarding the components of ERB, the researchers performed a factor analysis which yielded three separate factors of ERB. First, the Factor 1 dimension included behaviors that were geared towards helping other teachers and so was recognized as ERB towards team members. Factor 2 behaviors were generally identified as those that were aimed at improving conditions for the school as a unit and so was terms ERB towards the organization. Factor 3 consisted of behaviors intended to improve student performance or the quality of the teaching. It was termed ERB towards the student.

Correlation analysis was used to test the other hypotheses regarding the relationship between job satisfaction, self-efficacy, and collective efficacy and each of the three levels of ERB (team, organization, student). A positive and significant relationship was found between teacher satisfaction and ERB at all three levels. Self-efficacy and ERB towards the team and the organization were also found to be correlated but there was no relationship proven between self-efficacy and ERB towards the student. There was a positive and significant relationship found between collective efficacy and ERB towards the team, but no relationship between collective efficacy and ERB towards the organization or the student.

The results of the factor analysis support the discussion of ERB as multidimensional. The extent to which the three different variables were or were not correlated to the various levels of ERB further support this. All together, the research suggests that formal efforts to increase the levels of job satisfaction, self-efficacy or collective efficacy will have varying impacts on levels and types of extra-role behaviors.

Study #2 – Organizational Citizenship Behavior and Pay-for-Performance plans

This study addressed whether there was an unintended negative interaction between pay-for-performance plans designed to reward and encourage certain in-role behaviors and those behaviors that are not formally rewarded or controlled through job descriptions but which yield value to the organization. Also, the researchers looked to identify whether the degree of interest alignment between the employer and the employee moderated the impact of pay-for-performance plans on those organizational citizenship, or extra-role, behaviors.

The sampling frame was employees in certain benchmark jobs and their supervisors, from eight different utility companies across the United States. The supervisors of the benchmark jobs in each company were chosen first and then asked to randomly select up to 10 subordinates who were then surveyed about attitudes towards the job and the work organization (n = 660). The sample was further culled by matching to supervisor responses and confirmation of their participation in a pay-for–performance plan. The subsample remaining consisted of 146 employees.

The survey issued to participants included items from several pre-existing measures of organizational citizenship behaviors, value alignment, and procedural justice, as well as items developed purposely to measure the employee’s perception of the link between performance and pay.

A regression analysis allowed the researchers to determine the extent to which the effect of the pay-for-performance link on OCB was influenced by the employee’s value commitment. It was found that if the employees are not aligned with the organization, their level of OCB will be lower as they perceive a stronger link between their performance and their pay. Similarly, if the employee is committed to the organization, their OCB will increase as the pay:performance link is stronger. Procedural justice was found to be a significant predictor of OCB.

This research is valuable in indicating that pay-for-performance plans will not necessarily negatively affect the level of organizational citizenship behavior displayed by the employee. In fact, if the employee and organization are “mutually invested”, employees may actually increase their OCB in response to the organization’s commitment to reward performance with pay.

On the topic of validity, both studies described limitations of their research and results. First, the measures utilized in both were either specially designed for the purpose or adapted from different pre-existing measures. There was no significant discussion in either of the studies that the survey items were themselves validated. Second, although relationships between variables and OCB were evidenced, there was no testing of the directionality of the effects. Finally, both studies used only self-reports of behavior and attitude. Again, this means that the results indicate relationships but only from the perspective of one source type. Therefore, the results of both studies are not generalizable and so much less useful in pointing a clear and convincing way forward on specific policy or program changes that can be implemented to increase or protect OCB.

References:

Somech, A., & Drach-Zahavy, A. (2000). Understanding extra-role behavior in schools: The relationships between job satisfaction, sense of efficacy, and teachers’ extra-role behavior. Teaching and Teacher Education, 16(5-6), 649-659. DOI: 10.1016/S0742-051X(00)00012-3

Deckop, J. R., Mangel, R., & Cirka, C. C. (1999). Research notes. getting more than you pay for: Organizational citizenship behavior and pay-for-performance plans. Academy of Management Journal, 42(4), 420-428.

Managing validity

The concept of validity has been a lecture or exam topic in at least one class during each of my five semesters in graduate school. I have been asked to understand the validity of correlations between the Big Five personality traits with job performance, explain the validity of different tools used in recruiting and staffing and now finally, the validity of my own research. Over this time period, I had become familiar with, and even memorized, various definitions of validity and yet, I was still quite impressed with the explanation that was offered in our textbook. Dr. McMilllan provided an all of a sudden more relevant understanding for me, perhaps only partly because I knew that it was a concept worth 6% of our research proposal grade. On page 144, Dr. McMillan advises that validity is a “judgment of the appropriateness of a measure for the specific inferences or decisions that result from the scores generated by the measure.” For some reason, this explanation was much more about the “big picture of validity” than the other technical definitions I had learned previously. For me now, validity is as much about what I do or decide as a result of my findings as it is about how accurate the tools are that I use or the relationships that are observed. So as I develop this research aimed at observing the impact of merit pay incentives on teacher’s organizational citizenship behaviors, I want to carefully protect the instrument, the subjects, the treatment and the environment so that I am able to present inferences and suggest actions that are altogether appropriate and sound.

Although I am still searching for the best measure of organizational citizenship behaviors, I can foresee and will manage the threats to validity that the instrument itself can present. For instance, I know already that I cannot use a tool “off the shelf” that measures organizational citizenship behaviors (OCB) generally but instead, I require a measure that has been adjusted for the school setting. Luckily, there is a measurement model that has been developed for use with teachers but it includes additional scales that may not be relevant to the OCB discussion. So if I do make changes to this tool, the validity of the edited measure can only be confirmed if I reanalyze it’s “fit” to the theoretical model and to any other potentially valid measures.

In this study, I propose to measure and compare the levels of OCB exhibited by teachers who receive performance pay incentives to those who receive just a standard scheduled salary. In that the incentive compensation provided to the teachers is the “treatment”, I do not anticipate that the control group of teachers who receive only straight salary will inadvertently be “treated”. I do however need to stay extremely wary that both the control and the experimental group may be receiving a “treatment” via their environments, i.e. their departments, buildings, districts and communities, that impact their OCB levels. In this study then, the dependent variable is more at risk of being compromised by extraneous events, timelines and settings than the independent variable.

I sense that the greatest risk to validity comes in the course of selecting and maintaining study’s subjects themselves. As we all know, people are strange and generally act weirdly, especially when it comes to professional motivations and behaviors. That said, as much as I attempt to control for natural and cultural characteristics that affect OCB, I will undoubtedly fail to identify groups of subjects that are absolutely equal in their predispositions. Working in my favor though, previous studies of OCB have shown that individual characteristics like personality have limited associations with OCB and so it has been suggested that OCB is a result of cognitive and rational judgments and not emotional ones. If it were the reverse, I would need to even more carefully control for subject affectivity.

The effect of time and the announcement of the study itself on my subjects is another obvious threat to the validity of my study. Since it is not practical to announce the study and collect all the necessary data at one time, I can presume that my presence and the notice of the study will operate as an intervention in and of itself. The observation of OCB may already be compromised by changes in teacher’s actions in between the time the study is announced and when data is collected. As I finalize and summarize my literature review, I intend to keep my eye open for discussions about how this specific threat to validity can be minimized. I imagine it is a threat not only for my study but most others out there as well.

Even as I manage these threats to validity through careful design of the measure, documentation of the environment, and the “proper care and feeding” of my subjects, data analysis may still yield results that prevent any meaningful inferences or recommendations from being drawn. Although that would be very unfortunate, as Dr. McMillan would have us understand the concept of validity, recognition that a decision or inference should not be drawn from the study may still be a valid conclusion.

Beware the causes!

I have read two particular studies in the past few days, one for this class and another for a Guided Study in International Human Resources, in which the authors "point blank" warned against making assumptions about causality based on the data they collected. Both were designed to assess correlations between variables so I’ll first discuss the dangers of implied causation in correlational studies as they were highlighted by our textbook author. First, Dr. McMillan points out that even if a positive or negative or predictive relationship between two or more variables is proved, the causality of that relationship should not be assumed because a correlation is not designed to provide evidence about causal direction.

This particular problem was illustrated in an article about the factors that influence the work role adjustment of an expatriate manager in Japan. In his study, Black (1988) hypothesized that the level of family’s adjustment is correlated to the manager’s adjustment but he backed away from trying to assess causal direction because the family’s adjustment (or lack thereof) could be caused by the manager’s adjustment (or lack thereof) just as easily as the manager’s adjustment could be caused by the family’s adjustment.

The second study that mentioned causality as a limitation is one that I am reading for my research proposal, about the relationships between several individual and organizational characteristics and organizational citizenship behaviors (OCB) in schools. At the end of the article, the authors (Somech & Ron, 2007) suggest that since they used a cross-sectional survey for their study, they were not able to assess the extent of a causal relationship between the variables and the OCB that they otherwise were able to show were correlated.

The other warning that Dr. McMillan issues about inferring causation from correlational studies is more so a warning about taking actions based on observed correlations. Again, going back to the studies I’ve picked as examples, if one were to deploy 100% of available resources to address and improve the family’s adjustment to the international assignment when it is actually the manager’s issues that are causing the family’s disorder then the organization is misallocating its resources. And based on the OCB study, if it were determined that OCB precede the evidence of organizational characteristics, policies that focus resources on replicating the related organizational characteristics will be misaligned.

This assumption of causation is equally a problem in comparative studies. Even if a study shows that there are significant similarities or differences between groups when certain variables are assessed, it does not mean that the groups are similar or different because of the independent variable. To illustrate this, I’ll use an aspect of my own planned research proposal. If I were to execute a comparative study only and I determined that the OCB reported by teachers who receive merit pay incentives is lower than those teachers who do not receive merit pay, it would be very unwise to stop there with a conclusion that merit pay reduces teacher OCB. The lower OCB observation could very well be related to a different characteristic or variable that I have not even included in my study. My conclusion then should be to propose research that digs deeper into other causes or implications of OCB and how those might also interact with merit incentives.

Citations for the studies used as examples above are:

Black, J. S. (1988). Work role transitions: A study of American expatriate managers in Japan. Journal of International Business Studies, 19(2), pp. 277-294.

Somech, A., & Ifat, R. (2007). Promoting organizational citizenship behavior in schools: The impact of individual and organizational characteristics. Educational Administration Quarterly, 43(1), 38-66. doi:10.1177/0013161X06291254

Primum non nocere – First, do no harm.

I’ve wondered before about this adage because I naturally tend to assume that only a sociopath would ever enter a situation planning to harm anther being. Yet this protective principle is found front and center in the code of ethics of most professions, as it is in James McMillan’s “top ten” aspects of ethical concerns for educational researchers. I suggest that it is absolutely appropriate for professions and organizations to keep it in focus, not because it is a natural approach (at least for the psychologically stable) but because it is a universal principle whose operational definition is easily comprehended.

So what does it mean for us as educational researchers to 'do no harm'? To me, it means that we take utmost care to identify the implicit risks, the explicit risks and all of our stakeholders prior to conducting and designing research. Explicit risks are the obvious physical threats or psychological dangers that might be posed by our presence, by coercive techniques, by our withholding of important truths, or simply put, our negligence. The implicit risks are the “harm” that might be caused by our lack of planning or forethought, or our failure to make contingency plans. And beyond the need to manage risks, we have to exhaustively discover all of our stakeholders which I think is harder and takes longer than expected.

Using my research problem to illustrate this responsibility, I have to start first with the participants that I sample, namely the teachers who are compensated via performance incentives as well as those in the control group who are paid straight salary. In addition to the participants directly measured, I will require resources in the form of both data and time from their administrators, from their Human Resources support staff and from their district leadership. And beyond them, I can identify the other stakeholders as the students in their classrooms, their families, their communities, and the school budgets that pay them. All of these parties will be directly impacted by my research and certainly by any incremental change in performance or behavior that results from my presence or the conclusions that my research supports. Unless I take great care to identify all of the stakeholders involved in the system I intend to research, I cannot ever fully manage the risks of my methods.

Another ethical responsibility that McMillan mentions and about which I feel strongly is the role of the investigator in maintaining the confidentiality of the participants and the propriety of the study. To me, this means ensuring that data and records are protected and planning for all contingencies necessary to safeguard them, regardless of the financial and physical resources required. The researcher is also responsible for ensuring the work is all unique or is properly cited if not. As well, in the end, she must guarantee that the design, the data, the methods, the results and the conclusions are fully transparent and fully explained.

In this, I arrive full circle at the original, overriding guideline and to share a concluding operational definition. ‘To do no harm’ means to always leave the participants, their communities, and fellow researchers in a better place than where you found them. Even if my data reveals that performance pay does not impact the organizational citizenship behaviors of teachers, this conclusion can still in some way benefit those leaders who make educational policy on behalf of our children.

Sampling

My review of literature relating to merit pay in education is only a fraction complete. I’ve already taken a few wrong turns on my way to different grocery stores, all in the name of looking for the right fruit.

I am settling now on the concept of teacher organizational citizenship behavior (OCB) and how it is impacted by merit pay compensation. I intend to look further at research already available regarding the components, causes and moderators of OCB, in teachers and other professionals. I also will review the efficacy of using existing instruments to measure OCB in teachers. At this point though, I hypothesize that merit pay compensation programs will moderate the OCB of public high-school teachers in the United States.

I believe that the interaction between merit pay and teacher OCB will eventually be best explored through a mixed-methods analysis taking both a quantitative and qualitative approach to the problem. However, I propose to start with a quantitative study that compares the level of OCB in teachers who are compensated via a merit pay system to teachers who are compensated via a fixed salary schedule. I will discuss and debate the various sampling procedures here as if I am undertaking that particular research. In this case, my sampling frame will need to be much narrower than my target population of all public U.S. high school teachers because I must account for other environmental and organizational culture factors that may moderate OCB in the experimental and the control groups. Also, specifically for the experimental group, I will need to evaluate the design of the compensation plan to determine whether any extra-role behaviors actually become in-role behaviors according to the performance targets established by the merit pay plan.

The sampling procedure that I use will depend largely on the availability and size of a sampling frame to which I can gain access. If I can find a sizable school district that has a large enough population of high school teachers who are compensated via merit pay as well as a population of teachers in the same schools who are not being compensated with performance incentives, I may be able to utilize some probability sampling procedures. Assuming such a survey population is available, I could use a simple random or systematic procedure to randomly select subjects for the sample. The risk in taking these approaches is that I may not get a representative sample, i.e. male and female teachers or teachers with varying experience or education, which will bias the measurement OCB and thus, my results. To solve for this, again assuming that the sample frame is large enough, I would first identify and divide the population into different subgroups related to the dependent variable – OCB. For example, I would identify subgroups by gender, years of education and/or experience, subject taught, school taught in, etc. for each of the experimental and control populations. I could then take a proportional stratified sample from these samples or if I don’t have enough participants represented in each subgroup, I could take a disproportional stratified sample and then weight the results to reflect the whole population.

Using a cluster sample from within the district might be more appropriate, that is I could find one particular high school within the district that has teachers who are compensated with merit pay and those who are not. Although limiting the sample frame in this way may reduce the validity of the results, it may help to control for some of the environmental and organizational culture factors that moderate OCB.

In the case that I am not able to gain access to a school district large enough to support probability sampling, I will need to rely on nonprobability sampling. Using a convenience sample procedure, I would be identify a particular high school which has instituted a merit pay plan for some but not all of its teachers and then measure and analyze the OCB of teachers in that particular school. In the unexpected case that I find one such high school that meets these criteria yet is also big enough to afford me the chance to sample certain subgroups, I could effect a quota sample and select participants who represent the subgroups mentioned earlier, like gender and teaching experience.

If I were to pursue a mixed-methods analysis of merit pay and teacher OCB, I could proceed to use some purposeful sampling procedures like typical or extreme case sampling. In each of these, I would define and identify the “average” (for typical case) or “atypical” (for extreme case) high school teacher from a particular school division or high school and measure the OCB exhibited by those who are and are not compensated with merit pay. If I find both the “average” and the “atypical” teacher, I can effect a maximum variation sample.

In the case of this research proposal, a snowball sample seems inappropriate. I believe that there would be too many different cultural and environmental factors at play in such a network sample which would render the OCB measurements and conclusions invalid. In the same way though, I think that to control for these same factors, an element of critical case sampling will be valuable in this study. That is, I think the study will ultimately need to identify one particular school district or even one particular high school from which an experimental and a control group of teachers can be drawn.

If I use nonprobability sampling procedures and an element of critical case sampling as currently planned, I know that I won’t be able to generalize any discoveries about the relationship between teacher OCB and merit pay to the larger US high school teacher population. However, I still think that the results will be significant for the school district that is sampled and that the data obtained will inform us about how merit pay and teacher OCB interact. At the least, it will suggest complementary future studies.

Way Too Much Fruit

I am drowning in fruit now... aargghh!

Making Fruit Salad

As my 7-year old daughter was getting ready for school this morning, she asked me what I was doing today since Fridays are known to be one of Mommy’s important study days. I told her that I needed to write about fruit salad for a class. I knew this revelation would totally turn her on because she has been the self-appointed Iron Chef of fruit salads in our house for a number of years. So, Lindsay wanted to make sure I reported that “fruit salads are good and that if you use frozen fruit, they are cold also”!

Typology of fruit salad

When I reflect about the fruit salads we make at our house, I can actually categorize them in two different ways. First, there is the fruit salad that is thrown together when there is a ripe banana on the counter and you realize that there also happens to be more than one other kind of fruit in the fruit bowl that day. For the record, the critical threshold for a spur-of-the-moment fruit salad is three different types of fruit. Second, there is the fruit salad that you actually list on your weekly menu or for which you purposely shop because you are making dinner for guests or because you signed up to bring fruit salad for a potluck dinner. In the case of the second type, the ingredients are usually more numerous and exotic. I suggest that at least five different types of fruit are critical and that raspberries, kiwis or pineapple must be considered for fruit salads meant to be consumed by others. As for a juice or syrup to toss with the fruit, for both types of salads, the fruit:sauce ratio is very important since it is always more appealing when the fruits you eat are recognizable.

So what does a fruit salad have to do with a literature review?

But, how does this brief personal reflection about fruit salad add value to my goal to describe a literature review? As suggested in class, the fruit salad and its preparation, is a metaphor for the criticality of knowing why and for whom you are reviewing literature. Just as the quantity, type, and quality of fruit and the intricacies of any sauce depend on whether you happen into or plan in advance for making fruit salad, the quantity, type, quality, and “sauce” of your literature review will depend on your research problem, your audience, and the context. And just as how the taste of the dish is dependent on how you choose, prepare and put the fruit together, so does the significance and quality of your research depend on your efforts to find and analyze your sources.

Are these bananas appropriate for a fruit salad or are they best baked into a quick bread for breakfast?

Sometimes, before you even start your literature review, you have “fruit” to sort. No sooner had I selected the topic of merit pay in K-12 education for this class project than I started recognizing the topic everywhere. I read about it in popular magazine articles and newspapers, and I listened to it in a Planet Money podcast. I also happened to mention the topic to my Labor Economics professor and sure enough, she started sending me abstracts from economics journals. My point is that even though these resources were not found in the course of my formal, step-by-step literature review, they are fruit to sort through nonetheless. And importantly, even if they don’t end up as references in my final project, they have suggested other ingredients to consider. For example, the economist interviewed by the Planet Money team has published a study on the economic value of good teachers versus bad teachers – I can look for his name and his study when I shop in peer-reviewed journals. And via the suggestions sent by my professor, I can find other studies as well as journals that have reported on the topic. One of the abstracts she sent appears to be written as a book chapter which suggests that it is a secondary source and may be good map of other literature already out there.

So, I know that fruit exists but I still have to go shopping!

I despise going to the grocery store without a list. My list keeps me focused on the menu at hand; so that I buy what I need and I don’t end up with the stuff that just looks yummy. For this literature review, my “list” contains my main topic, merit pay in K-12 education, and related terms such as pay for performance, incentive pay, teacher morale, extra-role behaviors, and teacher performance management.

Before I get in my car to go shopping, I first check to make sure my grocery bags are in the trunk. That is to say, I need to make sure I have something to bring my fruit home in once I buy it. So I create an Excel spreadsheet to track author name, article/chapter title, and the title of the journal/book. I also decide to set up a way to keep track of my purchases in case I need to tell my dinner guests where I found such delicious fruit later on. So, I also take some time to recall my RefWorks password and I create a new folder there called EDUS 660.

With my bags, list and car keys in hand, I stop to realize that I still don’t know where I am going. I need to find out which grocery store is going to have the type and quality of fruit that I want. To accomplish this for the purposes of my literature review, I identify and save a set of databases to begin my search with and then finally, I am off to shop. Even with my list, I foresee the temptation to buy other items that are strategically placed near the fruit I am looking for. For example, if “teacher engagement” wasn’t on my original list but I notice that it is a keyword in the first sources I find, I’ll probably add it to the list. On the other hand, if the serpent is hocking the caramel dip next to the apples or even just distracting exposes of No Child Left Behind legislation, I will stay focused! I must remain in “Fruit & Produce”. I dare not wander to “Frozen Foods”!

Just as I do when I shop using a list, I may make substitutions based on what is ripe or in season. For example, I may realize that one of my keywords is consistently directing me to non-peer reviewed journals or popular magazines. I’ll substitute different terms then. Also, within the records that are retrieved based on keywords or topics, I will look through the selections and pick those that appear most relevant to my topic. Before heading to the checkout line, I will also make sure that I buy enough to feed everyone at the table and that the fruit I’ve selected isn’t too old or too mushy. Similarly, my literature review must be comprehensive enough but it must not include studies that are out of date or irrelevant based on current circumstances.

Is it a Berry Medley or a Tropical Fruit Salad?

Until I get home with my groceries and start cutting up and tasting the fruit, I might not know what type of salad will actually be served at dinner. In this same way, I look forward to what my literature review will produce and how my research proposal may change based on the sources I find. Secondary sources may turn me on to a whole group of studies that make my intended proposal less relevant. I may find primary sources which suggest an area for further research that is more significant than the one I originally identified. As I organize, summarize and relate the studies to each other, I could also end up with a whole new framework for the research that I propose. And then, who knows but I may even need to run back to the store for more fruit!

Merit pay in education: quantitative and qualitative research

For the purpose of this semester’s work in educational research, I will concentrate on learning more about merit pay in education. For me, this topic was found at the intersection of personal interest and professional curiosity. From the personal side, my husband is a high school physics and chemistry teacher, a career that he left an MBA and six-figure marketing salary behind to pursue. He is passionate about teaching science and equally so about the inefficiencies of the administration of teacher pay and performance. As a human resource professional, I have experienced the impact and intricacies of merit pay and performance management programs, albeit in the corporate world. I want to look closely at “research” that has already been done on the topic and identify a problem area where additional data may shed light on the validity of designing and implementing pay for performance models in our nation’s schools.

Based on experience and education, I know there are many social, economic and political forces at play in performance management and merit pay systems in the corporate environment. There is constant debate about what performance is and how to measure it adequately and objectively. Based on an initial online review, these same arguments surround the topic when it is discussed in the field of K-12 education. Identifying some of the forces that impact the effectiveness of pay for performance systems was a “jumping off point” for the identification of quantitative and qualitative research problems.

Quantitative research problems identify, specifically define and objectively observe the interactions between variables to confirm or deny expected relationships. An example of a quantitative research question in the area of merit pay in education is “Do high school students who are taught by teachers receiving base salary plus performance incentives perform better on end-of-course evaluations than students who are taught by teachers receiving base-salary only?”. In order to identify the relationship between performance and incentive pay in such a quantitative study, it will be critical to control the variables and structure of the research. For example, the two compensation models will need to be rigorously defined and implemented for chosen subjects and methods to measure end-of-course performance will need to be established.

Qualitative research problems provide an opportunity to observe what participants naturally do in reaction to psychosocial influences or changes in the physical environment . An example of a qualitative research question in the area of merit pay in education is, “Do teachers who participate in merit pay programs experience increased morale or engagement behaviors?”. In the context of this study, a researcher might interact with participating teachers over a period of time to observe, understand and identify the effects of performance pay programs on their professional activities and attitudes about teaching.

My next challenge will be to navigate the very active debate about what constitutes performance measurement and the value of incentive pay in the education field. As I pursue this research topic, especially in undertaking the literature review, I intend to carefully identify the underlying policies, methods, definitions and models that are reported in order to identify a pressing need that my research can fulfill.

Dedication to EDUS 660: "...sometimes you have to make stuff up..."

Arriving home from a great first class last night, my husband and I settled down to watch one of our favorite TV shows, The Daily Show with Jon Stewart. Since we are also fans of PBS's NOVA, we were excited to replay Tuesday's episode in which Jon Stewart welcomed back one of his regular guests, Neil geGrasse Tyson, who is an astrophysicist and host of the new NOVA scienceNow program on PBS.

Of course, my ears perked up when I heard Neil deGrasse Tyson say the following:

"When you are on the frontier of knowledge, between what is known and unknown, reaching out into that abyss, sometimes you do actually have to make stuff up that might be true so that you can organize a research plan to find out whether or not it is."

Cheers to a great semester and to the stuff that we might have to make up!

"AND when I grow up, I want to be ...": an integrative appraoch to your career path!

Before I dedicate this blog to EDUS 660 for a semester-long reflection on educational research, I want to memorialize some thoughts that I've been tossing around in my brain over the winter break.

The concept first struck me when I read Harvard Business Review blog posts by Roger Martin issued on 11/24/2010 and 12/10/2010. Mr. Martin discusses contrasts the logic, or lack thereof, of IF/THEN/BUT and IF/THEN/AND structures. He applies them to several corporate and artistic ventures where one can see the benefit of integrative thinking which produces something from nothing or at least something from that which wasn't expected or thought possible. I'll recommend Mr. Martin's posts for further and better description of his personal insights.

His reflections led me to think about the application of "BUT" or "AND" to one's career plan. When planning job changes or professional moves, isn't it freeing to say "AND what I want to do next is..." instead of, "BUT I need to make more money or have a grander title or expand the number of people or products that report to me?" This approach allows you to see the value of what you have done or been and apply it as you write the next chapter of your career path.

I just had occasion to catch up with an acquaintance from the local SHRM chapter and he described a new job he'd taken as what he had wanted to do "when he grew up." I am so happy that he found this opportunity. I also want to encourage him that this is an "AND" in his path; perhaps it is not the only time he can "grow up" as a professional.

Over the past few years, as I have taken time to step away from 60 hour workweeks in corporate HR roles, my "ANDs" have been graduate education, more hands-on motherhood, preschool teaching, as well as PTA leadership. I am looking forward to my next adventure as I return to work in the HR field on a full-time basis following graduation. I treat this upcoming adventure as an "AND" which adds value to what I have already been as mother, student, worker, wife. I look forward to what each of those accomplishments lead me to be able to do when I "grow up."