Hypothetically speaking of course, if I were to decide to pursue a qualitative study of teacher organizational citizenship behaviors and pay-for-performance incentives, I could do either a phenomenological or an ethnographic study. The chosen approach would depend on whether I focused on just the identification and existence of extra-role behaviors and a shared cooperative culture of teachers (ethnography) or on their experiences as they relate and react to the implementation of an incentive pay system (phenomenological study ). Or perhaps I could accomplish both!
In the case of an ethnography, I would start with a foreshadowed problem generated from the active debate within existing research and theory about the existence and definition of teacher organizational citizenship behaviors. I might choose to focus my study on high school teachers and then enter into a relationship with a particular high school in a particular district for the express purpose of sampling, observing, obtaining and analyzing data about the behaviors outside of those required by formal teacher job descriptions. I would seek to identify a pattern from these observations and data about why, how, when and for whom teachers express OCB.
Backing up to review individual aspects of an ethnography, in that I cannot possibly study the extra-role behaviors of a comprehensive sample of high school teachers across the U.S., or even likely all teachers in a particular district, I would have to engage with a purposeful sample by case of teachers in one or two particular high schools. To protect validity and support any efforts at generalizability , I would need to identify all of the subject/selection threats that were inherent in the demographics of the district and school with which I chose to engage.
To obtain data, I would work as a moderate participant to record comprehensive observations, both structured and unstructured. I would also arrange to observe applicable committee and team meetings as well as ask questions about behaviors that I do observe. Interviews would also be very important to my study so that I could engage individual teachers in conversation about their behaviors and their own observations of others’ behavior. I foresee that document reviews would be valuable in that I could review the job descriptions and performance evaluations of the teachers to ensure that I could accurately differentiate what was in-role versus extra-role behavior and expectations.
A separate, additional phenomenological study might also be intriguing in that I could then build a further understanding of teacher OCB as it is influenced by the implementation or existence of incentive pay schemes. It would be very cool to query teachers through in-depth interviews or focus groups about their observations, reactions, and feelings (drama!) about pay-for-performance. If I was then able to interpret the data with any significant measure of reliability, I could hope to build new understanding about the use of incentive pay for teachers. Insight on how teachers experience pay –for-performance vis a vis the behaviors that they report being incentivized or discouraged would be very valuable feedback to have when considering the design of future teacher compensation systems.
Outside of this particular research problem relating to teacher behavior, I think a phenomenological study of pay-for-performance in any corporate environment could be extremely meaningful. There are a plethora of reports and studies that pay-for-performance is ineffective, broken, and not worthwhile but I don’t recall seeing any that try to derive meaning about pay-for-performance systems based on a qualitative investigation of participant experiences with it.
Friday, March 25, 2011
Friday, March 18, 2011
A sample of my literature review
Study #1: Extra-role behaviors: job satisfaction, efficacy and multiple dimensions
The first study was by researchers in Israel who looked at the components of extra-role behavior (ERB) and the correlational relationship between ERB and three variables: job satisfaction, self-efficacy, and collective efficacy. The researchers defined extra-role behavior for the purposes of their study as,
"those behaviors that go beyond specified role requirements, and
are directed towards the individual, the group, or the organization as a unit,
in order to promote organizational goals." (Somech & Zahavy, 2000, p.650)
The sampling frame was elementary school teachers in northern Israel. Questionnaires were distributed to 375 teachers from 13 different elementary schools, of which 251 were returned to yield a response rate of 67%.
To develop a measure of extra-role behavior, they conducted a semi-structured interview of five principals and 25 teachers who were asked to list behaviors, outside of formal role requirements, that teachers exhibit to benefit of the student, their team or the school unit. The 60 items produced from this process were validated by teachers who were involved in a management-training program. The list was reduced in this way and then through factor analysis, it was organized into three subscales of eight items each, which measured ERB toward the student, towards the team, and towards the organization.
To measure the variables of job satisfaction, self-efficacy, and collective efficacy, three additional scales based on prior research were adapted for use with teachers. All three of these measures had respondents indicating their agreement or disagreement with the test items using a five- or seven-point Likert scale.
To test the first hypothesis regarding the components of ERB, the researchers performed a factor analysis which yielded three separate factors of ERB. First, the Factor 1 dimension included behaviors that were geared towards helping other teachers and so was recognized as ERB towards team members. Factor 2 behaviors were generally identified as those that were aimed at improving conditions for the school as a unit and so was terms ERB towards the organization. Factor 3 consisted of behaviors intended to improve student performance or the quality of the teaching. It was termed ERB towards the student.
Correlation analysis was used to test the other hypotheses regarding the relationship between job satisfaction, self-efficacy, and collective efficacy and each of the three levels of ERB (team, organization, student). A positive and significant relationship was found between teacher satisfaction and ERB at all three levels. Self-efficacy and ERB towards the team and the organization were also found to be correlated but there was no relationship proven between self-efficacy and ERB towards the student. There was a positive and significant relationship found between collective efficacy and ERB towards the team, but no relationship between collective efficacy and ERB towards the organization or the student.
The results of the factor analysis support the discussion of ERB as multidimensional. The extent to which the three different variables were or were not correlated to the various levels of ERB further support this. All together, the research suggests that formal efforts to increase the levels of job satisfaction, self-efficacy or collective efficacy will have varying impacts on levels and types of extra-role behaviors.
Study #2 – Organizational Citizenship Behavior and Pay-for-Performance plans
This study addressed whether there was an unintended negative interaction between pay-for-performance plans designed to reward and encourage certain in-role behaviors and those behaviors that are not formally rewarded or controlled through job descriptions but which yield value to the organization. Also, the researchers looked to identify whether the degree of interest alignment between the employer and the employee moderated the impact of pay-for-performance plans on those organizational citizenship, or extra-role, behaviors.
The sampling frame was employees in certain benchmark jobs and their supervisors, from eight different utility companies across the United States. The supervisors of the benchmark jobs in each company were chosen first and then asked to randomly select up to 10 subordinates who were then surveyed about attitudes towards the job and the work organization (n = 660). The sample was further culled by matching to supervisor responses and confirmation of their participation in a pay-for–performance plan. The subsample remaining consisted of 146 employees.
The survey issued to participants included items from several pre-existing measures of organizational citizenship behaviors, value alignment, and procedural justice, as well as items developed purposely to measure the employee’s perception of the link between performance and pay.
A regression analysis allowed the researchers to determine the extent to which the effect of the pay-for-performance link on OCB was influenced by the employee’s value commitment. It was found that if the employees are not aligned with the organization, their level of OCB will be lower as they perceive a stronger link between their performance and their pay. Similarly, if the employee is committed to the organization, their OCB will increase as the pay:performance link is stronger. Procedural justice was found to be a significant predictor of OCB.
This research is valuable in indicating that pay-for-performance plans will not necessarily negatively affect the level of organizational citizenship behavior displayed by the employee. In fact, if the employee and organization are “mutually invested”, employees may actually increase their OCB in response to the organization’s commitment to reward performance with pay.
On the topic of validity, both studies described limitations of their research and results. First, the measures utilized in both were either specially designed for the purpose or adapted from different pre-existing measures. There was no significant discussion in either of the studies that the survey items were themselves validated. Second, although relationships between variables and OCB were evidenced, there was no testing of the directionality of the effects. Finally, both studies used only self-reports of behavior and attitude. Again, this means that the results indicate relationships but only from the perspective of one source type. Therefore, the results of both studies are not generalizable and so much less useful in pointing a clear and convincing way forward on specific policy or program changes that can be implemented to increase or protect OCB.
References:
Somech, A., & Drach-Zahavy, A. (2000). Understanding extra-role behavior in schools: The relationships between job satisfaction, sense of efficacy, and teachers’ extra-role behavior. Teaching and Teacher Education, 16(5-6), 649-659. DOI: 10.1016/S0742-051X(00)00012-3
Deckop, J. R., Mangel, R., & Cirka, C. C. (1999). Research notes. getting more than you pay for: Organizational citizenship behavior and pay-for-performance plans. Academy of Management Journal, 42(4), 420-428.
The first study was by researchers in Israel who looked at the components of extra-role behavior (ERB) and the correlational relationship between ERB and three variables: job satisfaction, self-efficacy, and collective efficacy. The researchers defined extra-role behavior for the purposes of their study as,
"those behaviors that go beyond specified role requirements, and
are directed towards the individual, the group, or the organization as a unit,
in order to promote organizational goals." (Somech & Zahavy, 2000, p.650)
The sampling frame was elementary school teachers in northern Israel. Questionnaires were distributed to 375 teachers from 13 different elementary schools, of which 251 were returned to yield a response rate of 67%.
To develop a measure of extra-role behavior, they conducted a semi-structured interview of five principals and 25 teachers who were asked to list behaviors, outside of formal role requirements, that teachers exhibit to benefit of the student, their team or the school unit. The 60 items produced from this process were validated by teachers who were involved in a management-training program. The list was reduced in this way and then through factor analysis, it was organized into three subscales of eight items each, which measured ERB toward the student, towards the team, and towards the organization.
To measure the variables of job satisfaction, self-efficacy, and collective efficacy, three additional scales based on prior research were adapted for use with teachers. All three of these measures had respondents indicating their agreement or disagreement with the test items using a five- or seven-point Likert scale.
To test the first hypothesis regarding the components of ERB, the researchers performed a factor analysis which yielded three separate factors of ERB. First, the Factor 1 dimension included behaviors that were geared towards helping other teachers and so was recognized as ERB towards team members. Factor 2 behaviors were generally identified as those that were aimed at improving conditions for the school as a unit and so was terms ERB towards the organization. Factor 3 consisted of behaviors intended to improve student performance or the quality of the teaching. It was termed ERB towards the student.
Correlation analysis was used to test the other hypotheses regarding the relationship between job satisfaction, self-efficacy, and collective efficacy and each of the three levels of ERB (team, organization, student). A positive and significant relationship was found between teacher satisfaction and ERB at all three levels. Self-efficacy and ERB towards the team and the organization were also found to be correlated but there was no relationship proven between self-efficacy and ERB towards the student. There was a positive and significant relationship found between collective efficacy and ERB towards the team, but no relationship between collective efficacy and ERB towards the organization or the student.
The results of the factor analysis support the discussion of ERB as multidimensional. The extent to which the three different variables were or were not correlated to the various levels of ERB further support this. All together, the research suggests that formal efforts to increase the levels of job satisfaction, self-efficacy or collective efficacy will have varying impacts on levels and types of extra-role behaviors.
Study #2 – Organizational Citizenship Behavior and Pay-for-Performance plans
This study addressed whether there was an unintended negative interaction between pay-for-performance plans designed to reward and encourage certain in-role behaviors and those behaviors that are not formally rewarded or controlled through job descriptions but which yield value to the organization. Also, the researchers looked to identify whether the degree of interest alignment between the employer and the employee moderated the impact of pay-for-performance plans on those organizational citizenship, or extra-role, behaviors.
The sampling frame was employees in certain benchmark jobs and their supervisors, from eight different utility companies across the United States. The supervisors of the benchmark jobs in each company were chosen first and then asked to randomly select up to 10 subordinates who were then surveyed about attitudes towards the job and the work organization (n = 660). The sample was further culled by matching to supervisor responses and confirmation of their participation in a pay-for–performance plan. The subsample remaining consisted of 146 employees.
The survey issued to participants included items from several pre-existing measures of organizational citizenship behaviors, value alignment, and procedural justice, as well as items developed purposely to measure the employee’s perception of the link between performance and pay.
A regression analysis allowed the researchers to determine the extent to which the effect of the pay-for-performance link on OCB was influenced by the employee’s value commitment. It was found that if the employees are not aligned with the organization, their level of OCB will be lower as they perceive a stronger link between their performance and their pay. Similarly, if the employee is committed to the organization, their OCB will increase as the pay:performance link is stronger. Procedural justice was found to be a significant predictor of OCB.
This research is valuable in indicating that pay-for-performance plans will not necessarily negatively affect the level of organizational citizenship behavior displayed by the employee. In fact, if the employee and organization are “mutually invested”, employees may actually increase their OCB in response to the organization’s commitment to reward performance with pay.
On the topic of validity, both studies described limitations of their research and results. First, the measures utilized in both were either specially designed for the purpose or adapted from different pre-existing measures. There was no significant discussion in either of the studies that the survey items were themselves validated. Second, although relationships between variables and OCB were evidenced, there was no testing of the directionality of the effects. Finally, both studies used only self-reports of behavior and attitude. Again, this means that the results indicate relationships but only from the perspective of one source type. Therefore, the results of both studies are not generalizable and so much less useful in pointing a clear and convincing way forward on specific policy or program changes that can be implemented to increase or protect OCB.
References:
Somech, A., & Drach-Zahavy, A. (2000). Understanding extra-role behavior in schools: The relationships between job satisfaction, sense of efficacy, and teachers’ extra-role behavior. Teaching and Teacher Education, 16(5-6), 649-659. DOI: 10.1016/S0742-051X(00)00012-3
Deckop, J. R., Mangel, R., & Cirka, C. C. (1999). Research notes. getting more than you pay for: Organizational citizenship behavior and pay-for-performance plans. Academy of Management Journal, 42(4), 420-428.
Sunday, March 6, 2011
Managing validity
The concept of validity has been a lecture or exam topic in at least one class during each of my five semesters in graduate school. I have been asked to understand the validity of correlations between the Big Five personality traits with job performance, explain the validity of different tools used in recruiting and staffing and now finally, the validity of my own research. Over this time period, I had become familiar with, and even memorized, various definitions of validity and yet, I was still quite impressed with the explanation that was offered in our textbook. Dr. McMilllan provided an all of a sudden more relevant understanding for me, perhaps only partly because I knew that it was a concept worth 6% of our research proposal grade. On page 144, Dr. McMillan advises that validity is a “judgment of the appropriateness of a measure for the specific inferences or decisions that result from the scores generated by the measure.” For some reason, this explanation was much more about the “big picture of validity” than the other technical definitions I had learned previously. For me now, validity is as much about what I do or decide as a result of my findings as it is about how accurate the tools are that I use or the relationships that are observed. So as I develop this research aimed at observing the impact of merit pay incentives on teacher’s organizational citizenship behaviors, I want to carefully protect the instrument, the subjects, the treatment and the environment so that I am able to present inferences and suggest actions that are altogether appropriate and sound.
Although I am still searching for the best measure of organizational citizenship behaviors, I can foresee and will manage the threats to validity that the instrument itself can present. For instance, I know already that I cannot use a tool “off the shelf” that measures organizational citizenship behaviors (OCB) generally but instead, I require a measure that has been adjusted for the school setting. Luckily, there is a measurement model that has been developed for use with teachers but it includes additional scales that may not be relevant to the OCB discussion. So if I do make changes to this tool, the validity of the edited measure can only be confirmed if I reanalyze it’s “fit” to the theoretical model and to any other potentially valid measures.
In this study, I propose to measure and compare the levels of OCB exhibited by teachers who receive performance pay incentives to those who receive just a standard scheduled salary. In that the incentive compensation provided to the teachers is the “treatment”, I do not anticipate that the control group of teachers who receive only straight salary will inadvertently be “treated”. I do however need to stay extremely wary that both the control and the experimental group may be receiving a “treatment” via their environments, i.e. their departments, buildings, districts and communities, that impact their OCB levels. In this study then, the dependent variable is more at risk of being compromised by extraneous events, timelines and settings than the independent variable.
I sense that the greatest risk to validity comes in the course of selecting and maintaining study’s subjects themselves. As we all know, people are strange and generally act weirdly, especially when it comes to professional motivations and behaviors. That said, as much as I attempt to control for natural and cultural characteristics that affect OCB, I will undoubtedly fail to identify groups of subjects that are absolutely equal in their predispositions. Working in my favor though, previous studies of OCB have shown that individual characteristics like personality have limited associations with OCB and so it has been suggested that OCB is a result of cognitive and rational judgments and not emotional ones. If it were the reverse, I would need to even more carefully control for subject affectivity.
The effect of time and the announcement of the study itself on my subjects is another obvious threat to the validity of my study. Since it is not practical to announce the study and collect all the necessary data at one time, I can presume that my presence and the notice of the study will operate as an intervention in and of itself. The observation of OCB may already be compromised by changes in teacher’s actions in between the time the study is announced and when data is collected. As I finalize and summarize my literature review, I intend to keep my eye open for discussions about how this specific threat to validity can be minimized. I imagine it is a threat not only for my study but most others out there as well.
Even as I manage these threats to validity through careful design of the measure, documentation of the environment, and the “proper care and feeding” of my subjects, data analysis may still yield results that prevent any meaningful inferences or recommendations from being drawn. Although that would be very unfortunate, as Dr. McMillan would have us understand the concept of validity, recognition that a decision or inference should not be drawn from the study may still be a valid conclusion.
Although I am still searching for the best measure of organizational citizenship behaviors, I can foresee and will manage the threats to validity that the instrument itself can present. For instance, I know already that I cannot use a tool “off the shelf” that measures organizational citizenship behaviors (OCB) generally but instead, I require a measure that has been adjusted for the school setting. Luckily, there is a measurement model that has been developed for use with teachers but it includes additional scales that may not be relevant to the OCB discussion. So if I do make changes to this tool, the validity of the edited measure can only be confirmed if I reanalyze it’s “fit” to the theoretical model and to any other potentially valid measures.
In this study, I propose to measure and compare the levels of OCB exhibited by teachers who receive performance pay incentives to those who receive just a standard scheduled salary. In that the incentive compensation provided to the teachers is the “treatment”, I do not anticipate that the control group of teachers who receive only straight salary will inadvertently be “treated”. I do however need to stay extremely wary that both the control and the experimental group may be receiving a “treatment” via their environments, i.e. their departments, buildings, districts and communities, that impact their OCB levels. In this study then, the dependent variable is more at risk of being compromised by extraneous events, timelines and settings than the independent variable.
I sense that the greatest risk to validity comes in the course of selecting and maintaining study’s subjects themselves. As we all know, people are strange and generally act weirdly, especially when it comes to professional motivations and behaviors. That said, as much as I attempt to control for natural and cultural characteristics that affect OCB, I will undoubtedly fail to identify groups of subjects that are absolutely equal in their predispositions. Working in my favor though, previous studies of OCB have shown that individual characteristics like personality have limited associations with OCB and so it has been suggested that OCB is a result of cognitive and rational judgments and not emotional ones. If it were the reverse, I would need to even more carefully control for subject affectivity.
The effect of time and the announcement of the study itself on my subjects is another obvious threat to the validity of my study. Since it is not practical to announce the study and collect all the necessary data at one time, I can presume that my presence and the notice of the study will operate as an intervention in and of itself. The observation of OCB may already be compromised by changes in teacher’s actions in between the time the study is announced and when data is collected. As I finalize and summarize my literature review, I intend to keep my eye open for discussions about how this specific threat to validity can be minimized. I imagine it is a threat not only for my study but most others out there as well.
Even as I manage these threats to validity through careful design of the measure, documentation of the environment, and the “proper care and feeding” of my subjects, data analysis may still yield results that prevent any meaningful inferences or recommendations from being drawn. Although that would be very unfortunate, as Dr. McMillan would have us understand the concept of validity, recognition that a decision or inference should not be drawn from the study may still be a valid conclusion.
Sunday, February 27, 2011
Beware the causes!
I have read two particular studies in the past few days, one for this class and another for a Guided Study in International Human Resources, in which the authors "point blank" warned against making assumptions about causality based on the data they collected. Both were designed to assess correlations between variables so I’ll first discuss the dangers of implied causation in correlational studies as they were highlighted by our textbook author. First, Dr. McMillan points out that even if a positive or negative or predictive relationship between two or more variables is proved, the causality of that relationship should not be assumed because a correlation is not designed to provide evidence about causal direction.
This particular problem was illustrated in an article about the factors that influence the work role adjustment of an expatriate manager in Japan. In his study, Black (1988) hypothesized that the level of family’s adjustment is correlated to the manager’s adjustment but he backed away from trying to assess causal direction because the family’s adjustment (or lack thereof) could be caused by the manager’s adjustment (or lack thereof) just as easily as the manager’s adjustment could be caused by the family’s adjustment.
The second study that mentioned causality as a limitation is one that I am reading for my research proposal, about the relationships between several individual and organizational characteristics and organizational citizenship behaviors (OCB) in schools. At the end of the article, the authors (Somech & Ron, 2007) suggest that since they used a cross-sectional survey for their study, they were not able to assess the extent of a causal relationship between the variables and the OCB that they otherwise were able to show were correlated.
The other warning that Dr. McMillan issues about inferring causation from correlational studies is more so a warning about taking actions based on observed correlations. Again, going back to the studies I’ve picked as examples, if one were to deploy 100% of available resources to address and improve the family’s adjustment to the international assignment when it is actually the manager’s issues that are causing the family’s disorder then the organization is misallocating its resources. And based on the OCB study, if it were determined that OCB precede the evidence of organizational characteristics, policies that focus resources on replicating the related organizational characteristics will be misaligned.
This assumption of causation is equally a problem in comparative studies. Even if a study shows that there are significant similarities or differences between groups when certain variables are assessed, it does not mean that the groups are similar or different because of the independent variable. To illustrate this, I’ll use an aspect of my own planned research proposal. If I were to execute a comparative study only and I determined that the OCB reported by teachers who receive merit pay incentives is lower than those teachers who do not receive merit pay, it would be very unwise to stop there with a conclusion that merit pay reduces teacher OCB. The lower OCB observation could very well be related to a different characteristic or variable that I have not even included in my study. My conclusion then should be to propose research that digs deeper into other causes or implications of OCB and how those might also interact with merit incentives.
Citations for the studies used as examples above are:
Black, J. S. (1988). Work role transitions: A study of American expatriate managers in Japan. Journal of International Business Studies, 19(2), pp. 277-294.
Somech, A., & Ifat, R. (2007). Promoting organizational citizenship behavior in schools: The impact of individual and organizational characteristics. Educational Administration Quarterly, 43(1), 38-66. doi:10.1177/0013161X06291254
This particular problem was illustrated in an article about the factors that influence the work role adjustment of an expatriate manager in Japan. In his study, Black (1988) hypothesized that the level of family’s adjustment is correlated to the manager’s adjustment but he backed away from trying to assess causal direction because the family’s adjustment (or lack thereof) could be caused by the manager’s adjustment (or lack thereof) just as easily as the manager’s adjustment could be caused by the family’s adjustment.
The second study that mentioned causality as a limitation is one that I am reading for my research proposal, about the relationships between several individual and organizational characteristics and organizational citizenship behaviors (OCB) in schools. At the end of the article, the authors (Somech & Ron, 2007) suggest that since they used a cross-sectional survey for their study, they were not able to assess the extent of a causal relationship between the variables and the OCB that they otherwise were able to show were correlated.
The other warning that Dr. McMillan issues about inferring causation from correlational studies is more so a warning about taking actions based on observed correlations. Again, going back to the studies I’ve picked as examples, if one were to deploy 100% of available resources to address and improve the family’s adjustment to the international assignment when it is actually the manager’s issues that are causing the family’s disorder then the organization is misallocating its resources. And based on the OCB study, if it were determined that OCB precede the evidence of organizational characteristics, policies that focus resources on replicating the related organizational characteristics will be misaligned.
This assumption of causation is equally a problem in comparative studies. Even if a study shows that there are significant similarities or differences between groups when certain variables are assessed, it does not mean that the groups are similar or different because of the independent variable. To illustrate this, I’ll use an aspect of my own planned research proposal. If I were to execute a comparative study only and I determined that the OCB reported by teachers who receive merit pay incentives is lower than those teachers who do not receive merit pay, it would be very unwise to stop there with a conclusion that merit pay reduces teacher OCB. The lower OCB observation could very well be related to a different characteristic or variable that I have not even included in my study. My conclusion then should be to propose research that digs deeper into other causes or implications of OCB and how those might also interact with merit incentives.
Citations for the studies used as examples above are:
Black, J. S. (1988). Work role transitions: A study of American expatriate managers in Japan. Journal of International Business Studies, 19(2), pp. 277-294.
Somech, A., & Ifat, R. (2007). Promoting organizational citizenship behavior in schools: The impact of individual and organizational characteristics. Educational Administration Quarterly, 43(1), 38-66. doi:10.1177/0013161X06291254
Friday, February 18, 2011
Primum non nocere – First, do no harm.
I’ve wondered before about this adage because I naturally tend to assume that only a sociopath would ever enter a situation planning to harm anther being. Yet this protective principle is found front and center in the code of ethics of most professions, as it is in James McMillan’s “top ten” aspects of ethical concerns for educational researchers. I suggest that it is absolutely appropriate for professions and organizations to keep it in focus, not because it is a natural approach (at least for the psychologically stable) but because it is a universal principle whose operational definition is easily comprehended.
So what does it mean for us as educational researchers to 'do no harm'? To me, it means that we take utmost care to identify the implicit risks, the explicit risks and all of our stakeholders prior to conducting and designing research. Explicit risks are the obvious physical threats or psychological dangers that might be posed by our presence, by coercive techniques, by our withholding of important truths, or simply put, our negligence. The implicit risks are the “harm” that might be caused by our lack of planning or forethought, or our failure to make contingency plans. And beyond the need to manage risks, we have to exhaustively discover all of our stakeholders which I think is harder and takes longer than expected.
Using my research problem to illustrate this responsibility, I have to start first with the participants that I sample, namely the teachers who are compensated via performance incentives as well as those in the control group who are paid straight salary. In addition to the participants directly measured, I will require resources in the form of both data and time from their administrators, from their Human Resources support staff and from their district leadership. And beyond them, I can identify the other stakeholders as the students in their classrooms, their families, their communities, and the school budgets that pay them. All of these parties will be directly impacted by my research and certainly by any incremental change in performance or behavior that results from my presence or the conclusions that my research supports. Unless I take great care to identify all of the stakeholders involved in the system I intend to research, I cannot ever fully manage the risks of my methods.
Another ethical responsibility that McMillan mentions and about which I feel strongly is the role of the investigator in maintaining the confidentiality of the participants and the propriety of the study. To me, this means ensuring that data and records are protected and planning for all contingencies necessary to safeguard them, regardless of the financial and physical resources required. The researcher is also responsible for ensuring the work is all unique or is properly cited if not. As well, in the end, she must guarantee that the design, the data, the methods, the results and the conclusions are fully transparent and fully explained.
In this, I arrive full circle at the original, overriding guideline and to share a concluding operational definition. ‘To do no harm’ means to always leave the participants, their communities, and fellow researchers in a better place than where you found them. Even if my data reveals that performance pay does not impact the organizational citizenship behaviors of teachers, this conclusion can still in some way benefit those leaders who make educational policy on behalf of our children.
So what does it mean for us as educational researchers to 'do no harm'? To me, it means that we take utmost care to identify the implicit risks, the explicit risks and all of our stakeholders prior to conducting and designing research. Explicit risks are the obvious physical threats or psychological dangers that might be posed by our presence, by coercive techniques, by our withholding of important truths, or simply put, our negligence. The implicit risks are the “harm” that might be caused by our lack of planning or forethought, or our failure to make contingency plans. And beyond the need to manage risks, we have to exhaustively discover all of our stakeholders which I think is harder and takes longer than expected.
Using my research problem to illustrate this responsibility, I have to start first with the participants that I sample, namely the teachers who are compensated via performance incentives as well as those in the control group who are paid straight salary. In addition to the participants directly measured, I will require resources in the form of both data and time from their administrators, from their Human Resources support staff and from their district leadership. And beyond them, I can identify the other stakeholders as the students in their classrooms, their families, their communities, and the school budgets that pay them. All of these parties will be directly impacted by my research and certainly by any incremental change in performance or behavior that results from my presence or the conclusions that my research supports. Unless I take great care to identify all of the stakeholders involved in the system I intend to research, I cannot ever fully manage the risks of my methods.
Another ethical responsibility that McMillan mentions and about which I feel strongly is the role of the investigator in maintaining the confidentiality of the participants and the propriety of the study. To me, this means ensuring that data and records are protected and planning for all contingencies necessary to safeguard them, regardless of the financial and physical resources required. The researcher is also responsible for ensuring the work is all unique or is properly cited if not. As well, in the end, she must guarantee that the design, the data, the methods, the results and the conclusions are fully transparent and fully explained.
In this, I arrive full circle at the original, overriding guideline and to share a concluding operational definition. ‘To do no harm’ means to always leave the participants, their communities, and fellow researchers in a better place than where you found them. Even if my data reveals that performance pay does not impact the organizational citizenship behaviors of teachers, this conclusion can still in some way benefit those leaders who make educational policy on behalf of our children.
Saturday, February 12, 2011
Sampling
My review of literature relating to merit pay in education is only a fraction complete. I’ve already taken a few wrong turns on my way to different grocery stores, all in the name of looking for the right fruit.
I am settling now on the concept of teacher organizational citizenship behavior (OCB) and how it is impacted by merit pay compensation. I intend to look further at research already available regarding the components, causes and moderators of OCB, in teachers and other professionals. I also will review the efficacy of using existing instruments to measure OCB in teachers. At this point though, I hypothesize that merit pay compensation programs will moderate the OCB of public high-school teachers in the United States.
I believe that the interaction between merit pay and teacher OCB will eventually be best explored through a mixed-methods analysis taking both a quantitative and qualitative approach to the problem. However, I propose to start with a quantitative study that compares the level of OCB in teachers who are compensated via a merit pay system to teachers who are compensated via a fixed salary schedule. I will discuss and debate the various sampling procedures here as if I am undertaking that particular research. In this case, my sampling frame will need to be much narrower than my target population of all public U.S. high school teachers because I must account for other environmental and organizational culture factors that may moderate OCB in the experimental and the control groups. Also, specifically for the experimental group, I will need to evaluate the design of the compensation plan to determine whether any extra-role behaviors actually become in-role behaviors according to the performance targets established by the merit pay plan.
The sampling procedure that I use will depend largely on the availability and size of a sampling frame to which I can gain access. If I can find a sizable school district that has a large enough population of high school teachers who are compensated via merit pay as well as a population of teachers in the same schools who are not being compensated with performance incentives, I may be able to utilize some probability sampling procedures. Assuming such a survey population is available, I could use a simple random or systematic procedure to randomly select subjects for the sample. The risk in taking these approaches is that I may not get a representative sample, i.e. male and female teachers or teachers with varying experience or education, which will bias the measurement OCB and thus, my results. To solve for this, again assuming that the sample frame is large enough, I would first identify and divide the population into different subgroups related to the dependent variable – OCB. For example, I would identify subgroups by gender, years of education and/or experience, subject taught, school taught in, etc. for each of the experimental and control populations. I could then take a proportional stratified sample from these samples or if I don’t have enough participants represented in each subgroup, I could take a disproportional stratified sample and then weight the results to reflect the whole population.
Using a cluster sample from within the district might be more appropriate, that is I could find one particular high school within the district that has teachers who are compensated with merit pay and those who are not. Although limiting the sample frame in this way may reduce the validity of the results, it may help to control for some of the environmental and organizational culture factors that moderate OCB.
In the case that I am not able to gain access to a school district large enough to support probability sampling, I will need to rely on nonprobability sampling. Using a convenience sample procedure, I would be identify a particular high school which has instituted a merit pay plan for some but not all of its teachers and then measure and analyze the OCB of teachers in that particular school. In the unexpected case that I find one such high school that meets these criteria yet is also big enough to afford me the chance to sample certain subgroups, I could effect a quota sample and select participants who represent the subgroups mentioned earlier, like gender and teaching experience.
If I were to pursue a mixed-methods analysis of merit pay and teacher OCB, I could proceed to use some purposeful sampling procedures like typical or extreme case sampling. In each of these, I would define and identify the “average” (for typical case) or “atypical” (for extreme case) high school teacher from a particular school division or high school and measure the OCB exhibited by those who are and are not compensated with merit pay. If I find both the “average” and the “atypical” teacher, I can effect a maximum variation sample.
In the case of this research proposal, a snowball sample seems inappropriate. I believe that there would be too many different cultural and environmental factors at play in such a network sample which would render the OCB measurements and conclusions invalid. In the same way though, I think that to control for these same factors, an element of critical case sampling will be valuable in this study. That is, I think the study will ultimately need to identify one particular school district or even one particular high school from which an experimental and a control group of teachers can be drawn.
If I use nonprobability sampling procedures and an element of critical case sampling as currently planned, I know that I won’t be able to generalize any discoveries about the relationship between teacher OCB and merit pay to the larger US high school teacher population. However, I still think that the results will be significant for the school district that is sampled and that the data obtained will inform us about how merit pay and teacher OCB interact. At the least, it will suggest complementary future studies.
I am settling now on the concept of teacher organizational citizenship behavior (OCB) and how it is impacted by merit pay compensation. I intend to look further at research already available regarding the components, causes and moderators of OCB, in teachers and other professionals. I also will review the efficacy of using existing instruments to measure OCB in teachers. At this point though, I hypothesize that merit pay compensation programs will moderate the OCB of public high-school teachers in the United States.
I believe that the interaction between merit pay and teacher OCB will eventually be best explored through a mixed-methods analysis taking both a quantitative and qualitative approach to the problem. However, I propose to start with a quantitative study that compares the level of OCB in teachers who are compensated via a merit pay system to teachers who are compensated via a fixed salary schedule. I will discuss and debate the various sampling procedures here as if I am undertaking that particular research. In this case, my sampling frame will need to be much narrower than my target population of all public U.S. high school teachers because I must account for other environmental and organizational culture factors that may moderate OCB in the experimental and the control groups. Also, specifically for the experimental group, I will need to evaluate the design of the compensation plan to determine whether any extra-role behaviors actually become in-role behaviors according to the performance targets established by the merit pay plan.
The sampling procedure that I use will depend largely on the availability and size of a sampling frame to which I can gain access. If I can find a sizable school district that has a large enough population of high school teachers who are compensated via merit pay as well as a population of teachers in the same schools who are not being compensated with performance incentives, I may be able to utilize some probability sampling procedures. Assuming such a survey population is available, I could use a simple random or systematic procedure to randomly select subjects for the sample. The risk in taking these approaches is that I may not get a representative sample, i.e. male and female teachers or teachers with varying experience or education, which will bias the measurement OCB and thus, my results. To solve for this, again assuming that the sample frame is large enough, I would first identify and divide the population into different subgroups related to the dependent variable – OCB. For example, I would identify subgroups by gender, years of education and/or experience, subject taught, school taught in, etc. for each of the experimental and control populations. I could then take a proportional stratified sample from these samples or if I don’t have enough participants represented in each subgroup, I could take a disproportional stratified sample and then weight the results to reflect the whole population.
Using a cluster sample from within the district might be more appropriate, that is I could find one particular high school within the district that has teachers who are compensated with merit pay and those who are not. Although limiting the sample frame in this way may reduce the validity of the results, it may help to control for some of the environmental and organizational culture factors that moderate OCB.
In the case that I am not able to gain access to a school district large enough to support probability sampling, I will need to rely on nonprobability sampling. Using a convenience sample procedure, I would be identify a particular high school which has instituted a merit pay plan for some but not all of its teachers and then measure and analyze the OCB of teachers in that particular school. In the unexpected case that I find one such high school that meets these criteria yet is also big enough to afford me the chance to sample certain subgroups, I could effect a quota sample and select participants who represent the subgroups mentioned earlier, like gender and teaching experience.
If I were to pursue a mixed-methods analysis of merit pay and teacher OCB, I could proceed to use some purposeful sampling procedures like typical or extreme case sampling. In each of these, I would define and identify the “average” (for typical case) or “atypical” (for extreme case) high school teacher from a particular school division or high school and measure the OCB exhibited by those who are and are not compensated with merit pay. If I find both the “average” and the “atypical” teacher, I can effect a maximum variation sample.
In the case of this research proposal, a snowball sample seems inappropriate. I believe that there would be too many different cultural and environmental factors at play in such a network sample which would render the OCB measurements and conclusions invalid. In the same way though, I think that to control for these same factors, an element of critical case sampling will be valuable in this study. That is, I think the study will ultimately need to identify one particular school district or even one particular high school from which an experimental and a control group of teachers can be drawn.
If I use nonprobability sampling procedures and an element of critical case sampling as currently planned, I know that I won’t be able to generalize any discoveries about the relationship between teacher OCB and merit pay to the larger US high school teacher population. However, I still think that the results will be significant for the school district that is sampled and that the data obtained will inform us about how merit pay and teacher OCB interact. At the least, it will suggest complementary future studies.
Friday, February 11, 2011
Subscribe to:
Posts (Atom)
