Some time ago, I “replicated” Rynes, Colbert, and Brown’s 2002 study on HR practitioners’ beliefs about effective HR practices on a convenience sample of more than 140 LinkedIn users. The results of this “replication” closely resembled the results of the original study. On average, respondents correctly answered 19.4 out of 35 items, achieving a 55% success rate, which was very close to the 57% average success rate in the original study (and also quite close to the 50% success rate that corresponds to random choice, given the TRUE/FALSE response format).
I was curious to see how GPT-4 would perform in this test, as it had been evaluated on various standardized tests such as the SAT, GRE, Bar Exam, and AP. The prompts used had the following form: Read the following statement and indicate whether it is true or false. Keep in mind that the statement refers to general trends and patterns that apply on average but not necessarily to all cases. When evaluating the statement, ensure that you correctly interpret the words used in the statement and take into account existing scientific evidence. Give me the answer either true or false, without intermediate values, in a boolean way. Finally, briefly explain your reasoning behind your answer. The statement is as follows:…
So, what were the results? GPT-4 answered 29 out of 35 items right, i.e., it achieved an 83% success rate, which corresponds to the 99th and 97th percentiles in the original and “replicated” studies, respectively. GPT-4’s results were thus superior to majority of people who took the test.
However, even when it gave a correct answer, it did not always rely on correct facts and/or valid reasoning, which could be a problem if management decided to act on the answers provided. See the table below to check the details of its responses.
# uploading libraries
library(tidyverse)
library(readxl)
library(DT)
# uploading data
mydata <- readxl::read_xlsx("./gpt4Responses.xlsx")
# creating user-friendly table
DT::datatable(
mydata %>%
dplyr::select(itemId, item, gpt4Response, correctAnswer, gpt4Reasoning, researchEvidence, possibleContingencies) %>%
dplyr::rename("Item ID"=itemId, Item=item, "GPT-4 response"=gpt4Response, "Correct answer"=correctAnswer, "GPT-4 reasoning"=gpt4Reasoning, "Research evidence"=researchEvidence, "Possible contingencies"=possibleContingencies),
class = 'cell-border stripe',
filter = 'top',
extensions = 'Buttons',
fillContainer = FALSE,
rownames= FALSE,
options = list(
pageLength = 3,
autoWidth = TRUE,
columnDefs = list(list(width = '500px', targets = c("Item", "GPT-4 reasoning", "Research evidence", "Possible contingencies"))),
dom = 'Bfrtip',
buttons = c('copy'),
scrollX = TRUE,
selection="multiple"
)
) %>%
DT::formatStyle(1:7, 'vertical-align'='top')Showing 20 of 35 rows and all 7 columns. View the full interactive table in the original post.
| Item ID | Item | GPT-4 response | Correct answer | GPT-4 reasoning | Research evidence | Possible contingencies |
|---|---|---|---|---|---|---|
| item1 | Leadership training is ineffective because good leaders are born, not made. | False | False | The statement is false because there is substantial evidence that leadership skills can be developed and improved through training and experience. While some individuals may have innate qualities that make them more predisposed to becoming effective leaders, leadership is a complex skillset that can be cultivated and enhanced over time. Research has shown that leadership training can indeed have a positive impact on individuals' abilities to lead others, communicate effectively, and make better decisions. | Field study evidence that leadership behaviors and effectiveness increase following training (Barling et al., 1996). Evidence that leadership behaviors are only weakly predicted by dispositional characteristics (Judge & Bono, 2000) that are heritable (Loehlin et al., 1998; Reimann et al., 1997) | |
| item2 | The most important requirement for an effective leader is to have an outgoing, enthusiastic personality. | False | False | The statement is false because, while an outgoing and enthusiastic personality can be beneficial in some leadership situations, it is not the most important requirement for an effective leader. Effective leadership encompasses a range of skills and attributes, such as empathy, strategic thinking, adaptability, and effective communication. Different leadership styles can be effective in various contexts, and introverted leaders may excel in certain situations where they can listen attentively, motivate others, and make thoughtful decisions. | This kind of personality is, on average, an asset for leadership. A recent meta-analysis estimates a corrected validity coefficient of .31 between extraversion and leader effectiveness (Judge et al in press). However, intelligence has an even higher correlation (.52; Lord et al 1986). Also, some highly effective leaders are distinctly introverted (Bennis & Nanus, 1997; Collins, 2001) | Although it is unlikely that any future variables will produce higher validities than intelligence, work on leader behaviors (such as visionsetting and communication skills) is just beginning, so little is known about their average effect sizes. We do know (e.g., Barling et al., 1996; Baum et al., 1998) that leadership success can be affected by leaders’ behaviors and not just their traits, so future research would be useful. |
| item3 | Once employees have mastered a task, they perform better when they are told to “do their best” than when they are given specific, difficult performance goals. | False | False | Research suggests that specific, difficult performance goals generally lead to higher performance compared to vague or "do your best" goals. When employees have clear, challenging objectives, they are more likely to be motivated to achieve them, resulting in better performance. However, it is important to ensure that the goals are realistic and attainable to prevent frustration or disengagement. | Employees reach higher levels of performance when they are given difficult-yetattainable goals rather than told to do their best. This is one of the most robust findings in all of industrial/organizational psychology (Locke & Latham, 1990) | |
| item4 | Companies with vision statements perform better than those without them. | True | True | On average, companies with vision statements tend to perform better than those without them. A vision statement provides a clear direction and purpose for the organization, helping to align employees and stakeholders towards common goals. This clarity can lead to improved decision-making, better resource allocation, and increased motivation among team members, which can ultimately result in better overall performance. However, it is essential to note that the mere presence of a vision statement does not guarantee success, and its effectiveness depends on the quality of the statement and the commitment of the organization to uphold and pursue its vision. | Baum et al. (1998) found that growth oriented visions and strong vision communication produced significantly higher firm growth rates. Hoch et al. (1999) found that 93% of the most successful software firms had clear and ambitious visions, as compared with 25% of the least successful. The importance of having a product vision for successful product development has also been reviewed by Brown & Eisenhardt (1995) | |
| item5 | Companies with very low rates of professional turnover are less profitable than those with moderate turnover rates. | True | False | The reasoning behind this answer is that companies with very low rates of professional turnover may experience a lack of innovation and fresh perspectives, which could negatively impact their overall profitability. On the other hand, companies with moderate turnover rates benefit from a balance between retaining experienced employees and bringing in new talent with fresh ideas. This balance can contribute to improved performance and profitability. | Bain & Company (Reichheld, 1996) has analyzed the economics of professional turnover in several industries. For example, in brokerage firms, they calculated that an increase in broker retention rates from 80% to 90% results in an increase of 155% in the average net present value of a new broker. A general model of the “economics of employee loyalty” is also presented | This finding might not hold true in certain types of internal labor market contexts — e.g., organizations with strong employment “guarantees” (e.g., government employment), or organizations with very strong contingency reward systems or “up-or-out” promotion & partnership systems. More contingencybased research would be useful. |
| item6 | If a company feels it must downsize employees, the most profitable way to do it is through targeted cuts rather than attrition. | True | True | The statement is generally true because targeted cuts allow a company to strategically eliminate positions or employees that are less essential to their operations, while attrition relies on a more passive approach that may not result in the desired efficiency or cost savings. By making targeted cuts, a company can better maintain its productivity and overall performance, which can contribute to higher profitability. | Morris, Cascio, & Young (1999) tracked changes in employment of the S&P 500 companies over a 12-year period. They found that companies whose downsizing was associated with sale of assets (i.e strategic or targeted downsizing) had improved their return on assets by the second year after downsizing. In contrast, pure “employment downsizers” still had lower ROAs two years later than they had before downsizing. See also Cameron et al. (1993) | |
| item7 | In order to be evaluated favorably by line managers, the most important competency for HR managers is the ability to manage change. | True | True | While there are various competencies that HR managers need to possess, the ability to manage change has become increasingly important in the rapidly evolving business landscape. Line managers expect HR managers to lead and facilitate organizational change, such as adapting to new technologies, managing mergers and acquisitions, and adjusting to shifting market conditions. By demonstrating strong change management skills, HR managers can effectively support line managers and the organization as a whole, leading to more favorable evaluations. | This was found by Ulrich et al. (1995), based on more than 12,000 peer and supervisory assessments of the performance of nearly 2,000 HR professionals. Ability to manage change explained 41.2% of the variance in supervisors’ and peers’ evaluations of HR professionals, as compared with 23.3% for HR knowledge and delivery and 18.8% for knowledge of the business | |
| item8 | On average, encouraging employees to participate in decision making is more effective for improving organizational performance than setting performance goals. | False | False | While encouraging employees to participate in decision-making can have positive effects on organizational performance, research generally suggests that setting performance goals is a more effective strategy on average. Goal-setting theory, supported by various studies, has shown that setting specific and challenging goals leads to higher performance levels compared to when no goals or vague goals are set. Encouraging participation in decision-making can foster engagement and commitment, but it does not necessarily provide a clear direction or specific targets for performance improvement. | Meta-analytic evidence that the effects of participation are weaker (< 1%) than the effects of goal setting (20%; Locke et al 1980). These effects also hold at multiple levels of analysis. Further evidence shows that the effects of goal setting are robust (Locke & Latham, 1990), while the effects of participation are highly variable (e.g Wagner, 1994) | |
| item9 | Most managers give employees lower performance appraisals than they objectively deserve. | False | False | The statement suggests that most managers consistently give employees lower performance appraisals than they objectively deserve. While there may be individual cases or certain contexts where this occurs, it is not accurate to say that this is a general trend or pattern applicable to most managers. It is important to note that performance appraisals can be subjective and may vary depending on the manager's perspective, criteria, and assessment methods. However, it is not accurate to claim that most managers systematically underrate their employees. Managers generally aim to provide fair and accurate evaluations to support employee growth and development, and organizations typically have guidelines in place to help ensure consistency and fairness in the appraisal process. | Appraisal leniency is much more common than stringency (Jawahar & Williams, 1997; Longenecker et al 1987) | |
| item10 | Poor performers are generally more realistic about their performance than good performers are. | False | False | This statement contradicts the well-known Dunning-Kruger effect, which states that individuals with low ability at a task often overestimate their ability, while individuals with high ability tend to underestimate their performance. This cognitive bias leads to poor performers being less realistic about their performance compared to good performers. | Primary study evidence that poor performers are less accurate about their relative performance than are good performers (Kruger & Dunning, 1999) | |
| item11 | Teams with members from different functional areas are likely to reach better solutions to complex problems than teams from a single area. | True | True | The reasoning behind this answer is that teams with members from different functional areas bring diverse perspectives, skills, and expertise to the table. This diversity often leads to more creative and innovative solutions to complex problems, as team members can approach issues from various angles and challenge one another's assumptions. Additionally, a diverse team can better identify potential pitfalls and benefits that might be overlooked by a more homogenous group. | Multiple studies on cross-functional teams have shown positive outcomes for product and project quality (e.g Keller, 2001; Lutz, 1994; Northcraft et al 1995; Pelled et al 1999). At the top management level, Hambrick et al. (1996) found that more heterogeneous teams (with respect to functional area, education, and tenure) made bolder (although slower) competitive moves, causing an overall net positive effect on firm market share and profits | Although product outcomes are generally positive, psychosocial outcomes such as member satisfaction or stress are frequently negative. Also, it is possible that at very high levels of multiple types of diversity, interpersonal conflict might overtake the performance benefits of diversity. |
| item12 | Despite the popularity of drug testing, there is no clear evidence that applicants who score positive on drug tests are any less reliable or productive employees. | True | False | Research has shown that there is no clear evidence supporting the claim that applicants who test positive for drugs are less reliable or productive employees. Factors such as job performance, attendance, and safety records are more important when assessing employee reliability and productivity. However, it is important to note that this statement refers to general trends and patterns, and individual cases may vary. | Norman et al. (1990) drug tested more than 4,000 applicants and then followed them through more than 1 year of employment. Those who tested positive had a 59% higher absenteeism rate and a 47% higher involuntary turnover rate. Parish (1989) found significant results for disciplinary actions, and Winkler & Sheridan(1989) for vehicular accidents, absenteeism, and medical costs. See also McDaniel et al. (1988) | |
| item13 | Most people overevaluate how well they perform on the job. | True | True | Most people overevaluate how well they perform on the job. This phenomenon is known as the Dunning-Kruger effect, which is a cognitive bias where people with low ability at a task overestimate their ability, while those with high ability tend to underestimate their competence. This effect is widely observed in various domains, including work performance. | Meta-analytic and primary study evidence that self-ratings have higher means than peer and supervisor ratings (e.g Brown, 1986; Harris & Schaubroeck, 1988; Mabe & West, 1982; Thornton, 1980) | |
| item14 | Most errors in performance appraisals can be eliminated by providing training that describes the kinds of errors managers tend to make and suggesting ways to avoid them. | False | False | While training can help reduce errors in performance appraisals by raising awareness about common mistakes, it is unlikely to eliminate most errors completely. Performance appraisals are subjective by nature, and even with training, factors such as personal biases, time constraints, and insufficient data can still lead to errors. Training can be helpful, but it is not a guaranteed solution for eliminating most errors in performance appraisals. | Most performance appraisal errors are extremely resistant to change (Latham & Wexley, 1994). In addition, training to eliminate one kind of error often introduces other types of errors (e.g Bernardin & Pence, 1980). Many managers are well aware that they commit errors, but continue to do so for personal and social reasons (Longenecker et al 1987) | |
| item15 | Lecture-based training is generally superior to other forms of training delivery. | False | False | Lecture-based training is not generally superior to other forms of training delivery. Research has shown that various training methods, such as interactive learning, hands-on practice, and blended learning, can be more effective in fostering engagement and long-term retention of information. The effectiveness of a training method often depends on the context and the learning styles of the participants, so no single method can be considered universally superior. | Meta-analytic evidence that computerbased instruction is slightly more effective than traditional instruction (Kulik & Kulik, 1991). Indirect evidence is provided by educational theory suggesting that the medium for instruction is less critical than the events of instruction in determining learning outcomes (e.g Clark, 1983). In this regard, lecture-based training is a passive technique, while active techniques are typically more effective (Gagne & Medsker, 1996) | |
| item16 | Older adults learn more from training than younger adults. | False | False | On average, younger adults tend to learn more quickly and effectively from training than older adults. This is generally attributed to age-related cognitive decline, which can impact memory, processing speed, and other cognitive abilities. However, this statement does not apply to all cases, as individual variation exists and older adults can still learn effectively. | Meta-analytic evidence that age is negatively associated with learning outcomes (Colquitt et al 2000). Large sample evidence that age is associated with decreases in cognitive capacities associated with learning (Horn & Cattell, 1967). Primary study evidence of negative relationship with learning in organizational setting (Warr & Bunce, 1995) | |
| item17 | The most important determinant of how much training employees actually use on their jobs is how much they learned during training. | False | False | The most important determinant of how much training employees actually use on their jobs is not solely how much they learned during training. While the amount of learning during training is important, other factors also significantly influence the transfer of training to the job. These factors may include organizational support, job relevance of the training, individual motivation, and opportunities to apply the training on the job. | Meta-analytic evidence that post-training knowledge has a smaller relationship with transfer than some individual differences and contextual variables (Colquitt et al 2000). Primary study evidence that transfer of training climate has a stronger relationship with transfer than learning does, in an organizational setting (Tracey et al 1995) | |
| item18 | Training for simple skills will be more effective if it is presented in one concentrated session than if it is presented in several sessions over time. | False | False | The statement is generally false because, according to the spacing effect and distributed practice principles in learning, training for simple skills is typically more effective when presented in several sessions over time. This approach allows for better retention and understanding, as it gives the learner time to consolidate and internalize the information. | Meta-analytic evidence that training for simple skills is more effective when spaced over time than massed in one session (Donovan & Radosevich, 1999; Lee & Genovese, 1988) | |
| item19 | The most valid employment interviews are designed around each candidate’s unique background. | False | False | The statement is false because the most valid employment interviews are typically structured and standardized, rather than designed around each candidate's unique background. Research has shown that structured interviews, which use the same questions for all candidates and follow a consistent evaluation process, are more effective at predicting job performance and reducing bias than unstructured interviews, which may focus on each candidate's unique background. By following a consistent structure, interviewers can compare candidates more fairly and make better hiring decisions. | Meta-analytic evidence that structured interviews (where all candidates receive the same questions) have higher validities than unstructured ones (Schmidt & Hunter, 1998; Wiesner & Cronshaw, 1988) | |
| item20 | Although people use many different terms to describe personalities, there are really only four basic dimensions of personality, as captured by the Myers-Briggs Type Indicator (MBTI). | False | False | The statement is false because the Myers-Briggs Type Indicator (MBTI) is not considered a comprehensive or scientifically valid measure of personality by the majority of psychologists. Instead, the Five Factor Model (FFM), also known as the Big Five personality traits, is widely accepted as a more robust and empirically supported model of personality. The FFM includes five dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism (OCEAN). | There are five basic dimensions of personality—the “Big Five” of Conscientiousness, Extraversion, Openness to Experience, Agreeableness, and Emotional Stability/Neuroticism (Digman, 1990). Except for Extraversion, these are not the traits assessed by MBTI |
The possible takeaway from this finding? Although GPT-4 can be a handy tool for exploring possible solutions to specific HR-related problems, on its own and in its current form it cannot replace the good old systematic search for and retrieval of evidence, critical evaluation of its reliability and relevance, and its weighing and synthesis as conducted and/or supervised by human experts.
P.S. I didn’t test the reliability of GPT-4’s responses, nor did I set its temperature to 0, so it’s possible that you might obtain somewhat different results if you decide to replicate the test. In addition, please keep in mind that the comparison presented here is not entirely an apples-to-apples comparison, mainly due to the fact that new evidence may have emerged that does not match the correct answers in the original study conducted more than 20 years ago.
Related notes
- Evaluation of the results of the evidence-based HRM knowledge test
- Evidence-based HRM knowledge test
- Can genAI help people managers lead better?
- Good manager or lucky team?
- Novel way to measure leadership skills via causal inference (and AI)?
📄 Read the original post with full outputs on my blog.