An Objective
Structured Screening Procedure
in the
Selection of Surgical Trainees
Reynaldo O. Joson, MD, MHA, MHPEd, FPCS
Bernardo S. Fernandez, MD., FPCS
Harry K. Go, MD, FPCS
Jose Joaquin Hernandez, MD, FPCS
Edgardo P. Penserga, MD. FFCS
Hipolito A. Reasonable, MD
Eleanor M. Cruz, MD
1996
Abstract
The screening procedure for the selection of applicants to the general surgery residency program of the Ospital ng Maynila was revised in 1996 in an attempt to make it more objective and structured. Choice of selection parameters was based on qualities and expectations that the screening staff wanted their prospective trainees and future graduates to have and to be able to accomplish. The use of uniform checklist-rating scales and consensus grading promoted objectivity. At the end of the screening and selection, the screening staff felt satisfied and confident that the parameters used would be predictive. A potential problem identified was sustenance of an extensive and labor-intensive screening procedure. Formative evaluation would be done in the future to further develop as well as to ensure sustenance of a screening procedure that would be found to be objective and structured.
Content
Introduction
Background
of the new screening
procedure
How the 1996
screening procedure was
conducted
The
entrance written examination and the average academic
grade
The
interview
The
practical examination
The
"pre-residency"
The
selection parameters and
process
The
rationale of the screening procedure and selection
parameters
Potential
problems
Assessment
of the screening
procedure
Extra
features of the 1996 screening
procedure
Summary
References
Appendices
1A. Interview Checklist and Rating Scale for Qualities and Potentials
1B. Interview Impression Checklist and Rating Scale
2. An Example of a Proedural Station in the OSCE
3. Rating Scale for Attudinal Competence
4. Case Presentation and Discussion Evaluation Guide
Screening procedures for selection of applicants to a surgical residency program vary from one institution to another. Conventionally, the screening procedures consist of grades from the medical school and an interview. Some institutions would add an entrance examination and an observational period usually called "preresidency". Some would also include a psychological test (1).
Although the choice of a particular set of screening procedures depends on a lot of factors, the ultimate decision rests on the people vested with the authority to accomplish the task. These people would rationalize the choice hoping at the same time that it would be effective. Only time and a formal evaluation will tell whether the screening procedure is really effective and predictive.
This paper describes a screening procedure for the selection of applicants to the general surgery residency program of the Ospital ng Maynila used in 1996. It discusses how the screening procedure was designed to be objective and structured and also its potential problems.
Background of the new screening procedure
Before 1996, the screening procedure being used in the selection of applicants to the general surgery residency program of the Ospital ng Maynila consisted of 1) grades in the medical school and Philippine Board of Medicine examination; 2) entrance examination; 3) interview; and 4) observation period or "pre-residency" of about 3 months. The decision in the selection was primarily done by the chairman and the training officer of the department and the hospital director. The basis for the selection would often be unclear and questions of objectivity raised.
In 1996, the new training officer (the senior author) of the department proposed a review. The planned change was to make the screening procedure more objective and structured.
How the 1996 screening procedure was conducted
On October, 1996, there were 25 applicants vying for 3 first year slots in the general surgery residency program. The department had 2 months to accomplish its task of screening.
The following screening methods were conducted, entrance examination, interview; practical examination; and "pre-residency" of one month.
The screening started with an entrance examination. An average academic grade was then computed from the grades in the medical school, Philippine Board of Medicine examination and entrance examination. Through a consensus of the screening staff, 15 applicants with an average academic grade of at least 71% were selected to go through an interview.
Only 13 applicants came for the interview after which only 12 came for the practical examination. At the start of the "pre-residency", there were 9 applicants. One applicant dropped out leaving a total of 8 applicants who completed the entire screening procedure.
The entrance written examination and the average academic grade
The entrance written examination consisted of 100 multiple choice questions on general surgery. Together with the grades from the medical school and Philippine Board of Medicine examination, an average academic grade was computed. This grade was used primarily to evaluate the testmanship and potential of applicant to pass the Philippine Board of Surgery written examination and secondarily, basic medical knowledge.
The interview was used to evaluate the personality, oral communication, and other pre-entry skills of the applicants. More specifically, it evaluated the essential qualities, potentials, and skills possessed by the applicants to become the general surgeon the department envisioned. Each applicant was interviewed for about 30 minutes by a panel of 3 consultants and residents who at the end gave a consensus grade. A uniform interview question set and rating scale were used ( See Appendices 1A and 1B).
The practical examination was used to evaluate the basic operative dexterity, illustrative drawing, handwriting, and written communication skills of the applicants. It consisted of objective structured clinical examination (OSCE) (2) on elemental operative skills and instructions to write a referral letter and to line draw the gross anatomy of the various parts of the human body.
A week before the examination, the applicants were told about the OSCE and its content. Six procedural stations were established. These consisted of proper operating room attire; scrubbing; gowning and gloving; handling of basic surgical instruments; hand-knot tying; and hemostasis.
In each station, an applicant was evaluated by a consultant and a resident through consensus grading and using a uniform checklist and rating scale (See Appendix 2).
In the "pre-residency", the applicants acted like first-year general surgery residents. They went on 24-hour duty every 3 days in the hospital. They assisted in operations and in management of patients in the emergency room and outpatient department. For one month, they were closely observed by the senior residents who then made an evaluation on their attitudinal competence. The rating scale for evaluating attitudinal competence of general surgery residents as recommended by the Philippine College of Surgeons was used (3) (See Appendix 3).
During the first week of the "pre-residency", all the applicants had a whole day seminar-workshop on management of a patient process and medical presentation. On a set date, they were asked to present a case and discuss it demonstrating skills taught in the seminar-workshop. A rating scale was used by the consultant and resident evaluators (See Appendix 4). The applicants were graded through a consensus from the consultant and residents.
The selection parameters and process
After the applicants had completed the entire screening procedure, their names and their respective grades in the selection parameters as well as overall ratings were tabulated. The screening staff consisting of the department chairman, training office, 3 general surgery consultants, and the chief resident then reviewed the data collected and checked the accuracy of the numerical computation. By consensus, the screening staff decided on the relative weight of the 4 sets of screening selection parameter and then recommended the top three applicants to the hospital accreditation committee and the hospital director for acceptance of the general surgery residency program in 1997.
Table 1 shows the selection parameters used and the percentage weight distribution. Table 2 shows the ratings of the applicants in the four categories of selection parameters.
Table 1. Selection parameters and percentage weight distribution
|
Parameters |
Percentage | |||||
|
Average Academic Grade
|
50% | |||||
|
Management of a Patient Process |
20% | |||||
|
Attitude and Personality |
20% | |||||
|
Skills
|
10% | |||||
Table 2. Ratings of the Applicants
|
Applicant |
Average academic grade |
Management of a patient process |
Attitude Personality |
Skills |
Overall grade |
|
1 |
35.5 |
20.0 |
14.6 |
8.00 |
78.10 |
|
2 |
36.5 |
19.0 |
14.2 |
7.88 |
77.58 |
|
3 |
34.0 |
18.0 |
16.2 |
7.96 |
76.16 |
|
4 |
33.5 |
19.0 |
14.6 |
8.15 |
75.25 |
|
5 |
34.5 |
07.6 |
14.8 |
7.53 |
64.43 |
|
6 |
34.5 |
06.6 |
15.2 |
7.01 |
63.31 |
|
7 |
31.5 |
06.6 |
15.0 |
7.61 |
60.71 |
|
8 |
32.0 |
09.6 |
11.4 |
7.20 |
60.20 |
The top four applicants in Table 2 were acceptable to the screening staff based on what they envisioned their trainees to possess and graduates to be. However, there were only 3 slots available. Analyzing the ratings of the top 3 applicants, Applicant 3 was highest in attitude and personality; Applicant 2, highest in academic grade; and Applicant 1, in management of a patient process and skills. In the parameters where they were not highest, the applicants also had good grades.
The rationale of the screening procedure and selection parameters
The screening staff envision its graduates to pass the written part of the Philippine Board of Surgery (PBS) examination as the first step in becoming a diplomate. From experience of the screening staff, passing this part of the PBS examination depends largely on testmanship of the examinee. In a study of Almonte in 1996 (4), it was shown that the general weighted average grade alone, which was derived from the academic grade in medical school, was a good predictor in the general surgery residents in-training written examination. Thus, the screening staff decided to use the average academic grade as a measure of testmanship and a parameter in the selection.
The screening staff envision also its graduates to pass the oral part of the PBS examination. From the experience of the staff, oral communication skills and prompt, rationale thinking, problem-solving, and decision-making processes contribute to passing the oral examination of PBS. These qualities were evaluated in the interview and the case presentation and discussion and were used as parameters in the selection.
Stable, mature, and humane personality; honesty and ethical values; motivation; proper study and work habits; interpersonal skills; communication skills, both oral and written and legible handwriting; rational thinking, problem-solving, and decision-making skills; basic operative dexterity; illustrative drawing skills; and medical presentation skills are important attributes of competent, effective, efficient, and humane general surgeons. The screening staff would like to see these attributes in their graduates. These qualities were evaluated in the interview, practical examination, case presentation and discussion, and one-month observation period.
Teaching as a rule is not easy and since practically all the consultant staff of Ospital ng Maynila are part-timers, the screening staff decided to select applicants who were quick to learn and would be easy to train. These qualities and parameters could be assessed from all the evaluation methods but most tangibly observed in the case presentation and discussion where the applicants had a prior seminar-workshop training.
The selected applicants would be expected to work harmoniously with the present resident staff. Interpersonal skills; reliability and responsibility; ethical and moral values; intellectual integrity; and proper study and work habits are attributes for a harmonious relationship. These qualities and parameters were evaluated in the assessment of attitudinal competence by the senior resident staff.
The 1996 screening procedure could be said to be extensive in preparation and labor intensive. This could pose as a problem in sustenance in the future.
1. It utilized 5 screening procedures and 4 sets of selection parameters.
2. It covered a 2-month period with 6 formal screening sessions requiring the presence of consultant staff (interview; practical examination; 3 meetings for the case presentation and discussion; and deliberation).
3. It required a one-day seminar-workshop on management of a patient process and medical presentation.
4. It conducted training of the consultant and resident evaluators to promote validity and reliability of the results.
5. It needed resources to prepare and check the entrance examination; to prepare for the interview and OSCE; and the procedure checklist and rating scales, interview guides, and hand-outs for the seminar workshop.
Assessment of the screening procedure
The screening procedure for applicants to the general surgery residency program of the Ospital ng Maynila was revised in 1996 with the objective of making it more objective and structured. At the end of the task, the screening staff felt satisfied and confident that the parameters used would be predictive.
The 1996 screening procedure could be said to be structured in the sense that what the screening staff envisioned the graduates to be and to possess were clearly spelled out and screening procedures and selection parameters were made on this basis.
The 1996 screening procedure could also be said to be objective in the sense that uniform screening methods and assessment tools were utilized. The use of checklists and rating scales promoted objectivity of the assessment tools. So was the use of consensus grading.
The screening procedure is definitely more complicated than the usual procedure using academic and interview. From the experience, however, the screening staff felt that they got a clearer picture of the qualities and potentials of the applicants more than what could be gotten just based on academic grade and interview alone. The observation period gave a clearer picture of the attitude and personality of the applicant. The OSCE added validity to the screening procedure considering the psychomotor skill required of the surgical profession. The other practical examinations gave a clearer picture of the pre-entry skills of the applicants in terms of communication and medical presentation. Lastly, the case presentation and discussion gave a tangible picture of problem-solving and decision-making skills, study habit, and learning rate of the applicants.
A potential problem identified is sustenance due mainly to its being labor intensive. A formative evaluation is being planned on a year to year basis so as to make the screening sustainable, at the same time improving and maintaining its objectivity, validity and reliability.
Extra features of the 1996 screening procedure
As implemented, beside being an objective and structured screening method, the 1996 procedure was a learning experience to the consultants, residents, as well as the applicants. The consultants found a better way of screening applicants. The residents actively involved in the screening learned in the process, especially on the patient management process, basic operative skills, and medical presentation. For the applicants it was an orientation to the hospital and the department of surgery cum training in basic surgical competence.
A design in making a screening procedure for the selection of surgical trainees objective and structured was described. The rationale for choosing the screening procedure and selection parameters was discussed. Complexity that resulted in the pursuit of a more objective and structured screening procedure could pose as a problem in sustenance. Formative evaluation will be done to develop a valid, reliable, and practical screening procedure.
1. Communication with Dr. Arturo dela Pena, 1996 Chairman of the Residency Committee, Department of Surgery, Philippine General Hospital.
2. Harden RM. What is an OSCE? Medical Teacher 10(1): 19-22, 1988.
3. Surgical Curriculum for General Surgery. Philippine College of Surgeons, 1995.
4. Almonte, JR. Evaluation of the residency training program in general surgery of the UP-PGH Medical Center. 1996. (Unpublished).
Appendix 1A. Interview Checklist and Rating Scale for Qualities and Potentials
Applicant _______________________________ Total Score ___________
The responses show that the applicant has the:
|
|
N |
D |
Y | |||||||
|
|
0 |
1-2-3 |
4-5 | |||||||
|
I. Essential qualities that must be present
|
|
|
| |||||||
|
II. Potential and pre-entry skills
|
|
|
| |||||||
|
Total Score |
| |||||||||
* N = No; D = Doubtful; Y = Yes
Remarks and Comments:
Appendix 1B. Interview Impression Checklist and Rating Scale
Parameters:
1. General Appearance (general impression created by dress, grooming and health)
2. Friendliness/Likeableness (reflected in manner, voice, gaze, and bearing)
3. Poise-Stability (maturity, bearing, sense of balance)
4. Verbal Fluency (ability to express; method/manner of speech)
5. Mental Alertness (ability in comprehend questions speedily and anticipate interviewer's thoughts; capacity to transfer attention from subject matter to another; reaction time)
6. Enthusiasm/Drive (high motivation for a surgical career)
Note: For each parameter, there are 5 scales, 1 to 5, with 5 being the highest possible grade. Each scale has a description of behavior to guide the evaluator.
Example:
Mental Alertness
5- Keen, alert mind; grasp ideas quickly
4- Perceives most points readily; above average mental alertness
3- Normally alert
2- Rather slow at times; may require repetition before understanding
1- Slow to respond; waits too long before answering
Appendix 2. An Example of a Procedural Station in the OSCE
Station N: Hemostasis
For the Applicant:
Scenario:
You just made an incision through the skin, subcutaneous tissue, and deeper tissues.
You encounter a bleeding cut vessel in the subcutaneous layer. This bleeding cut vessels is represented by a silk suture in a subcutaneous tissue in a pig leg,
Instructions:
You are to control the bleeding with the use of surgical instrument and then tie it with a suture.
For Use by the Evaluator:
Examinee: ____________________________________________
|
|
|
|
|
Checklist and Rating Scale | ||
|
|
|
|
|
Yes |
Somewhat |
No |
|
|
|
|
|
3 |
2 |
1 |
|
Clamping at the tip of the suture |
|
|
| |||
|
Tip of the clamp used |
|
|
| |||
|
Secure knot |
|
|
| |||
|
Gentle maneuver |
|
|
| |||
|
Steady |
|
|
| |||
Remarks:
Evaluator: ____________________
Appendix
3. Rating Scale for Attitudinal Competence
(Adapted from the Philippine College of Surgeons)
Parameters:
1. Intellectual Integrity
2. Moral/Ethical Values
3. Reliability/Responsibility
4. Bedside Decorum/Relationships with Patients
5. Study/Work Habits
6. Relationship with Co-health workers
7. Emotional Maturity/Reaction to Emergency or Stress
8. Social Responsibility
Note: For each parameter, there are 4 scales, 1 to 4, with 4 having the highest possible grade. Each scale has a description of the behavior to guide the evaluator.
Example:
Reliability/Responsibility
1- Irresponsible, unreliable; needs repeated reminders of assignments; does less than prescribed work.
2 - Usually prompt but does just enough to get by; usually dependable but sometimes needs reminders of assignments.
3- Performs duties promptly and efficiently without being reminded
4- Performs duties promptly and efficiently without being reminded; is resourceful and innovative; takes initiative to spend additional time.
Appendix 4. Case Presentation and Discussion Evaluation Guide
Expectation:
Within the time alloted, the presentor is expected to present a case discussion
|
Criteria |
Rating (3-2-1) |
Feedback | |
|
CLEARLY |
|
| |
|
Communication |
|
| |
|
Audiovisual aids |
|
| |
|
Organization |
|
| |
|
Adequately (content) |
|
| |
|
Management goals |
|
| |
|
Rapport |
|
| |
|
Clinical diagnostic process |
|
| |
|
Paraclinical diagnostic process |
|
| |
|
Treatment process |
|
| |
|
Indication for referral |
|
| |
|
Advice |
|
| |
|
EFFECTIVELY |
|
| |
|
Audience comprehension |
|
| |
|
Rational management process |
|
| |
|
EFFICIENTLY |
|
| |
|
Management of time, resources, etc. |
|
| |
|
Others: |
|
| |
Key:
3 - very satisfactory
2 - fair
1 - unsatisfactory