we are conducting thishere at opm's headquarters in washington, dcin our auditorium. we do have a fewpeople here today, but the majority of our audienceis watching us via webcast and so i just wantedto welcome all of them. but i also wantedto mention that the slides that we'regoing to be speaking to today are attached tothe omb max web page, where everyone was toregister for this session.
so you can goback to that web page, just scroll down to thebottom and you'll see a little linkfor the attachments, if you need to access those. and at the verytop of that page, you can find the email address where you can submityour questions to us today. my name is tara ricci. i'm going to bepresenting this session
along with mycolleague julie osowski. we are personnel researchpsychologists here at opm. we work in thestandards, competencies, and assessmentdevelopment group. we collaborated on astructured interview guide. that guide formsreally the basis for the trainingthat we're doing today, but it's alsoavailable on our web site so i encourage youto go to our web site,
and downloada copy, if you like. one of the very lastslides of our presentation has the web address that youcan go to, to get those slides. i'm going to quickly go overwhat we're going to cover today, in today's session. we have five sections. we are going to talk first, just give an introduction towhat structured interviews are, what makes themstructured, and talk about
how to develop the structured interview questions themselves. we have two differenttypes of interview questions that we'll talk about. then, we'll talk abouthow to actually conduct structured interviews, usinga panel of trained raters. of course following that,we'll talk about how to evaluate candidates who complete thestructured interview, using a standardized rating scale and rating process,
and then also just somegeneral tips and techniques for conducting interviews, that are helpful toshare with anybody who is going to be a partof the interview process. so, what is astructured interview? it's basically any...it's an assessment method. if you attended our firstsession, i talked about lot of different typesof assessment methods that are used,and this is one of them.
it can be used as a part of therating and ranking process. the structuredinterviews are also used just by the selecting official. the main difference is that, if it's being usedby selecting official, it doesn't necessarilyhave to be scored. it can be scored, but that'sreally the main difference. if it's being used as a partof the assessment process, a part of therating and ranking,
then there will needto be a scoring process that's a part ofthe interview itself. the reason it's structured is because it does usea very standard question and scoring process,and i'm going to talk about what really makes up thisstandard structured process that we talk about. so, those of youwho are here today, how many of you have experiencedusing structured interviews?
okay, we have a fewpeople, that's great. i think that they'rebeing used more and more, in the federal government we're certainly usingthem a lot in our office, and it's reallyworking out well. i really think that the peoplethat go through the process think it's very fairand makes sense to them. so i'm glad to seethat so many of you're wanting to participatetoday and learn more about it.
so, some of the keystandardized features; all the questions are developed by subject matter experts, and the questions aretargeted to competencies that have been shown tobe required for the job. so it's meeting those importantkey features that you need in any assessment methodthat's based on a job analysis. now, all the candidates aregiven the same opportunity to provide information aboutthemselves because they're being asked the same questions,in the same order.
everybody is exposed to thesame assessment process. we have a panel oftrained interviewers that are taking detailednotes on the responses. this provides criticaldocumentation for the scoring process. the responses are evaluated,using a standard process and the interviewers agreeon what acceptable answers are. so, basically everybody isbeing held to the same standard, which ensures a fair process.
this is very differenton unstructured interviews where the questions can vary,based on who comes in the door. but the real purpose of astructured interview is that you have definedthe competencies that you want to assess, and you go througha very standard process for everybody who is involved. so why is structure andstandardization important? there are several key featuresthat you want to look for
when you're evaluatingan assessment method. and these are some of thethings that we talked about during our first assessment,training session. first thing, ofcourse, is validity. does the assessment method measure what it'sintended to measure? structured interviews actuallyhave pretty good validity compared to otherassessment methods. they are really one of the best.
so, you really, because you'refocusing on job requirements, and the questionsare being assigned by subject matter experts, if you'refollowing all the steps in developing a goodstructured interview process, they really dohave good validity. so you have that... but inaddition they're very reliable. the content, the rating scales, they're standardizedacross all the candidates.
so everybody is beingassessed in a consistent way. so, if you remember reliabilityis an important piece of getting that validity, so youhave the reliability as well. it's fair; candidates aregiven the same opportunities to provide information, and they're objectively evaluated on the same scale. so again, it's definitelyperceived as fair by people who gothrough the process. and it's practicalbecause you're focusing on
the few key competenciesthat are important for the job. so you're really focusing, and you're takingadvantage of this opportunity to really gather asmuch information as you can from the candidateon the competencies that you're targetingfor assessment. so that's an overview of whatstructured interviews are. they are a really greatassessment method to use, but now we need to talk about,
now if i want to usea structured interview, how do i goabout doing that? we actually haveabout eight key steps that go into developing structured interview questions, and a structuredinterview process. i'm going to talk abouteach of these in turn, and again i would refer you to our structured interviewguide for more information. the first thing andthis is the first step
in designing anyassessment method is you need to haveyour job analysis. in many cases, you mayalready have a job analysis that's available. of course, if it's nota current job analysis, we would definitelyrecommend that you have your subject matter experts review the information, and confirm that the job tasks and the competencieshave not changed.
and if they have changed,how have they changed. if you do nothave that information, then you'll need todo the job analysis, and of course,you can refer to the delegated examiningoperations handbook that gives you all theguidance that you would need on how to conducta job analysis, including the formsthat you can use. so you have donethe job analysis.
typically, you're goingto have more competencies as a result of the job analysisthan really is feasible to assess in thestructured interviews. so you're going tohave to figure out, well, which of the competenciesdo you want to assess. first you might want to consider what competencies are goingto be assessed elsewhere in the selection process, so maybe you canexclude some right there
that are already goingto be assessed elsewhere. the other thing youmight want to consider is that thestructured interview is, one of the thingsthat's really good for, is assessing things thatwe call like soft skills or high touch competencies, things that are morelike lnterpersonal skill, teamwork, customer service, those types of competencies.
they are ideal for thosetypes of competencies. and we do typicallyrecommend that you assess between four andsix competencies. much more than thatand you run the risk of the interview justbeing too long. we estimate it takes maybe... you want to give the candidatesat least five minutes, and sometimesmore to respond to each and every of your question.
so once you start adding thatup, it can get to be long because you also have tothink about you're going to have time upfront whereyou're giving candidates an overview of theinterview process, and then at the back end, you're going toallow time for them to ask questionsabout the position, and the organization and such. so it's really just figuringout what is feasible.
you do not want theinterview to run too long, particularly if you're goingto have a lot of candidates that you're goingto be assessing. i can also suggestthat you go to opm's assessment decision tool. it's on our web site. this is for those of you who did not participatein our first session. we talked alittle bit about that,
but the adt canbasically help you, based on the competenciesthat you want to assess, help you identify whatassessment method is best suited forthose competencies. and it actually canwork the other way, where if you know you wantto do a structured interview, it can help you figure out, well, what is thestructured interview? what competencies isit best for assessing?
so i do recommendyou look at that. now i'm just going totalk about some general tips on developinginterview questions. the first thing that youwant to think about is, of course, you haveidentified the competencies that are important for the job, are the ones you want to assessthrough the interview process. but in terms of justwriting interview questions, there are certain thingsthat you want to keep in mind.
first thing is that thequestions are open-ended. the interview is really intended to gather informationfrom the candidate so you certainly wantto write questions that are going to enable themto provide that information. so using open-endedquestions is critical. clear and concise questions; you do not want questionsthat are too lengthy. you want them to befairly straight forward
because you certainlydo not want to have people interpretingthem in different ways. so it's always important that you just be veryclear and concise. think about the reading level, depending on the positionthat you're filling, if it's an entry level job, you may need to think about thereading level of the questions. you do not want to use language
that's a littlebit more complex. just reallyusing simple language. so, for example, insteadof saying something like tell me about a time,you affected a policy change, you can make it much simpler, and just say tell me abouta time you changed policy. it's just simple things likethat, using plain language. and the other thingwe talk about is keeping things free of jargon,
especially in thefederal government we have lots of acronyms, terminology that the longer you'repart of federal government, the more thatjust becomes ingrained in your day-to-day work. but if you're doing an interviewprocess where there may be people coming from outside,you just want to think about have i used appropriatelanguage for those people. will they understandwhat we're talking about?
and the other thing thatwe recommend is to consider using superlative adjectivesin the questions themselves. and what thisbasically means is that instead of saying tellme about a time, you say, tell me about the besttime or the worst time, tell me about the bestproject you ever worked on? tell me about the most difficultteam you ever had to lead? when you use thosetypes of words that can help the candidate,trigger in their mind,
oh! yeah, i know,i can tell you exactly about the hardestproject i ever had to lead, or the most difficultteam i ever had to work on. so it just helps thecandidate in coming up with a good example thatthey can share with you. and that just helps them,and helps you as the rater to get the bestinformation out of them. one of the things thatwe were going to touch on throughout the session todayis what we call the star model.
and this is just themodel that we use in developing both the interview questions themselves, but also the follow-upquestions that are used, and even the rating materials. and the star model really laysout the three key elements that we look for in definingthe interview questions, and what we would expect tohear back from the candidate, and their responseto those questions. so, these questionsshould be designed
to elicit the threeimportant piece of information. the first is thesituation or the task, the context or thebackground of the situation. so it sets thestage for the experience that they're aboutto share with you. the action that's whereyou're actually getting out what the candidate did,what their role was, how they dealtwith the situation? and then, the result isof course the consequence,
what was theoutcome of that event? so we really look for thatthroughout this process. and it's somethingthat we'll refer back to, to keep in mind throughout. there are two types ofstructured interview questions that we use,and that we talk about in our structured interview guide. the first is behavioral,and these are questions that actually draw from acandidate's actual behavior
during past experience. so these are questions that areactually asking the candidate to tell you about their experience or education or something thatthey did in their past that helps demonstratethe competency that you're trying to assess. and in this case, it's a verycommon promising psychology that the past behavior is the bestpredictor of future behavior. now for thesituational question,
this is a little bit different. it's no longer asking them toshare about their experience, or behavior, in the past, but we're actually posinga dilemma to the candidate, a problem,a hypothetical situation, and they have to then respond on how they wouldhandle that situation. so they're gettinga different things, but they're both very effectivein assessing competencies.
and the one thingi would mention is that the situationalquestions might be better for people who have entry level, who don't have lot ofexperience to drawn. so it might makemore sense to uses for those entry level positions, where of course behavioralwould be ideal for people who have experience formaybe higher level position, or you can use both.
they are both useful, you get good informationout of both of them. so now we've talkedabout some of the guidelines for developinginterview questions, the star modelabout things being open-ended, clearand concise language. now it's time to actuallydevelop the questions themselves and this is where you'll needto call on the assistance of subject matter experts.
typically, you'll need toconvene a panel of maybe 6-7, much more than that, it mightget a little bit too difficult to manage that much input, but usually a panel6-7 is effective. you want to spend some timewith the subject matter experts, we sometimes call them smes, just letting them getfamiliar with the materials. the competencies thatare going to be assessed, the job task that are associatedwith those competencies.
you want to of coursetalk about the star model and other general tips forcreating interview questions. so they have a goodunderstanding of what the purpose is, if you're going to usejust behavioral questions or if you're goingto use situational, all of that needsto be presented to the subject matter experts, so they can get understandany of the purpose.
but their role is becausethey know about the job, and they're going tobe in the best position to craft these questions that are appropriatefor the position. so it's just a little bitdifferent for each one. in both cases, you're going toneed subject matter expert, but the process is going tobe a little bit different. for the behavioral questions,you're going to want to have them brainstormsome of experiences
that they mayhave had themselves and they can do this as a group, but they'll havethe competencies. it would be ideal to getthem a few sample questions that they can referback to as a model, and just let them spendsome time as a group, talking aboutand just brainstorming, different ideasfor what would be a good behaviorallybased question
for the particularcompetencies that you're focusing on. and it's always agood idea to have them generate more than you're actually going to need, because as you gothrough the review process and the pilot testing process, some of those questionsare going to be discarded. so it's always a goodidea to have more. and again,just have them refer back to
the tips on developinggood interview questions. so here is an examplefor behavioral question and this is based on the competency ofinterpersonal skills. and this is justa good way to illustrate how the competency and thecompetency definition itself is really a good material for the subject matter experts to use in craftingthose questions. if you look at the definition,
it covers a lot of different elements of interpersonal skill, that those subject matter experts can then pull from to draft their questions. so in this case,now the example question is, describe a situationwhich you dealt with individuals who are difficult,hostile or distressed and you can see that comesstraight from the definition. the other thingi wanted to point out, is that the question itself
and this is what wecall the lead question, we'll also talk aboutfollow up questions, but this lead question covers all three elementsof the star model. it's got the situational task, dealing with individuals who are difficult, hostile ordistressed, who is involved. it gets at the action. what were the specificactions that you took and then it has the results,and what were the results.
so it has all threeelements in there. has anybody in the audience choose behavioralquestions before? are there any examples, anybodywho would like to share? we have microphones, if anybody? no. okay, yes. so what parts canthen streamline or well, for thestructured interview
that's just been used bythe selecting official, they wouldn't... they're certainly not going to be doing the scoring piece, that they don't need to. so they can skip the process of having to giveindividual scores. with the selectingofficial, the rating and ranking hasalready been done. so these people have alreadybeen basically assessed and
demonstrated that they'reready for the job. it's just a matter for theselecting official to figure out what information they needto make that final decision. so the questions, we stilladvice using the same questions for everybody but youwouldn't necessarily have to go through thesame process of working with subject matter experts to develop the rating skills, do all of that piece of it. you could just decide,
what are the competenciessomeone has assessed during the structuredinterview piece and just go from there. there doesn't necessarily haveto be the rating and ranking. so from the eight key steps i would focus on determiningthe competencies and justdeveloping the questions. it might still be agood idea to pilot test. that's one of the key stepsout of their eight steps
that we're going tobe talking about, so they might want to alsodo that piece as well. let's talk about situational and developing situationalinterview questions. now this process is going tobe a little bit different. you are actually going to needmore subject matter experts to develop situationalquestions, because you'll have two panels of subject matter experts. the first panel isgoing to follow
what we call thecritical incident method and this is a commonmethod that's used in assessment development because it providesgood information that can be used to developassessment questions, test questions. and basically, it hassubject matter experts go through a processwhere they think about really effective or really ineffective events
that have occurred,typically in the work place. so they'll just spend sometime really telling a story of what this criticalincident is. so this first panel ofsubject matter experts, that's really all they're doing. they have been provided withthe list of competencies that are going to be apart of the interview and then what they'rejust going to do is think about these effectiveor ineffective events
that have occurredin the workplace, and they'll writea story about it. and it pretty much followsthe same star model. they'll talk about thesituational task, what the action was, whether itwas effective or ineffective and why and then what theend result would be? and then in the case of anineffective critical incident, they'll talk about whywas it ineffective and what could havebeen done differently.
in our structuredinterview guide, we do have the formsthat you can use for those subject matterexperts to tell these stories. it has all the keyelements laid out. so the first panel writesthese critical incidents. it's then important tohave another group kind of review and confirmwhat they have come up with. and so the second group is going to review thesecritical incidents,
but they're not goingto look at them in the context ofthe competencies. actually their job is to thenassign the critical incidents to the competencies and that'sjust a confirmation step. so it's just a wayto make sure that what this initial group thought measured interpersonal skills, it really does measure interpersonal skills. so they go through and theymatch them to competencies. the criticalincidents that match
to the appropriatecompetency are retained. the ones that aren't agood match are discarded. so again you're goingto want to have the smes develop more than you're going to need and then once they havefinished doing that process, the second group then actuallywrites the situational questions and they're basicallyhypothetical situations and those subjectmatter experts, they can work as a group
because they really have thebest knowledge of the job itself and they can really craftwhat is a realistic dilemma or situation for thecandidate to respond to. so here is an example, again working with competencyof interpersonal skills. so here is the scenario. a very angry clientwalks up to your desk. she says she was told her officesent her an overdue check five days ago.
she hasn't received the check. she has bills to pay, andno one is helping her, how would you handlethe situation? so again, it's just ahypothetical situation. it's realistic of whatmight occur on the job and it's typed to thespecific competency so that's the process ofdeveloping the questions, the lead questions themselves, but another key pieceof the interview
are the follow-up questions, what we call the probequestions because it maybe that candidates don'tprovide enough information based on that lead question or you need toclarify information or try to get morespecific information out of the candidate.so you want to decide when you're developingyour structured interview, if you want to have thesefollow-up questions and if so,
you'll need to come up witha set of these questions that can be used foreach question in the interview. what i would say about theprobe questions is again, you're going to wantto make sure that you're trying to get it allthree elements of the star model the situation, theaction and the result. but we also have justsome general dos and don'ts for developing probe questions. you don't want toleave the candidate.
so the goal of the probe is youwant to get the information you need withoutnecessarily leading them to a particular answer. you do want to get at theirspecific role or action. sometimes when acandidate is responding to a structuredinterview question, they're talking about a projectthat they were a part of, the team that they were on, but not necessarilygetting real specific
about what their role was. and so the probequestions can be useful and following up on that; getting at what theirparticular role was. you don't want tochallenge them. you want these probequestions to be written in a very neutral way. so instead of sayingsomething like, why didn't you trydifferent approaches?
why didn't you try adifferent approach? you could just make itmuch more neutral and say what differentapproaches did you try? and then of course,in most cases you want the probequestions to be open-ended because you're reallytrying to draw out some additional informationfrom the candidate. so, we just have some samples for following the star model and
for both the behavioral andthe situational questions. so the probes for the behavioralquestions are very much - because this is the personwho is talking about their actual experience, they're very specificfactually; what was involved, what factors led up tothat particular situation, what was your role, howdid you respond? and you can actually, once you have developedyour questions themselves,
you can actually makethese a little bit more specific to the question.these are fairly general. and then we have the situational probes and because these aretargeted towards a hypothetical situation, they're just going to bewritten to be more appropriate for that type of question. so, does anyone in the audiencehave any example probes
that they would like to share? have you used probe questionsin your structured interviews? no, okay. alright. so you have developedyour questions and your follow upprobe questions. so you have gotall that laid out. the next key thing thatwe'll need to work on is deciding how you're goingto evaluate candidates on the questions.
so you'll need to come upwith your rating scale and the associated materialthat goes along with that. so the first thing is todecide on the range for the proficiency levelscale for the rating scale. we usually use between 3,we recommend using between 3 or 7 rating points. if you use fewer than 3 you're probably not going toget enough distinction among the candidates, butif you go much above 7
than sometimes the interviewers find it more difficult to make those foreign distinctions. and in some cases i have seen rating scales that are 9 levels, but then even withinthose 9 levels, they have been collapsedinto sub-level. so, it just seemslike between 3 and 7 it seems to work thebest for the interviewers. you want to label at leastthree of those levels. if you're going to beusing more than 3,
just really you want toanchor the lowest, the mid and the highestlevel of response. now at opm we do use a 5 levelproficiency level rating scale for competencies. level 1 is awareness,that's the lowest level. that's for somebody whocan apply a competency in various simple situations, but they're going to needa lot of guidance. and the highestlevel 5 is expert.
and in that situationthat somebody who can apply competencies in extremely difficult situations and they're really seenas the key resource, people go to them for help in applying the competency. so, that's the scale we use. any examples from the audience? do people use like a 3level scale or a 5 level? yes, five levels, yeah. yeah that seems towork well for us.
so once you have decidedon your rating scale and the number of levels, you'll need to developexamples to go along with that. the examples are veryhelpful for the raters, the ones who areconducting the interview and actually evaluatingthe candidates on the rating scale itself. and so in this partof the process, you're going to once want to go to your subject matter experts
and get them to help you generate these examples, because the examples aredrawn from the job itself. so if it's a position,say in engineering, you want to have examplesof what an expert level in that particular competencyis in the engineering field. and so they're really theones that can generate those types of examples that you can then linkto the different levels and the rating scale.
and it's usuallygood idea to use the same subject matter experts because they're already familiar with the process and what the goal is andwhat you're doing. so they all individuallycome up with these examples. they can discuss it as a group. it's important thatthey reach consensus so that everybody has acommon understanding that indicates thatit's good example,
if everyone is inagreement on, yes, this is an example ofan intermediate level, yes, this is an exampleof expert level. so, to get them to discussit and come to consensus and then you can just link those to the rating scale itself and you put thatinto the rating form that will be used bythe interviewers. and really for thesituational questions, it's very much the same.
you want to have the subjectmatter experts come together, and talk about what it means to really answer the situational question at the highest level. and they can just spendsome time detailing what would they reallyexpect from somebody who answers this questionat the highest level versus what wouldyou expect of somebody who didn't quite addressall of the elements that are important to answeringthe situational question.
and so they again, cando that as a group, spend a little time working on drafting thesehypothetical responses, independently discussingit as a group and then coming to consensus on what are the best ones to use then again you can just putthat into your rating form. that's the process fordeveloping the interview itself, but it's not everything because particularly if you'reusing an interview
as a part of theassessment process, there is more that goes into. you have to figure out howthe whole process itself is going to work. and so at first, one of thefirst things we recommend is that you pilot testit to make sure that the question are working, but only the questions,but the whole process itself, particularly if you're goingto have quite a few interviews
being conducted, maybeone after the other, you want to work outthe whole process and make sure everythingis running smoothly and that everybodyunderstands their role. so we definitely recommenddoing some type of pilot test. but the other thingthat you want to do is you want to develop an interviewer's guide. the interviewer's guide isreally the resource that those interviewers willuse throughout the process.
it will have the questions, the competencies thatare being assessed. it will have rating forms thatthey will use at the end when they're makingtheir ratings. it has the instructionsthat they will use when they're talking tothe candidates and tell them or walking them, this is what is goingto happen here today as a part of thisstructured interview,
it has got all of thatlaid out for them. so there won't needto be some time spend just coming up with this interviewer's guide and again we do have checklist, list that you can refer to, sample forms in ourstructured interview guide. some of the other things thatyou'll need to think about is for a panel interview where it's being done aspart of the rating
and ranking process. you need to decide how manyinterviewers you're going to use and if those are going to be... if you're going to have the same interviewers for each panel or if you're going tolet different people serve on different panels, if you're going to havea lot of applicants and you're going to berunning a lot of people through structured interviews, it may not be feasible
to have the same interviewers conducting eachinterview because they just won't havetime, i mean, everybody hasother responsibilities. but the other important thing isthat everybody gets trained and that's what the remainder ofour training session is going to be on, is the actual training of the interviewers. but you'll need to decide that,how many interviewers and how they're going tobe on each panel,
but the other thing is,the final rating process. using a panel interview, each person is going to bemaking their independent rating, but then you'llhave to decide well how we are going to do afinal score for the candidate. you can average those ratings or you can go througha consensus process where you require thepeople who served on the panel to discuss theirindividual ratings
and then come to consensus and there are differentways you can set that up, different rules you can use. in some cases, if you're justhaving two interviewers you may want to require thatthey come to perfect agreement, meaning that they bothagree that the individual scored at a level 3 onthe interview question. if you're having morethan two people, sometimes it's okay to say,
okay, everybody has tocome within one point and then you could justtake the majority. so if you have got threepanelists, two of them scored, the person at 3 ison the question, the other one who scored at 2, that's considered consensus. but, if you find that thereare more of a range than that, typically we wouldrecommend that you go through aconsensus process where they have to discussthe scores that they gave
and how they came up with them. and the reason that we do that'sbecause structured interview is, it's a live event and theinterviewers are documenting what's going onduring that event. but they have misseddifferent things or they may have interpretedthings differently. so when you use aconsensus process that gives them the opportunityto discuss as group, share what theyhave come up with
and maybe clarify thingsamongst themselves to make sure that everybody was using the same information that was shared duringthat interview because again, you can'tcapture everything. that's what nice abouthaving multiple interviewers. i did mention that the trainingis going to be very important. the remainder of our sessionis going to be really in the context of trainingthe interviewers. so, for those of you whomight be in a position
where you're developing a structured interview, and then you're going to haveto get people up to speed to serve as interviewers, we do recommend that you can usethe remainder of the slides to conduct your owntraining at your agency. and then just the last stepis to document the process. anytime you're developingan assessment method, you're going to want todocument the process. you're going to want todocument who you helped,
who your subjectmatter experts were, how you came up with theinterview materials, what references you used, then you have the job analysis, all this just goes intomeeting on the requirements of developing anassessment method. so you want to make sure youhave got that documentation. so at this point, i havewalked you through, developing the interview itself.
before i turn things over tojulie, i just want to see, if there is any questionsin the audience, yes? well for a structuredinterview, yeah. the question was, when do youuse the structured interview and when do you don't use it? the thing with the structuredinterview is that, it's time and it can be timeand resource intensive. so it's maybe not somethingyou would want to use as a first hurdle or as a first part of the assessment process.
you may want to use ita little bit later on. that's just themain consideration. it's just thinking about the amount of resources that you'll haveto bring to bear to do the structured interviewsand making sure that you can manage the applicantsthat you're going to be getting. so if you're going to getting way too many applicants to really feasibly do a structured interview, then i wouldn'trecommend doing it;
that's the first thing, think about using itlater on in the process. yes? yes, that's a good point. the structuredinterview questions; it's the assessment,it's a test. it's what the applicantsare going to be... all of the applicantswill be responding to. so it's important to when you'retraining the interviewers to
talk about maintaining thesecurity of the materials. it's a live test.it's being used. so you want to want totrain your interviewers that they need to maintain the materials in a secure place and not share the information. but you also... when you'redeveloping your interview materials, you want to instruct that the candidatesthat they too, should not be sharing thisinformation with others;
that's part of theassessment process. so, it's very important aswith any assessment method to make sure that you'reputting the protocols in place to keep everything secure. so, any other... yes? well, it's ideal. the question was, is it notappropriate to use these for technical positions? and the structuredinterview is ideal for
competencies that are morewhat we call high touch or soft skills, but itcertainly can be used for technical competenciesor technical positions. it's just a matterof figuring out what are the most competencies that you want to assess and if indeed those are theones that are most important, it's certainly okay to useit for those competencies. any other questions? no, okay.
alright, well thankyou very much. i am going to turnthings over to julie and she is going to talk about conducting thestructured interview. thank you. good afternoon, my nameis julia osowski i am a personal researchpsychologist here at opm. as tara alluded to, the remainder of today'spresentation
is going to go over aprocess for conducting and evaluating structuredinterviews, just considered astraining material. i will begin with abrief overview of the interviewer's role. then i will discussone procedure we have found to be effective for conducting and evaluatingstructured interviews. i will close withseveral general tips
for ensuring a soundinterview process. please keep in mind, again, that the informationi am going over is geared towards interviewers. so whether you're listening because you will beconducting interviews or you're going to beconducting interview training or because you'rejust interested, please turn on or put on your interview hats for now.
the job relatednessand standardization of the structuredinterview content provide a critical foundation for interview effectiveness. however they'rejust the beginning. the effectiveness of structuredinterviews also heavily depends on a good and standardizedadministration process. therefore you, the interviewershave a critical role. first and foremost you want tocreate a comfortable environment for candidates.
so that they will feel at home and relaxed sharingdetailed information about their job relatedbehaviors and intentions. most candidates are atleast somewhat anxious when they come for an interview. your behaviors can definitelyhelp reduce this anxiety. for example, justbeing nice, polite and respectful willgo a long way. also remember that ahost of information
is going to be providedby the candidate in a short time frame. it's critical foryou to be focused, to give the candidateyour undivided attention and to take good notes, so that you can capturea holistic picture of the candidate andevaluate his or her responses most appropriately. finally leave any personalbiases at the door.
you are responsible for treatingall candidates the same for being objective throughoutthe entire interview process. the way that you actis also important because you represent thejob and/or the agency and while you're going to be interviewing candidates, they're going tobe evaluating you. they're going to formimpressions about the job and about the agency andtheir desire to work there from your professionalism andinterpersonal skills and
competence as an interviewer. so be prepared andact professional. remember to make sure that youhave all relevant materials prior to every interview, dress appropriately, know how to pronouncethe candidate's name and use it frequently. make frequent eye-contact and use an encouragingtone of voice.
and most importantly makesure that you monitor your body language and only use appropriatebody language. for example, don't frownor raise your eyebrows when candidates respond, don't slouch, doodleor look disinterested. there are several logisticaland administrative duties that needs to be completedprior to interviewing. we have found it help todesignate one individual
to be the chairpersonand to ask him or her to be responsible for these behind the scenes duties. examples of logisticalfunctions include verifying that the interview lengthand setting are appropriate. we'll get into more detailsabout these shortly. examples ofadministrative functions completed by the chairperson include verifying that all interviewers have been trained. they know the candidateinterview schedule
and they have allnecessary materials, including the interviewguide, the competencies, their definitions, thequestions, the rating scales, and all of the rating forms. it should be verifiedby the chairperson or another individual that each interview has been scheduled for the same length of time and that this length oftime is appreciate. the length of time shouldallow for introductions,
and interview instructions, responses to every question, an opportunity for the candidate to ask information about the job or about the agency andthe rating process. we found in recent interviews that it takes approximately ten minutes for introductions instructions, about ten minutesfor a response to one question, about ten minutes foran informal discussion about the job and 15-30 minutes for the evaluation process.
naturally, the length of yourinterview is going to vary depending on how manyquestions you ask. prior to the interview,the chairperson or another individualshould also verify that the interview settingis private and quiet and accessible to candidateswith disabilities. this is consistent with the goalof trying to make candidates feel as relaxed and comfortableas possible and welcomed. for example, you probably don'twant to conduct an interview
in a dimly lit,cold stores room, or in a busy cafeteria where there is lots of noise, distractions, and eavesdroppers. it's also a goodidea to make sure that there is a separate area for candidates to waitwho haven't been interviewed. this will help reducethe likelihood that information about theinterview will be shared. once it has been verified,that all administrative and
logistical functionshave been arranged, you are ready forthe candidates. upon arrival of the candidate, the chairperson typicallygreets him or her and brings him or her tothe interview setting. the chairperson ensurescorrect pronunciation of the candidate's name and initiates introductions, again with the goal oftrying to break the ice and make the candidate feel welcomed and comfortable.
the chairperson then explains the interview processto the candidate. this could be justin verbal form or verbal form with written form for the candidateto read along. this will help ease thecandidate into the process because he or she willknow what to expect. following the instructions,it's a good idea to allow the candidatea couple of minutes
to ask questionsabout the procedure, again, just tomake sure that he or she is comfortable and ready to go. then the chairperson formallybeings the interview. once the interview has begun, the interviewers take turns asking their questions oneat a time and in order. while the candidate isresponding to each question, any interviewer can askthe probe question.
earlier, my colleague tara riccitalked about probe questions, but just as a refresher,only use probe questions if they are allowed andmake sure you stick to the approved probequestion sheet. try to limit your use of probes. only use them when it's absolutely necessary to clarify acandidate's response or to get additionalinformation. also, make sure that yourprobe questions are neutral
aren't keying candidatestowards right or wrong answers. additionally, whencandidates are responding, monitor your body language, make sure that neither yourfacial expressions, nor body language are conveyingright or wrong answers, and remember, never challengea candidate's response by word or expression. can anyone share examples ofnon-neutral facial expressions that you would not want touse during an interview?
one more time. crossing your arms, again. because crossing yourarms makes you maybe look disinterested oreven evaluative. laughing;laughing is definitely not something you want to do while a candidate isresponding to a question. rolling your eyes? again,it should be avoided while candidates areresponding to questions.
it cannot be emphasized enoughthat interviews should only ask the approved questionsand the approved probes. therefore, under nocircumstances, should an interviewerever ask a question about the demographiccharacteristics of the candidate. there is a fewexamples on this slide. in addition to beingunrelated to the job disrupting the standardizationof the interview process,
questions about these characteristics are likely to make candidatesfeel uncomfortable, unfairly judged, and may eveninvite legal challenge. does anyone have anyexamples of questions that they have been asked or that they know thatthey shouldn't ask during a structured interview? okay. while candidates areresponding to the questions,
interviewers need to maintainfocus on the details of the responses andtake good notes. remember, a lot of information is going to be provided by the candidate in a short time frame, and once the candidateleaves the room, your notes are the only evidenceof his or her responses. therefore, taking goodnotes will help reduce the burden on you to remember specific details and will also help you inthe evaluation process.
when taking your notes, keep in mind that inthe event of an appeal, your notes can be reviewed. therefore, be carefulabout you write. make sure you are alwaysprofessional with your notes. this slide summarizes the dosand don'ts of note taking. note should be summaries of thecontent and be detailed enough for you to accuratelyrate the candidate. do not write evaluationsof the responses.
just document them. any evaluations at this time could be interpreted as premature or bias in judgments. for example, youwould never write, "this is the best responsei have heard yet, we better hire this guy." also, do not write anyperceptions of the candidate. for example, "it sounds likeshe is exaggerating" or "he sounds like he isfull of himself."
just keep reminding yourselfwhen you are note taking, no evaluation, justdocumentation. throughout thequestioning process, the chairpersonkeeps track of time and also makes sure that theinterview stays on topic. candidates sometimesget excited about the information thatthey are sharing and they go into waytoo much in unnecessary detail or they jump to topicsthat aren't relevant.
the chairperson should beprepared to tactfully and politely interruptthe candidates and bring them back on track or to encouragea candidate to wrap it up. once all of the questionshave been asked, the interviewers have had time to gather enough information, the chairperson formallyends the interview and all note taking should stop. the information gatheringportion is done.
during this time, the candidateis given an opportunity to informally askquestions about the job, about the agency, etcetera. keep in mind that interviewersdo not take notes about these questions do not evaluate the candidatesbased on these questions. during this time, candidatessometimes ask how did i do, or related questions to try to geta sense of their performance. keep in mind that neitherof the chairperson
nor any of the interviewers should give the candidate anidea of how he or she performed. after the candidate hasasked any questions, the chairperson explains thenext steps in the process. for example, how long thecandidate should expect to wait before hearing back, or if thereare additional assessments in the process. the chairperson thenthanks the candidate, reminds them to keep the details of the interview confidential,
and dismisses the candidate. this concludes our section on conductingstructured interviews. the next set of slides willprovide you with information on evaluating structuredinterviews. so keep yourinterviewer hats on. as with the previous slides, on conductingstructured interviews, remember this isjust one method
that we have found tobe effective. after the candidate has leftthe interview setting, the focus shifts frominformation gathering to information evaluation. the approach we haveused typically begins with each interviewer taking turn to read his or her notes aloud to the other interviewers. this gives everyone achance to make sure that they have captured asmuch information as possible
on the candidate and that they have a completepicture of his her responses before evaluating them. interviewers are notsupposed to discuss their evaluations at this time. they are onlyexchanging information. after informationhas been exchanged, interviewers individually rate the candidate's responses. one way of doing thisis for each question,
first read the competencyand the definition and the question, followed by theexample responses for each proficiency level. this will help you get aright mind set for evaluation. then read your notes andcompare what you have written in your notes to theexample responses. then choose the mosteffective rating. once the interviewers have hadtime to make individual ratings
there may be an additionalconsensus rating process. this would have been decided prior to the startof the interview. if there will be aconsensus process, interviewers transfertheir individual ratings to the consensus rating form and they examine whetheror not consensus exists, based upon previously established criteria. for example, they couldexamine whether or not
all the ratings are withinone point of one another for each question response or whether or not allthe ratings are identical for each question response. if consensus does exist, again based upon the previouslyestablished criteria then the interviewers signand date the consensus form. however, if consensusdoesn't exist the interviewers engage ina consensus discussion.
excuse us for one second we are trying to find the slide. during the consensus discussion each interviewer providestheir reasons for ratings using their notes. panel members should listenintently to one another and should be non judgmentalwhen commenting. the consensus discussionis not about who is right and who is wrong, but aboutdetermining the rating that best reflects the candidate's response.
sometimes interviewers wantto barter with another or trade off on changingtheir individual ratings. for example, i will give in thistime on interpersonal skill if you will change ourrating on team work. this is not the point ofa consensus discussion. panel members shouldcontinue discussing until they have legitimatelyreached consensus. additionally interviewerscan be obstinate and engage in polarizing.
this is where they refuseto budge their ratings for the interview responses. if this should occur theinterview coordinator should be notified todetermine the next steps. after consensus has been reachedinterviewers should sign and date all ratings forms and initial changes to individual ratings. the chairperson collectsthese ratings forms and any othersensitive materials,
such as the questions andthe ratings scales and the chairperson stores thesematerials in a secured place until giving them tothe point of contact. again, these are alltest materials and you want to keepthem secured. this concludes the section on evaluatingstructured interviews. the final set of slides providesa summary of tips to help you be effective interviewers.
remember, yourobjectivity and accuracy during the structuredinterview administration and evaluation arecritical components of the structured interview. relying in first impressions; do not make quick decisions about the qualificationsof a candidate based on limited information. reserve your judgment
until the candidate hasresponded to all questions and you have receivedall information from them. focusing on superficial factors; do not allow any characteristics about the candidates appearance to influence yourperception of their responses or to distract you from objectivelyattending to their responses. candidate order errors; do not compare andcontrast candidates
with others who havepreviously been interviewed. for example, if you have justfinished interviewing a candidate who hasperformed very well, do not allow thiscandidate's stellar performance to make you judge subsequent candidates more strictly. evaluate eachcandidate's responses based against thecompetency rating scale only. non-verbal behaviors,
candidates arelikely to be at least somewhat nervous during theinterview proceedings and they may exhibit behaviors,influence of this anxiety. for example,they may tap their foot. they may have a shaky voice and pause a lot,they may even fidget. try not to give thesebehaviors undue weight. while they mayfactor into your ratings on a competency likeoral communication,
they shouldn'tfactor into your ratings on non related competencies. rater bias; do not allow anyopinions or biases about personalities or people or any other nonperformance related factors to influence your ratings. in a similar respect, don'tgive the benefit of the doubt to individuals whoseem similar to you.
for example, you found out theywent to the same high school that you did or they seem to have thesame personality as you. concentrate on theresponses not on the person. halo effect; do not allow positiveor negative information or performance on one competencyto influence your ratings and other competencies. some individualsare great speakers
and everything they saytends to sound really good. take this into considerationwhen rating them on relevant competencieslike oral communication, but not on unrelated competencies. as another example a candidatemay get off to rough start at the beginningof the interview. do not let thisinitial weak performance influence your perceptionof their subsequent responses. treat each question as a newopportunity for performance.
leniency; do not give high ratingsto all candidates. there is likelygoing to be candidates who could use further development in some areas. allow your ratings toreflects these differences across competenciesand across candidates. strictness; do not give lowratings to all candidates. there may be outstandingcandidates whose
demonstration of competencieswarrant high ratings. again, allow your ratings to reflect theseindividual differences. central tendency; avoid ratings competencies atthe center of the rating scale. so for example if your ratingscale is from one to five, this would mean avoidgiving only threes. when hesitating overgiving somebody a five, remember this doesn'tmean perfect performance.
it just means that the candidatedemonstrated behaviors that are consistent with thehighest proficiency level. when hesitating over a one,remember this doesn't mean that the candidate doesn'tpossess the competency. it just means that he or shedid not represent much of it with his or her responses. relevant information; do not be influenced by irrelevant information that the candidate mightmention during the interview.
for example, whileresponding to questions, candidates might providepersonal information that's not relevant to the competencies being assessed. try to ignore this informationand not be influenced by it and don't let it factorinto your ratings. we already wentover brooding, trading, ordering and polarizing in relation to theconsensus process. again, during the consensusprocess, the goal is to arrive
at the most accuraterating of the candidate. so be flexible with your ratingsand recognize that others may have captured informationthat you have missed. if i had to sumup the interview, administration and evaluationprocess into one slide, this would be it. as interviewers, know the competencies beingassessed and the questions, the rating scale and theexample of responses.
focus on the candidate'sresponses, not the candidate; take detailed notes on theresponses; not on the candidate. compare thecandidate's responses to the example responsesfor each proficiency level and objectivelyrate the candidate. in closing, remember that youare responsible for establishing a comfortable, fair andstandardized interview process and for giving every candidatean equal opportunity to succeed. make candidates feel ascomfortable as possible
by being nice, respectfuland professional. be attentive,focus on their responses and take detailed notes. do not give negative reactionsor comments about the responses or be influenced or distractedby non-job related factors. as mentioned by tara ricci, if you would like additional information on developing, administering or evaluatingstructured interviews, you can refer to the guide
that we have postedon our website. that's accessiblethrough the very first link; personnel assessment andselection resource center. the meetings linksprovide information on the development and useof assessments in general. thank you for your interestin this presentation. at this time we wouldlike to open up the floor for any additional questions. mine is more a comment.
i just talked to andrea and wehave a list of sources up here, but i also wanted to letindividuals know about an additional resource. i work for the competencyassessment branch; my name is geno novick. some of you mayhave participated in this session last week and that was a sessionthat i conducted. our group, thecompetency assessment branch
does structuredinterview training and we also do developmentand instruction interviews. we also do job analysis. so if you are interested indeveloping structured interview that is somethingthat our group, the competency assessmentbranch can help you with, we can do thejob analysis piece and we are also availableto do detailed training based on the competenciesthat you select
through thatjob analysis process. and the training actuallyuses the exact documents you would use aspart of your training and part of yourinterview process. so, that's something that we doand you also can practice using all of your own materialsand get psychologist's input into the ratings thatyou have provided. so that's taking this and justmaking it a bit more customized to the actual interviewyou would be conducting.
so i just wanted tolet everybody know that that is something that'savailable as well. are there any additionalcomments or questions? okay, well, thank youagain for your interest.
0 komentar:
Posting Komentar