employee training literature review

as joann indicated this is the first of foursessions. what i hope to do is get people familiar with methods of systematic reviewing,but at the same time bring up as many questions as i provide answers, so that you will beaware of the possible problematic aspects of a lot of things that go into creating evidencethat then is used to answer questions. so i want to discuss what is considered evidenceand why and how we, after we find evidence, we look at it, determine it's good, bad orindifferent and then we synthesize it and how in a last step, evidence is turned intorecommendations for clinicians and other practitioners. if you are familiar with issues of guidelinesdevelopment, they pretty much turn to these three steps and then there is an additionalstep of writing guidelines. in this series

we won't go as far. slide 1: we have four topics. today is moreor less the basics, how it was developed. the qualification of evidence and basicallywas expressed in a pyramid. next time we will look at how aan, the american academy of neurologyand others have developed refinements on the basic pyramid both developing pyramids forquestions other than what's the best treatment and going beyond what i tend to call designwith a capital d, and look at design with a small d.then the session after that we will look at the grade approach, which involves many ofthe refinements that the aan has, as a matter of fact, and in its latest versions aan isborrowing from the grade people, but also

very much look at how grade emphasizes theoutcomes for people with a disability. people who are served by practitioners as needingto be a primary item in developing guidelines. and then the fourth session that will be sixweeks from today will look at what else might be happening, needs to be happening, whatcan we expect to develop in the near future and bring up issues of what role should peoplewho are active in disability and rehabilitation play in that.can we develop this community of practice that joann was talking about to a degree thatthere is a continuous ongoing high level discussion of issues of evidence and systematic reviewsin general, and their use in disability and rehabilitation specifically?any questions on any of this? seeing nobody

starting to type, i will quickly go to background. slide 4: joann already indicated who i amand what i am with respect to nidrr research. for the last ten years i have very much beenfocusing on the type of issues that we will be dealing with evidencebased practice, systematicreviews, meta analysis and stuff like that. slide 5: and then, of course, joann will besupport person and communicate with you and be the person to send information to if youhave any specific questions that you suggest are dealt with on a future session or areturned into future session. slide 6: okay. let's start with what are theinfluences on a clinician's decisions? well, first and foremost, it might be training andexperience. and, of course, this training

and experience was provided by experts directlyor through textbooks and other information that experts provide which experts in andof themselves may not be up to date with the latest in clinical research. and i am emphasizingclinical research because that's where we are going to be talking about. it's this typeof research that is to provide the evidence base.so these clinicians get training in basic science, didactic clinical science and practicumsclinical science and after they finish their school program they will move into continuingeducation and various inservice training and, of course, people as they practice if theirown field will have their own experiences and build up expertise based on what specificallyworks with their clients sub groups of clients,

et cetera. the second influence on the clinician'sdecision should republican the values and preferences of clients. what outcomes theywant to achieve, what goals they want to achieve, what are they able and willing to do in orderto achieve those goals to the degree that it's within their ability.and then we get to what's more and more maybe in very strong influence on clinicians, societyand the healthcare system driven by primarily underlying societal values that then get translatedinto how the healthcare where a clinician or professional works in, works within lawsand regulations that specify professional roles and privileges, the reimbursement fordiagnostic treatment and management actions, and sometimes that also involves feasibilityif you don't have an m.r.i. machine in your

office, it may not be able to get one.and then there is the direct organizational mandates, and pretty much any rehabilitationdisability professional who hasn't hung out his or her own shingle works within an organizationand very much the organization will dictate what people are allowed and are not allowedto deliver in terms of patient assessment, treatment, et cetera.and then lastly, we would hope that decision making would be very much guided by clinicalresearch, and that can be primary studies but because there is so much being publishedthat it's impossible to keep up with many people will rely on what i have called ebpresources which is systematic reviews, clinically assessed topics, various journals that noware being published that have short summaries

of primary studies with clinical bottom line.there are additional types of resources available. the most important or terminal step probablyis clinical guidelines. and i provide here the reference with somepeople who would like to read more about them. so to the degree that clinical research isrelied on to help people make decisions, it becomes, quote, unquote, evidence, and nowlet's start looking what evidence is. a dictionary tells me it's an outward sign, something thatfurnishes or tends to furnish proof, a medium of proof, approved testimony. and that ispresumably where we are dealing with when we get to these primary studies. and thenthe archaic is the state of being evidence, which ties in with the latin roots of theword evidence, which goes back to videre,

to see, clear, distinct, plain, visible, evident,and resulted in evidentia the quality of being manifest. which will suggest that not allevidence is the same whether in single piece or in combination. we always want to justwhat is offered as evidence, how relevant is it here? does it provide information foror against a specific proposition? does it provide evidence relevant to a clinicalquestion? sufficiency, is it enough by itself or does it need to be enlarged by corroboratedby other pieces of information on the same topic, and if i have one or more pieces ofevidence, is it trustworthy? and we can look at internal proof that something may be weakand we can look at external proof, who put this evidence together

and what ax may they have had to grind? wewill have opportunities to look at issues of conflict of interest.if you go to wikipedia, there is an article on burden of proof which is in the area oflaw, and you may be aware of the fact that in the legal situations there are a numberof standards of evidence running all the way from the weakest, the reasonable suspicionthrough probable cause, some credible evidence, substantial evidence in some cases you needpreponderance of evidence to find somebody guilty.it might be clear and convincing evidence. sometimes the legal standard is beyond reasonabledoubt, and it seems that beyond that there is even beyond a shadow of a doubt. so wemay want to not necessarily look for direct

parallel to these nine possible grades withinthe law, but certainly keep in mind that not all evidence is created equal. and we mayneed to have before we make decisions doing something or not doing something as a practitioner,we may want to take a very hard look at the burden of proof, what level, what quantity,what quality evidence do we have. and learning when we get into the area ofevidence practice, the term evidence can mean two things. it may be a single study, whichthen in order to be evidence before or against a particular action needs to be of relevanceof sufficient quality, et cetera. or it may be the body of all studies that are relevantof sufficient quality, et cetera, et cetera. and preferably not just in a raw body butsummarized or synthesized qualitatively or

quantitatively so you may want to keep inmind that whenever you hear me say evidence or you read it in literature that deals withevidence based practice, you can ask yourself, well, what are they talking about here? onepiece of evidence, a single study or a body of evidence?this is more or less a step back to when i talked about influences on decisions. ideallythe evidence based practice process happens as follows. the clinician, practitioner startswith a question. what's the best way to diagnose something? what's the best way to treat something?how should i be screening? is it worthwhile to be screening? in order to give an answerto that, either the practitioner herself needs to put the evidence together or if he or sheis lucky, there is already a body of evidence

put together by systematic reviewer whichprovides an overview of the quality, the quantity, the variety of the evidence as that is determinedusing specific criteria that still needs to be balanced by the practitioner with his orher own values, those of his organization, of his or her patients, costs often are abig consideration, and there may be other things like feasibility and speed and lotsof issues that determine what the answer to the question might be.do we have questions this far? if not, don't be afraid to start typing while i start talkingagain because joann will call to my attention that the hand has been raised. okay. we aregoing to go towards systematic reviews and if you go to pub med or med line you willfind their position of review which an article

or book that reviews published material ona particular subject. and generally when i say review or when youread review in the literature without seeing the word systematic, it means more traditionalreview and qualitative review where somebody based on his or her own knowledge and preferencesand for all we know in conflicts of interest decides to make some recommendations. as opposedthe definition of systematic review which i took from the aqasr glossary, aqasr standsfor assessment and quality no, assessment of quality and applicability of systematicreviews. and the reference here is at the bottom of the page, which specifies that asystematic review synthesize research evidence focused on a particular question. that's alwayswhat it starts off with, and follows an a

priori protocol to systematically find primarystudies to assess them for their quality, three, extract relative information, four,synthesizes information qualitatively or quantitatively. and the glossary also suggests that systematicreview does create bias in the process and improve the dependability of the answer tothe question through the use of a protocol, extensive, electronic and manual literaturesearch, very systematic, careful extracting of data and critical appraisal avid studies.and generally the extracting of data and the appraisal done by two people independentlyand if they have disagreements, those are results.so results always taken from the aqsr manual with an overview of the various steps andi will run through this very quickly.

r. we start off with the focus of a questionin the bottom in green, which ideally lead to systematic review protocol that is writtenbefore the review itself is started. ideally the protocol is peer reviewed itself. so thatexperts look at it and say how come you are not looking at. i suggest you also include,and that type of stuff. then in the blue column, row in the middle.we start with database searching, which is followed by scanning of the abstracts thatwere found, selected abstracts that are considered to be relevant are moved into a next stagewhere we scan full papers. these papers are submitted to quality assessment where we willlook at how good was the research either for all of them or for the better ones. we thennext extract the information that is most

specific to the question that is synthesizedqualitatively or quantitatively in a meta-analysis leading to a set of conclusions and recommendations.between the gray line and the blue row, we have in white with red borders a list of thedocuments, forms, et cetera, that are used in these various steps. very often alreadycreated as part of the protocol and certainly drafted as part of the protocol, but maybefinalized based on some later issues. and then at the very top, we have some steps inyellow boxes with the blue border that refer to steps in the systematic reviewing processthat are very much recommended, but not always done.one is inquiries from experts. you find people who are experts in a particular area and saywhat else do you know? what studies are you

aware of that may not have been published,not have been published yet, but we should be looking at? ancestor searching is simplyonce you have in your database searching found applicable papers. you go to the referencelist and see whether there are additional studies there that you might not have found.journal hand searching is almost never done, but that's actually sitting down with 40 yearsof the journal of disability studies and leafing from item to item in the table of contentsto see whether something there that's relevant. very often the information in published studiesis not sufficient, not detailed enough to either assess the quality or it leaves outinformation that we would like to have in evidence table. so there is communicationwith others. and then lastly, there is peer

review, which can refer to initially reviewof the protocol, but later on review of the report which here i will focus on as consistingof the evidence tables and the conclusions and recommendations.in a slightly different view here we start out with an entire bibliographic database,all of cinahl, all of pubmed, what have you using key terms and thesaurus terms, et cetera,we split that content into two parts, things that are possibly relevant versus everythingin their that's irrelevant. then we have ideally two or more people look at the abstracts usinga few fairly broad terms and now separate the pile of abstracts into a smaller pilethat have promise, and a very big pile of stuff that's all irrelevant.next we get a copy of all of the promising

papers, and, again, have two or more peoplelook at each one with now fairly well defined narrow terms, and make a final selection onapplicable studies and irrelevant studies. and the last step we have, again, two or morepeople extract information that inform the quality of the studies, whether that's internalvalidity or external validity generalizability. and that information at the minimum is usedin the rest of the systematic reviewing sometimes people throw out all of the poor quality studiesand we will have more about that to say soon, more commonly, high, medium and low qualitystudies are all kept, but how they are used for decision making is different as you wouldnot be surprised if we are looking at quality of evidence, we have high quality studiesshould count for more than medium or low quality

studies.questions on any of this? no. okay. let's start looking more in detail about determinantsof the quality of the evidence and here i am talking about the individual primary study.first of all, of course, is research design and this is design with a capital d is thestudy randomized clinical trial, and i will be using the term rct without wasting my timeon fully expanding that term or rct with crossover, is it a study with historical controls, isit a pre-post study, is the subject design, and if so what type of single subject design.might it be regression continuity design, time series, et cetera.knowing that overall design of a study, you already know a lot about the quality of theevidence that you? general might expect. but

there are, of course, other issues to be considered.what's the quality of the outcome measures that are being used? is this a homemade measureof some outcome or are the researchers using a very well known well studied outcome measurewith very good psychometric elements. if there is an intervention study, what is the interventionand what is the quality control on the intervention? are they doing any monitoring of the fidelityof the actual provision of the intervention? and then a big issue is blinding. are thepeople who do the intervention blinded for diagnostic studies we have similar issuesin blinding to, say, the person who does the diagnosis with a new measure should be blindto the diagnosis provided by the gold standard measure. we could be talking about blindingof the patients, subjects, clients, if there

is an independent assessor, which is alwaysto be referred, that person should be and always can be blinded.we can blind statistician, and then to have that outcome analysis statistically, we mayapply in assigning subjects to study arms. we may do such things as blocking stratifying,matching. so all of this could be, should be in the proposal for research. but onceyou start doing the research, plenty of stuff can go wrong. and in order to look at evidenceand the quality of evidence, we should very much be aware of, be informed by what happenedduring the implementation of the research. what was the number of subjects they actuallyrecruited and how did that relate to the target number that they specified specifically bypower analysis.

and i should have maybe put power analysison the previous slide. was there any failure of the blinding, either clinicians or patientsor assessors break the blinds? how much attrition was there and is there any suggestion thatthis attrition was not random, but specific? and we pretty much know that attrition almostnever is random. certain sub groups of people disappear.how many missing data do we have and what's the nature of the missingness? is this completelyat random? very specific or somewhere in between? and statisticians have developed nice waysto look at this. even if nothing went wrong, we have the issue that with sample sometimeswe have an unlucky hand with sampling and the people that end up in our study happento be people who in general have bad outcomes

and another time we have issues that peopletend to have poor outcomes and at least when we have used good methods for distributingpeople over study arms, we have opportunities to apply inferential statistics and come upwith some confidence intervals telling us what are the limits that likely the true valuesfor the population lie. whether or not we had an unlucky hand withsampling fluctuations, the research implementation ideally should be intend to treat, but veryoften people report poor protocol which creates problems and we can see evidence and sometimesthey not just go per protocol, but go on complete fishing expeditions which means that theymay publish one of twenty outcomes because that one happened to be statistically significant,and the other nine teen was no difference,

and they stick it in the drawer.so that's what you are up to if you have to evaluate the quality of research in orderto determine what its value might be as evidence. so we have to find some way to look at thaton the level of an individual study. now, for a systematic review, we generallyare dealing with a number of studies, not just one, but hopefully five, six, to ten,20. and now when we start looking at the quality of the evidence overall, we have to look atwhat's the number of studies? what was the quality, the grade of these individual studies?how large were these studies? individually and combined? what's the overall number ofsubjects? what were the findings, and are the findings consistent or do we have onethird that found no difference between experimental

group and control group, one third that foundthat the treatment was effective and one third that found that the treatment made peopleworse? well, good luck if you have that situation.we need to know the effect sizes of individual studies and we need to be able to calculatean average effect size at least when the various studies do not point in all directions butmay be just in two, some in different, some in favor. and then, of course, we again haveto maybe before we start synthesizing answer questions on applicability. did all of thestudies deal with the same population or draw from the same population? did they all handlesimilar outcomes? are those outcomes patient valued outcomes or proxy measures? and proxyhere, i do not mean that the answers to questions

were provided by family member, but i'm talkingabout a proxy for a real life outcome if the real life outcome is ability to move one'slimbs in spite of arthritis, then a proxy might be a certain serum level of chemicalthat may be related to severity of the arthritic process but cannot stand in for an outcomethat patient valued themselves. so that's a proxy outcome. and we have theissue of if it's an intervention study, did all of the studies use more or less the sameintervention and is it feasible? okay. about 20 slides back referred to the issue of unsystematicreview, the qualitative ones where authors picked the primary studies randomly, the authorsweighed the primary studies randomly, they could just say, well, this study wasn't verygood, disregard the findings wrought necessarily

having a priori standards for, a, what isgood, and, b, how much you should discount the value of a week study.and they, as i indicated might draw conclusions that might correspond well to their priorconvictions fit with their interest. the first meta analysis were done in social sciencesand initially people just found some studies and combined them without really looking systematically,evaluating systematically, et cetera, et cetera. systematic reviewing pretty much was inventedin medicine in the 1980s where a lot of the stuff we do now as part of systematic reviewingwas first developed to make the whole process of reviewing and synthesizing the evidenceinto careful step by step process that is very transparent so that people can see whatyou did, why you did it, how you did it, what

criteria you used, et cetera, et cetera.and as part of this invention, quote, unquote, of systematic reviewing and evidence basedmedicine, the first classification to my knowledge of primary studies in terms of their strengthof evidence was created by sackett in 1986, and sackett is a physician that was active,still is active at mcmaster university in ontario which we may call a hot bed of evidencebased practice and they have made many major contributions.okay. in 1986 paper which was focused on treatment for chest disorders or diseases, sackett tomy knowledge pretty much ad hoc and without prior work, although if i were to go talkwith him, he may set me straight on that account. came up with five levels. first of all, thereare rcts with low false positive, and low

false negative errors, what we now call highpower studies with alpha point o5 or less and beta .20 or less..second is studies that are still rcts, but they lack power. rct is randomized controltrials, and make a note of that, because i will tend to not use the full expanded work.i will just talk about rcts. third level is any study that didn't randomize but comparedbetween a treated group and a nontreated group that had to be contemporaneous group whichseparates it from level 4 where, again, there is a comparison between a treated and a nontreatedgroup, or maybe between the group that got the new type of treatment, the new inventionversus the people who got the old agreement. but in this case, those people in the past,information on them is harvested from clinical

records or from old research. and then lastly,we have level 5 case series without controls just a group of people who are all treatedwith the proposed new treatment and based on the results the investigator says, hey,this is much better than we used to have. here is where we got evidence hierarchy orevidence sometimes you hear the pyramid from the very highest at the top, rcts that cohortstudies, then case control studies, then case series and case reports. now, sackett in thesame paper related this to levels of evidence and grades of recommendation. or better said,he related level of evidence to a grade of recommendation. he had a grade of recommendationa, which is the strongest, criteria were that there would be at least one, preferably morelevel one rcts.

then there is a grade b, and we will lookat a second at what those grades mean. for that he required support by at least one level2 rct, and then c supported only by level 3, 4 or 5. so i managed not to put in whatthe grades mean. okay. we presumably will fix this problem when this is archived forpermanent status on the website. but imagine it's something along the line grade a supportedby at least one preferably more high level rcts. this is something you should do, gradeb supported by at least one level 2. this is something that you may want to do. andlevel c is something that you at liberty to do or you are free not to do.and ms.grubs is typing something. mixed method studies. they do not fit in at all at thispoint in time. at the time that sackett was

publishing and other people were inventingsystematic reviewing, qualitative research didn't exist. mixed method studies didn'texist. i will make a note of this question and make sure that we hit the issue at somepoint later on, probably in the fourth session. so nothing right now. okay. now, this is whatoften happens with sackett's levels or even with other schemes. people make a dichotomybetween rcts and the rest of the world. and if there are rcts, they are the basis formaking recommendations and if you do not have an rct, you don't have anything and you makeno recommendation. this still to some degree is maybe not the initial policy of the cochrancollaboration but the drift of many of the groups in cochran where they do a carefulsearch of literature, evaluate the various

studies, discover that none of them is anrct, and come up with the conclusion there is nothing to recommend because there is nohigh quality evidence good by products and we will have to discuss opportunity to discussthings like this very much starting next week. or even over here, many of you, certainlythe people who work for asha, and i saw three asha staff members on the list of people whoplanned to participate, may have heard of the work that cicerone, et. al., have beendoing in systematically reviewing any and all research that has something to say aboutinterventions for cognition and communication in people with traumatic brain injury, strokeand some very closely related disorders. and cicerone too has hierarchy, but it has onlythree levels.

at the top he has well designed prospectivercts, i don't know how you could have retrospective rcts but that's a different story. with truerandomization or with quasi randomization which there are some schemes that mean thatthe people who want, come for treatment on monday, are assigned to treatment a, the peoplewho come on tuesday are assigned for treatment b.research has shown that these quasi schemes very often are abused and the end result isnot quasi. so dr. cicerone may want to review this and maybe change his criteria. number2 level, prospective nonrandomized cohort studies, retrospective nonrandomized casecontrol studies, clinical series with well designed controls that permitted between subjectcomparisons of treatment conditions, such

as multiple baseline across studies. so exceptmake the single subject designs is pretty much coincides with sackett's level, roman2, roman 3, and roman 4. and then we have third level for cicerone clinical series withoutcongruent controls, result single cases with appropriate single subject, multiple baselinewith adequate results. relationship 27, 29. this is 27 i went back to these enrollmentsare not the same as cicerone romans. one of sackett's coincide with 2 of cicerone. and5 of sackett corresponds with roman numeral 3 of cicerone. and that's a problem that wehave in the systematic reviewing world that there are a number of hierarchies and theyall have different number of levels and even if they have the same number of levels, themeaning not necessarily is the same.

so if somebody tells you we have level 2 evidence,you always have to go and say what does level 2 mean? pretty much is means what is the bigd of design. very often some of the little d's get into the mix too. this is what ciceronedid based on the level of recommendations, and the reference is below, but we distinguishpractice and practice guidelines and practice options. and practice standards, you needto have one well designed class one study, with an adequate sample or overwhelming class2 evidence. of course, directly addressing the effectivenessof the treatment in question. good evidence to support recommendation whether the treatmentshould be specifically considered. practice guidelines is at least two well designed class2 studies with adequate samples, correctly

addressing the question, fair evidence, andit's suggestion but not something that you should do and then practice options class2 or 3 based with some at additional grounds to support it, and in the end it is stillsomething that is an option and nobody should look at you ugly if you as a clinician decidenot to follow the recommendation and use this intervention.big d versus little d i addressed previously. big d is what we call generally designed randomizedclinical trial versus case control study versus cohort study versus prepost study, et cetera,design with a small d i refer to as all of the other options that make a study withinits big d category stronger or weaker. did you do blinding? did you have great outcomemeasures? how did you plan to analyze per

protocol or intend to treat and then issuesof actual implementation, what's your attrition level? what's your missing data level? etcetera. questions? rachel is typing. thank you. okay.so we are close to the end of this. well, if there is so much difference between studies,based on their design with either the big d or the small d, how can we use that informationin the classification of studies? and you already saw that sackett and cicerone basicallyonly used the big d although there are some qualifications on this. for instance, sackettrefers to big enough to give you acceptable alpha and beta error levels. cicerone hassome other qualifications of studies, but basically they use only the big d design.now, as a lot of people who have suggested

that we should do more than just use big d,we should look at the little d elements. and one of them was a fellow called jadad andi'm not giving a reference because this is pretty much outdated and even though i stillsee people referring to, i don't think it's a good idea to use it. jadad essentially hasfive questions. is the study described as randomized? andthis pretty much is looking at only abstracts, but even if it's, you have the full text,you can answer these questions. is the study describes as randomized? is doesn't ask isthe study randomized. no, no, is it described as randomized. so the next time you do a sloppycase study, call it randomized and mr.�jadad will be fooled.question two, is the method of generating

the randomization sequence appropriate? andthis ties back in to such things as where cicerone would except for his level one astudy that use quasi randomization. jadad would not accept that and would only accepttrue randomization, which isn't even flipping a coin, but either you have prepared sequenceenvelopes with a number, and after, only after you have informed consent, you open an envelopeand see what the subject is designed to. or after you have informed consent, you callthe pharmacy and the pharmacy tells you your next patient will be put on drug or put onplacebo. question three is double blinding used? so he is referring only to blindingof the investigator or the people who admin sister's treatment, and to the patient, thesubject, the client. we will have a lot to

say about double blinding because in moststudies in rehabilitation and disability blinding is plain impossible which has major implicationsfor how our studies are looked at by all of these scales to look at the quality of evidence.number 4 is the method of double blinding appropriate? did you indeed manage to puta blindfold on that was not likely to be broken. for instance, there are certain drugs thatnot just do good things in your body but also leave a dry mouth. well, you have to tellthe patient before hand that the drug, the experimental drug may give them a dry mouth,but then if they get placebo and do not have a dry mouth, they pretty much know they areon placebo. so what to do? well, the best studies administerto the placebo group a drug that has only

one effect, creating a dry mouth. that's whyyou have an appropriate method of blinding. and there are other tricks like that. number5, is there a description of withdrawals and dropouts? again, notice it doesn't say arethere too many withdrawals? are there too many dropouts? but just is there a description?so if you honestly describe that 60% of your treatment group and 67% of your control groupdropped out, that's fine with mr. jadad. among you, people who are in physical therapyarea may have heard of the pedro scale. and as usual i have forgotten what pedro standsfor but p is for physiotherapy, e is for evidence, i don't know what d is but without any doubt,joann remembers and she will type it in. pedro consists of eleven things, here are 8 to 11.here is 1 to 7. number 1 is not scored on

the scale. but because the pedro items camefrom a specific delphi exercise where one of the items was eligibility criteria specificationthey kept it as a nonscored item. number two, random allocation, number three,concealed allocation, which pretty much means nobody knows which arm the subtle be assignedto until after the person has consented to be in the study. number four, similar at baseline.if you have a study with very small groups, you may end up that in your treatment groupeight out of ten people are male, and in the control group three out of ten people aremale. they are unbalanced. well, a requirement inpedro is that they are balanced regarding at least the most important prognostic indicators,those factors that are expected to have impact

on outcomes. blinding of all subjects, blindingof all therapists who administer the therapy, blindness of all assessors.if a study didn't use assessors but either subjects themselves filled out the questionnaire,they are the assessors, and if they weren't blinded, the assessors weren't blinded. ifon the other hand, it's the therapist who rates the functioning of the patient, andthe therapist is the assessor, and the therapist isn't blinded, the assessor isn't blinded.measures of at least one key outcome were obtained for more than 85% of the subjectsinitially allocated. so unless jadad who just says i want a reportof who dropped out, how many, pedro says if you have more than 85% dropout, you are introuble. you cannot be in perfect study. some

other people are using 80% just for your information.number 9, all people for whom outcome measures were available received the treatment or thecontrol condition as allocated. there was no erroneous switching or on purpose switching.well, mrs. jones, you end up in the control group, but as i see that you very much needa good treatment, we decided to put you in the experimental group. no, no. that's a no,no. and if that happens on purpose or sometimes by accident, people have to be analyzed bythe group they were originally assigned to, not the group that they were treated as. number10, between groups comparisons and report for at least one key outcome, and number 11,the study provides both point measures and measures of variability for at least one outcome.and that means don't just give the mean for

the experimental group and the control group,but give the mean and the standard deviation because then we can calculate an effect size.if you control percent cure, the percent is effect side by itself. so there is ten itemson which you can score 0 or 1. so the pedro scale gives you total score between 0 and10. now, both jadad and pedro are checklists sorry, are rating scales. they have a setof items that count the number of yes's or add up 1s and 0s and come up with a score.there are other people who just use the checklist, but not necessarily add up items. they justwant to know what are the weaknesses of this particular study.and once they have that information, they do other things like saying, okay, this isan important item, the rest are not that important.

this is an important item, and if you don'thave a positive on this particular item, i am throwing out your study. or if you do nothave this particular item, we consider you a level 2 study. for instance, in both ciceroneand sackett, if you do not have randomization, you cannot be a level 1 study. you can atmost be a level 2 study. so we can use the item information to eliminate studies, toweigh studies, or sometimes even do a sensitivity analysis where you start with looking at whatis the pool result for all of the studies? what happens if i throw out the studies thatdidn't use blinding? okay. this happens to the average effect size.now, what happens if i throw out all of the studies that in addition do not have truerandomization? this is what's happening. so

more or less determine whether it makes adifference whether or not you are weak or strong or one or more of those items. as opposed,say, the rating scales whether that's pedro or jadad, and there are others that combineitems on those scores to calculate and study quality score and why do i say use individualitems? this should be combined items. joann, will you make a note that we shouldfix this? the combined items. for instance, you will read many studies on physicalitytherapy where they used pedro scale and said, okay, we threw out any studies that had apedro score of less than six. and it doesn't matter what keeps you away from perfect whetherit's lack of blinding or lack of concealment or lack of intent to treat analysis if youlose more than four points, you are out.

of course, the score can also be used to weighstudies, and to do sensitivity analysis along the lines i just mentioned. and a big issuethat nobody except for one person in the literature has talked about is, well, is adding up itemsthe best way to handle it? that's what we are used to because that's what we do in thefim and all kinds of other things. but maybe we should use the lowest item.now, within o1 scale, the lowest item means if one item has a 0, the total score is 0,but there are other scales, for instance, the black and down scale that has 20 plusitems scored three levels or more that could use other methods. you could think of multiplicationof items. again, when you multiply with items that have 0 possibility, your 10 turns theproduct 0, so that's no good, but there are

other ways to do better than sum scoring,and maybe we could talk more about that. questions, ann has filled in the meantimepedro means physiotherapy evidence database. where they get the ro from, i have no idea,and if somebody can resolve that, that would be nice. probably they use it because it iseasier to remember pedro than ped or they wanted to avoid confusion with pedestrians.and debby herbert is going to give the what is the most commonly used scale, pedro? no.you see the jadad scale used. you see the pedro scale used. you see what i refer toas the black and down scale used. both the jadad scale and the pedro scale,almost all of your points come from randomization and blinding t well, we in rehabilitationand disability studies have problems having

blinding. we still can do randomization, butsometimes even that is problematic or not ethical. very often we cannot do blinding.there are other areas of healthcare where they have similar problems, behavioral medicine,public health, et cetera, et cetera. so very often they do not have any rcts ornot many good rcts, but have also case control studies, and they prefer to use a rating scalethat looks at more of the discern ratio than just randomization and blinding. and thenblack and downs is very good measure to use even though it's quite lengthy. and thereare additional rating scales that that may be used in particular areas of healthcareor in particular countries. i would not off the top of my head want to make a judgmentas to which one is used overall, and not even

which one is used in areas that i'm more familiarwith, for instance, disability and rehabilitation research.and joann has found that they added the ro to make pedro catchy just as i thought. okay.this is it for today. miracle of miracles, within the hour and a half. thank you forparticipating even though your participation was nothing more than listening and askingquestions when what i did was what i showed or talked about was not clear. we invite youto provide input on today's session. and at the bottom of the page is the link to theevaluation form that will take you less than five minutes.share your ideas for future sessions, and that can be either information that fits intothe next three topics that you hope and pray

that i will address or it may be somethingthat's much bigger and certainly joann and her colleagues have no problem with extendingthe series. we may not be able to do it right after number four because i'm vacationing,but we certainly can have a longer series with some interruption or with a longer interruptionand start a new series. start asking questions of one another, startreporting to one another as to what you are doing, what the problems are that you arehaving in doing systematic reviews or employing systematic reviews and let us see what wecan help. so for any issues, contact joann, and we will have you as intimate guests in14 days. thank you so much. >> joann starks: thank you very much, thatwas a wonderful presentation, and thank you

to everyone for participating today. we hopeyou found the session to be informative and that you will join us for the next three webinars.as marcel mentioned we do have a brief evaluation form and would really appreciate your input.the link is on the last page, and we will also be sending an email with the link tothat evaluation form. and as a reminder, we will make the recording of today's sessionavailable in a few days. on this final note, i would like to conclude today's webinar witha big thank you to dr.marcel dijkers, for myself, ann williams and all of the staffat the ktddr. we appreciate the support from nidrr to carry out the webinars and activities.we look forward to your participation in session two on june18th. good afternoon and goodbye.