knowledgehut training review

>> coordinator: welcome and thank you allfor standing by. i would like to inform all participants you will be in a listen onlymode until the question and answer session of today's conference call. if you would liketo ask a question, please press star 1 on your touchtone phone. today's conference call is also being recorded.if you have any objections you may disconnect the call at this time. now i would like toturn the call over to elizabeth mertinko. thank you ma'am. you may begin. >> elizabeth mertinko: great. thank you lisa.welcome to the child welfare information technology systems managers' and staff webinar series,brought to you on behalf of the health and

human services administration for childrenand families, children's bureau, and presented by icf international. today's webinar is entitled cmmi and you,the right fit. i am elizabeth mertinko. i'm your host and your moderator for today's webinar. for new attendees and for those who may havemissed previous webinars, here's the list of the previously recorded webinars whichare posted to the children's bureau web site. we're currently working to identify futurewebinar topics for the summer and for early fall. we encourage our attendees to participatein our webinar with questions and comments

as well as polls that we'll be operating throughoutthe session. all of the participant lines are muted now, but we will open them for aq&a session at the end of the presentation. please be aware that you can submit questionsat any time using the go to webinar chat feature. and those will be addressed during the q&asession. should we run out of time, we'll respond to your questions via email. and should you have additional questions,you may submit those to me at the email address listed on the slide. also, if you have any topics that you wouldlike to recommend as potential webinars, or if you would like to volunteer to be a presenteron one of our webinars, please contact me

sat the email address listed above. the division of state systems within the children'sbureau, continues to provide a series of monthly webinars supporting information sharing anddiscussion. understanding who is attending the webinarshelps us identify content that's applicable for everyone participating in an agency'scwis efforts. i'm going to go ahead and start our first poll and ask that you identify yourrole on your project. and for the first time ever our technologyis failing me and has jammed on the poll. okay. i'm just going to ask our audience tohold for one moment while we unjam our technology. while we wait i'm going to go ahead and introducetoday's presenter.

audra stewart is a certified scrum masterwith over 17 years of experience in information technology, including software development,cybersecurity, information assurance, process improvement and quality assurance. ms. stewart has worked extensively with varioussoftware development lifecycle models, including spiral, incremental, waterfall, iterativeand agile. she currently leads the icf agile workinggroup, an internal team focused on training, coaching and assisting project teams in transformingand adopting agile practices. in her current role, she serves as the qualityand process improvement manager at icf international, which includes ensuring that the organizationmaintains a credible cmmi maturity level 3

credential. audra, i'm going to ask that you go aheadand start your presentation while i restart the go to webinar software. and if folks onthe phone could just hang in there, we'll have the visual up for you in just a moment. >> audra stewart: okay. excellent. thank youelizabeth. cmmi and you, is this the right fit? the short answer? maybe. today you'lllearn more about that. and perhaps be able to make an educated decision. what about allof the other models that are out there? i like to refer to this as models; processes;certifications; oh my. we'll have polling questions that we'd like your feedback on,as part of this section of the webinar. we'll

review the results once the poll has closedand of course once the technology will allow you to participate in the poll. so we'll come back to that polling question.the first slide that you aren't able to see at the moment, is slide 7 which is a representationof the traditional approach to project management. before we evaluate which process improvementmodels or best practice models are right for your organization, let's get on the same pageabout models. let's take the traditional model for example - initiate; plan and design; execute;monitor and control; and lastly, closing. simple right? this is the model we're most familiar within project management and the pmi, the project

management institute would agree that theseare the most common project level approaches to project management. the next model that i'm sharing with you isthe agile project management approach on slide 7. more specifically, agile delivery of businessvalues to end customers. some of you may already be using scrum, kanbanor even lean principles in the management of your project or perhaps at the state level.but basically, looking at this diagram, you can see that you plan a backlog of items thatneed to be accomplished. you perform iterations some of you may referto them as sprints and conduct daily reviews. typically, if you're using a scrum approach,you perform various releases of your project,

whether it be software development as a release,or perhaps it's something else, such as training. and those are your deliverables. you receivefeedback and loop back through this approach known as iterations. and this is at a high level, a good exampleof agile project management. next, with regard to models, we have agile program management.so let's take this a step further. how do you know if a program management and portfoliomanagement at scale? what if you have multiple teams across anenterprise or an entity, that need to be following similar practices for the most part? how doesit all fit in? in this diagram, while it has a lot of partsto it, demonstrates why program management

and portfolio management needs a common setof practices and perhaps approaches, if you want all the parts to come together. so at the top what you're seeing, on the leftarrow, is the activity at a portfolio level that would be mapped to the industry bestpractices of the pmi or the project management institute. in the center you're seeing two bands - onefor program management and the ones directly below it, for project management. within thosebands there are different level of activities that are being performed in order to satisfyand fulfill the portfolio's objectives. this could be, for example, an agency suchas children's bureau, where you need to maintain

multiple programs and implement projects withinthose programs. these are two areas where agile practices could benefit an organizationacross the board. at the bottom there you're seeing operationsor the service line. some of you may be familiar with a term known as idle. what's importanthere is that it's service level management. and operations needs to interact with allthe other levels of a program or a portfolio based organization. and sometimes having a different model toimplement those activities is necessary and perhaps required. on the right hand side thereare various terms listed here, some that you may be familiar with.

but in reality, there are programs that requirefederal consideration such as compliance with regulations. perhaps the type of acquisitionor the color of the money that you're using to run your program organization or your agencyor backers that need to be taken into consideration. and so those are listed as just an exampleor a representative sample of the types of things that need to be considered throughoutan entire program's life. so one of the polls that we wanted to conduct at this juncture,was have you ever worked with any of these system development models. if you could participate in the poll thatwould give us an understanding of how much exposure you've had.

>> elizabeth mertinko: and audra, it lookslike the software is back up and running and the polling feature has not crashed. so we'vegot about 30% of people who have cast their vote. if i could ask all of you to just goin and let us know what your experience is, that would be really helpful to us. and we just need the last 18% of you to hopon there and answer the question for us. i'll just give you another couple of seconds todo that. okay. we have almost everybody having answered the question so i'm going to go aheadand close this one and share the results. and there we go. we have 87% that have experience with traditionalwaterfall; 47% agile scrum; and 13% agile

kanban. >> audra stewart: excellent. so this givesyou all a feel for each other as to where most of your experience lies with regard tomodels. and it's not uncommon that traditional or known as waterfall, would be the one thatmost of us have exposure to. i know it certainly was one of the first that i learned. so let's take that information and let's talkabout it a little bit from the perspective of common sense. now i just used common senseplease. there are so many models out there. let's take a look at that further. there aremany acronyms available to us. with so many of them it's common that peopledon't understand what they're supposed to

be doing; why they are doing it. if, for example,your organization is off agile and the constituents of your organization don't understand agileor kanban or lean kit or cmmi or pmi. they don't know what they need to be doing.they just don't know what they don't know. and some do and will ignore the processesanyway. so this seriously - the important messageis to your organization, is that you communicate what you want them to be doing; what modelor best practices you want them to be leaning on; and how they are going about doing it,so that the acronym soup is palatable to your staff. let's explore this further. i like the representationof the umbrellas hanging from trees. do models

grow on trees? it may feel like it. but inreality, there are branches within a tree if you will, a model, where all of them intersect. in reality, pmi, cmmi, idle, ieee, kanban,six sigma, traditional iterative recursive - all of these different approaches to managingour program's projects and implementing process improvement, have a lot of overlap and similarities. the question to ask your organization is whichof these is right for you and your organization? you may consider that question at the levelof portfolio management; program management, perhaps just at the project team level, andmaybe just within your interactions with your state's constituents.

but before we can ask and answer those questions,you want to ask yourself what problem or challenges the business is trying to solve. if we moveonto learning about the cmmi, it may help you identify some solutions to your problemstatement. so what you're seeing on the screen is anexample of the cmmi known as the capability maturity model integration, a representationof it with its five maturity levels. at level one, you will see that it's considered initial.this does not yield a rating to any vendor or organization. it basically says we've got processes, we'refollowing them. no, we haven't written them down. perhaps we're not controlling them.no one's enforcing anything. it's largely

reactive. it's an organization designed toleverage the cmmi to help them improve upon their processes. and they decide that they would like to achievea cmmi level 2 maturity rating. this is an organization that is managing their processes.they've characterized them at a project level. and it's still often a bit reactive. an organizationthat is going a step further is saying let's define our processes. let's train our people. let's ensure all projectshave exposure to the processes, to training and let's set some expectations if you will,some policies and guidelines for how the organization should operate. this would be considered alevel 3 maturity.

the box to the right of this waterfall-ishlooking diagram includes process areas that would be implemented by an organization atlevel 3. so at level 2 you would be implementing several of these process areas. and at level 3 you'd build upon that and itincludes organizational process areas, represented by the ot and that ops and opm, which arenot listed. at level 4 an organization is saying okay, let's take our defined processesthat we've been following, and let's measure them quantitatively. this is where six sigma can offer some insightsfor an organization or other types of measurement or quantitative analysis approaches can beused.

at level 5 the organization is looking atthat data very discreetly and saying, how do we optimize or get the most out of theseprocesses that we've been measuring? and where do we improve and make changes? so let me tell you a little bit about thecmmi as far as what it is and where it came from. it's a capability improvement modelthat can be adapted to solve any performance issue at any level of the organization, reallyin any industry. the model provides guidelines and recommendationsfor helping organizations guidance its problems and improve their performance overall. itis used by over 5000 organizations from more than 70 countries all over the world. thecmmi can help identify and achieve measurable

business goals. what you're seeing on the screen is a processmaturity model quick look. this came from the cmmi institute who is now the sponsorof the cmmi. and what you're seeing are the various appraisal responses from different parts of the world. for example, asia has had 6412 appraisals;50% - 56% of their organization is organizational population is using the cmmi. and that hasincreased over the years. north america is still using the cmmi as a best practices frameworkbut certainly not as much as asia is. what this tells you is that it's still aninternational model that makes sense. while it's not always being used within the unitedstates or in north america, primarily the

united states, it's a model that has a commonlanguage across our international borders. the cmmi consists of a set of process requirementsbased on industry best practices that are organized into 22 different process areasacross four categories. those categories are process management; project management; engineering;and support. so the cliff notes for you are that the cmmiis a capability improvement model that includes guidelines and recommendations for an organizationto implement in order for them to be well positioned for a cmmi appraisal which yieldsratings. it's used by over 5000 organizations in morethan 70 countries. so we're in good company in other words. the cmmi also comes in variousflavors. they call them constellations. these

are known as the cmmi core development fororganizations that are doing software or system development. whether they're using agile or not. they alsohave cmmi for services which primarily is used by organizations that have a servicecomponent such as professional services, such as help desk; call centers; those types ofservices that may also have a little part that is considered it or software development. they have people in them which is a modelused for managing an organization of resources that are human in nature, meaning how do weinteract with the people within our organization? and how do we improve upon the interactionof our organization at the people level?

there's also the cmmi for acquisition whichis a model that can help organizations determine how do we want to manage and run our acquisitionlifecycle, i.e. procurement? this is very relevant in the - in the world of contractmanagement. so there are various constellations of thecmmi that can be leveraged by organizations. many organizations leverage cmmi as a processimprovement framework and a capability baseline tool to help them ensure performance needsand internationally recognized standards. now as far as some background information,the cmmi and its predecessors, known as the cmm, the capability maturity model has beenworked on by the software engineering institute for over 25 years.

now the software engineering institute wasfunded by the department of defense for many years as a research and development tool sothat the dod could have a measurable way to work with contractors, i.e. consultants; i.e.vendors so that when they were doing software development or system development or shipsystems building or air frame development, aircraft development, etc., they would beable to have a common language between all of those types of work and deliver a resultof high quality. in 2012 the dod decided that it wanted thesci to focus all of its resources on evolving other technologies more urgent to the dod.and they believed that the cmmi had reached the level of maturity that it no longer neededto be its sponsor.

therefore, the cmmi institute came about withcarnegie-mellon, the university to basically the sponsor for cmmi at the juncture. youcan go out to the cmmi institute web site and see all about it. so let's take this informationa step further. so what if your organization does want toleverage cmmi? we'll talk further about models on the next slide, where we take a look atthe cmmi ideal model. what you're seeing on the screen in this graphic is basically anorganization has a stimulus for change of some time. context needs to be set for the type of changethat is needed. and sponsorship needs to be developed. this is a model for improvementand cmmi has coined the phrase, the ideal

model. so it has multiple phases - initiating;diagnosing; establishing; acting; and learning. and within each phase there are very specificactivities such as creating a charter, characterizing your current stage. that would be doing ananalysis of your current capabilities against a particular model. say for example, you wanted to assess yourorganization against the cmmi for development, then you would do a gap analysis based onthat in your diagnosing phase. and perhaps you already know some of the areasof improvement in your business and so you would add those on and perhaps try to alignthem to where the cmmi might help you to address them.

you would then develop recommendations, setpriorities, develop a tactical approach within your organization, create action plans, derivesome solutions, perhaps pilot and test some solutions within your organization. refinethose solutions. implement them across the board. analyze and validate and propose future actions.this is actually the model that icf international has used for years for its process improvementorganization. for more than eight years, i've been managing our in process improvement - ourprocess improvement organization. and for more than 15 years we've had a capabilityin house of leveraging the cmmi ideal model as a way for us to look at and solve our businesschallenges internally. or perhaps optimize

the business processes that exist. another model that could be helpful for anorganization is six sigma. in this graphic six sigma is here - define; measure; analyze;improve; and control. you'll notice that a lot of the same terms are used in six sigmaas are used in the process improvement model known as cmmi ideal. six sigma may be used in an organization thatis interested at perfecting its process as close as possible to being a defect free organization.this is not necessarily the most appropriate for all organizations, but perhaps part ofyour organization would benefit from it. it's often used in manufacturing or as partof lean principle adoption. project management,

while it is not a process improvement model,could be seen as a way to improve upon your work. and that's why it's represented hereas plan, do, check, act. you may have heard these terms before withregards to project management. and the bottom line is you do some level of planning. youperform the work. you check in on your work and you see how that work is going againstthe plan, i.e. monitoring and controlling it. and then you take action. some may refer toit as corrective action. others may refer to it as improvements. and you basically continuethrough that process over time. so these are some examples of models at work and what'sincluded in them.

so let's have a polling question to see yourlevel of experience with any of these models. >> elizabeth mertinko: okay audra. i've goneahead and opened up the poll. so i'd ask our participants to go ahead and let us know.the question - which of the project management models have you worked with or are you workingwith currently? you can choose multiple answers. and we have a little under half of you thathave voted. so if the rest of you could just jump on there and let us know what your experiencehas been with these different models. and i'll give you just a few more seconds to goahead and put your answer up there. okay. with about 80% of our audience voting,i'm going to go ahead and close the poll. and there are our responses. so it looks like15%, cmmi ideal; 23%, six sigma; 46%, project

management institute or pmbok; and 77%, statelevel or agency specific pmo models. >> audra stewart: excellent. this is alsovery insightful. and in our question and answer session i'll ask some questions around thepmo models. thank you for taking that poll. it's very helpful. so let's move to the nextarea which is reality checks and takeaways. so i don't know if you've heard this on theradio before. it's the washington post so if you don't get it you don't get it. thisis really going to take us into the next part of this webinar. so let's go forward withthat and talk about risk. it's in there. there's one way to think about if we don'tget it we don't get it, which is the organization that's truly embraced whichever model or approachthey are going to leverage; train their people

and have identified risks associated so thatthey can move from the paradigm of magic and art to the other side which is where riskis reduced in the realm of predictability, repeatability. and some may call that morescience or hard science and engineering. the idea is that we would move from heroicsand luck and agility and unpredictability to a more productive higher quality environment. and that typically comes with an organizationsetting the tone, setting the practices, training their people, listening to their needs andthen moving into what we call a monitoring phase, which is ensuring that it's being performed.this is not an easy task. but models are a great way to help us with that.

let's look at that a little bit further andtalk about what's in it for me? so here are some questions that you may be asking yourselvesand you may have other questions that you've been asking yourself. i bet you want someanswers. so let's look at the first question - whatdo i get when a vendor has a cmmi rating? vendors do not have, get or do cmmi. that'sone of the first things that should be known. they leverage the model and create policies,practices, techniques, possibly templates for how they do business, against a cmmi model.so for example, if my organization has adopted the cmmi for development at 22 process areasthere that i need to demonstrate my organization is implementing.

well what if my organization has 3000 projects?are all of them doing all of these processes? likely not. some organizations will - willensure that one project is implementing all activities at level 2 and level 3 and arecompliant 100% with the cmmi's goals. but not all parts of the organization maybe implementing all of those activities. on the opposite end, an organization that has3000 projects may not need to have all of them following the exact same process or theexact same set of processes, because perhaps the business is different. it's very important to understand your organizationand its business and not apply a general rule to the organization as a whole, but fullyunderstand its dynamics, its processes and

the nature of the work. because there maybe multiple approaches to performing that work. when it comes to pmo, not exactly sure whatyour pmos are saying to you specifically. so if it's something like we want more repeatableapproaches to the way we run projects, programs, interfacing with vendors, i can tell you thatthe cmmi will likely help them, it could likely help you if they understand how to leveragethe model as a tool but not as say for example, as the bible, as this is the de facto wayof doing business. which can sometimes create a rigidity thatdoesn't allow for the business to be flexible. want to know how to evaluate a vendor?

i thought it would be good to show you howyou can leverage the cmmi one process area at a time, even if your organization doesn'tneed to leverage all aspects of a particular cmmi constellation. so at this time elizabeth, i'd like to givethem a demonstration of the decision analysis and resolution workbook. >> elizabeth mertinko: okay. >> audra stewart: and to do so, i will sharemy screen. >> elizabeth mertinko: let me go ahead andchange over to you. and it should be inviting you to share your screen.

>> audra stewart: and it is. >> elizabeth mertinko: and there we go. >> audra stewart: okay. so one of the mostpopular process areas of the cmmi in our organization is decision, analysis and resolution. this is the process area of the cmmi thatallows an organization to use a quantitative approach to make decisions, significant decisions,such as which vendor should i use for a particular contract? how do i know that that's the rightvendor? so one of the things that we did in implementingthe decision analysis and resolution process, was to create a workbook that would allowus to do quantitative and qualitative analysis

of particular vendors. you could also leverageit for various technologies. so what you're seeing on the screen is a workbookwhere basically we've created a template which was actually the template that our staff woulduse, where they are able to fill in this information specifically for the decision that needs tobe made. they would conduct an evaluation and thenthey would make a decision and document that decision as to why they came up with it. sowhat i thought i would do is share with you one that is fully filled out, so that youcan see an example of how you could leverage one of the process areas at cmmi. so one of the first things you have to dois decide what decision am i making? and in

this case, with selecting a subcontractorto provide web design services on a particular project or contract. and you would identifyyour alternative. this may ring true to you, especially in thefederal government, where you need to do the three comparisons. and so here we have anexample of four. these are specific companies that we've madeup as an example, so that we can then describe that company in enough detail that would makeenough sense for the individuals reviewing this analysis. then we would go down and establishthe criteria. so here under criteria name, we've listeda few. you could have as many as needed. for example, if you were evaluating a proposal,you could use this as a way to evaluate a

proposal. the criteria of cost - i think that'salways important. experience, timing; for example, when are they available? have we done business with them before? andthen write a description for that criteria. so i'll evaluate costs as the most importantfactor in the decision as we have a tight budget. well how will i evaluate each vendor'scosts? we have a column here for evaluation methods. and we also provide additional guidance througha project team on, you know, here are some evaluation methods - modeling and simulation;engineering studies; cost studies. well which one makes the most sense for this criteria?a cost study.

so this would be a comparison of quotes receivedfrom each potential company or in this case, through evaluating a proposal, each vendor.and you could do the exact same thing for each of the criteria that would be associated. then you move over to the evaluate tab. i'mjust going to make it a little smaller so we can see it better. here is where you willsee that the criteria we entered in the prepared example, automatically comes over. from there,we can see that there's a column for applying a weight. it's important to identify how important arethe criteria? well first you must set a scale. and that scale for us is one, is the lowestpriority and three is the highest.

so because that's our scale, we are able toevaluate each vendor's costs at a rate of three which in this case, is of highest priorityto us, because we're mostly budget constrained for this example. and then this is a 2, a1 and a 2. what happens next is that each vendor mayprovide you with a cost analysis that you want to evaluate that perhaps you want somemins and maxes or some thresholds that you want to look at in more detail. and that way you can break up your ratingfrom a scale of 0 to 5 and in this case 5 being the best option. as you move to the right in this workbookyou'll see that each company that we're evaluating

has populated at the top, which allows usto do quantitative analysis against each vendor based on this specific criteria. so in this case we would rate the vendor basedon cost and say well they came in at 2-1/5 to $5000 a month for this particular service.and so because our rating details give it a 3, we're going to give it a 3. you can come up with your own quantitativerating approach. but in this example we have come up with 1 to 5, skipping the 0. thenwe calculate a rate or score. again, quantitative analysis that tells us they found data whatthe score is for that vendor in the criteria of costs.

this would continue on for each of your criteriaand it would give you a total score. now we're using excel in this case and we have createdsome formulas to help us to do the quantitative analysis. what's happening here that you'llsee is that this is the only cell in green. why? because it scored the highest. that particularvendor had the overall highest score. perhaps if you're using other types of techniquesin the industry such as delphi or wideband delphi or kokomo or other types of estimatingtechniques, you might apply some of that logic to this quantitative analysis. and say well wait, i thought that aim serverservices would be the one that we would end

up choosing but the data is telling me weshould choose cisco hosting corporation. why is that? and that's exactly the idea behinda quantitative analysis. it can help you look at the data and thenask questions. well why did it come out that way? are we sure we have the right weight?are we sure we have the right rating details? perhaps our rating and scoring of a particularvendor and a particular criteria may need to be revisited. or perhaps it's actually on the mark and itjust means you have to you are changing your decision that you were preconceived to make,before you came into this more structured approach to coming to the decision.

whatever your decision is, you would go tothe decide tab and you would document that decision. and describe why you chose the companythat scored maybe a lower value over the one that was the higher value. and there may belegitimate reasons for that. so what i'm sharing with you is an exampleof an organization being able to create a tool to leverage a process area of the cmmiand put it to work for you right away. now we do have a process description that definesall the details of this process. but at a minimum, we train our staff in theuse of this workbook and hopefully help them to come to make better decisions so that theycan provide those recommendations to their customer. so at this time that will concludethe demo of this particular workbook, and

we can go back to our webinar. >> elizabeth mertinko: okay. i'm just grabbingback control of the slides. and we should be able to see them now. >> audra stewart: excellent. so let's advance.can i get a guarantee - i'm sure most of you got some questions when you work with vendorsthat say they have the cmmi maturity level rating. the honest and short answer is ofcourse, no. but you can evaluate the scope of cmmi applicability in an organization. here are some questions that are potentiallyvaluable to ask of a vendor if they will be working with you, or of the state pmos thatmay be assessed at a cmmi maturity level rating.

it's important to note that you can leveragethe cmmi and never receive a rating. and the reason is that the cmmi will allowyou to do a staged or a continuous representation implementation approach which means if anorganization is doing staged or leveraging a staged approach, and they want their maturityof their capability assessed, i.e. having an outside organization come in and do anappraisal or an assessment of their capability, they will potentially yield ratings. those are called scampi and there's an applicablemodel for conducting a scampi. and only scampi as which there are three types; as, bs andcs; only as yield ratings. you can scope an appraisal in an organization to only lookat level 2 maturity practices.

or you can have it look at levels 2 and 3and 4. and if an organization is going to go for a level 4 appraisal that means they'vedemonstrated level 2, level 3 and level 4 process areas of the cmmi. so they build uponeach other. there is a technical report that i have providedto elizabeth as well, to patricia, that goes into more depth with regard to evaluatinga vendor and their cmmi credentials. because it's very important that the industryand the federal government as well as local and state governments, as well as commercialindustry, understand what they're getting when an organization says they are cmmi. andnot always do we know. so is the organization named in the appraisaldisclosure statement? so for example, if i

tell you that i worked at icf and i workedwithin a specific division in icf you may want to ask the question well, what is yourappraisal disclosure statement regarding the organization you work in, that will be performingthe work? has that organization been subject to an appraisal?or how many programs were included in the appraisal? for example, a 30,000 person company withover 20,000 projects could have one project appraised and they claim they are cmmi levelsomething, which would be an inaccurate representation of their capabilities. was the appraisal performedless than three years ago? why is this question important? because everythree years the cmmi requires a reappraisal

of the capability. let's see. why would thatbe important? if an organization is continually reassessing their capability then they'retruly committed to the improvement activities that go into that. because it is not an easy feat for an organizationto be in the position of being rated at a cmmi level. and so they expire every threeyears. so for example, icf expires in february. we are going for a reappraisal in february.which process areas were included in the appraisal? this is also important, because an organizationcould only focus on three of 22 process areas, which means that perhaps the problem they'retrying to solve is only in those three process areas.

but they may communicate that they do allof cmmi. and remember do is not something that an organization does with regard to cmmi.they leverage it and use it as guidelines to help them come up with a process that theywill implement in their organization. so while you can't get a guarantee there aresome things you can do. you can leverage the dar approach and do an analysis quantitativelyof your various vendors against their credentials with cmmi. so at this time we're going tomove onto some key takeaways. so process improvement is a journey, whetherit's at your state level from a pmo perspective or within your particular agency, or in privateindustry. no model will solve process challenges on its own. it's people, processes and tools.processes and practices are implemented by

people. in order for them to succeed, we have to givethem what they need in order to do so. and that can be a hurdle to overcome in an organization.right size your program management office processes by assessing the need and/or gaps.and you could leverage with cmmi i feel, as an approach to doing that. so at this time, i've come to the end of thepresentation. so that's all folks. i'm able to open it up to questions elizabeth, if youthink the timing is appropriate. >> elizabeth mertinko: sure. first, i justwant to take a minute and - and give you a huge thank you. and when i asked you to dothis presentation i didn't know that you were

prepping icf for recertification in cmmi. so i thank you particularly with the timing,for taking time out of your schedule to put all of this together and present it for us.and also, for dealing so gracefully, when our software crashed on us. so i really appreciatethat. lisa, could you give us instructions on howfolks can line up on the phone for questions please? do we have our operator? >> coordinator: thank you. at this time ifyou would like to ask a question, please press star 1 on your touchtone phone. you will bepromoted to record your name prior to asking your question. again, if you have a question,please press star 1 on your touchtone phone.

>> elizabeth mertinko: and while we're waitingfor questions on the phone, we do have one that's come in online. who is the entity thatconducts the cmmi level re level , reappraisal and then awards the appraisal disclosure statement? >> audra stewart: that's an excellent question.so the cmmi institute which is now the organization that sponsors all of the cmmi models and theproducts that come out of the cmmi, they require what they call cmmi lead appraisers, to gothrough extensive training in the model and in the fancy arc requirement which is therequired methodology for conducting an appraisal. since yes, there is a process for that. andthat lead appraiser would work with an organization and develop an appraisal team that would conductan appraisal. so to bring it home, for example,

our appraisal in february is a ten day appraisal. it consists of six icf staff and one cmmilead appraiser. so a cmmi lead appraiser can perform level 2 and level 3 maturity assessmentswith a team for high maturity appraisals, i.e. there's that level 4 and 5. the appraisermust have high maturity appraisal training. so they have to understand in more depth,what level 4 and level 5 organizations need to achieve where someone at level 2 and level3 don't have to have all that additional extensive training. hopefully that answered your question.let me know if it didn't. >> elizabeth mertinko: okay. perfect. thankyou. and can an agency lose a level? so can you be a level 4 and then go down to a level3?

>> audra stewart: we, in the industry, callthat backsliding. and can you lose it? technically, let's put it to you this way. your cmmi doesnot certify you, but i know that's a term that you use. so for the sake of conversation,a cmmi appraisal rating, i.e. certification, does not get lost. but an organization can backslide if theydo not continuously monitor the implementation of their processes that were appraised. hencethe reason why there's a three year expiration on an organization's appraisal. >> elizabeth mertinko: got it. lisa, do wehave any questions on the phone? >> coordinator: at this time i show no questions.

>> elizabeth mertinko: okay. so next questionthat we have online, is this an expensive process? does it make more financial sensefor an agency to add cmmi processes to their own state pmo approach without getting theappraisal? and i'm just remembering back to our poll,i think about 72% of our participants were using a state or an agency specific pmo approach. >> audra stewart: so back to the pmo approachis being used, i'm not sure what they're based on. but i'm going to take a gander to assumethat they're stocked with expertise from the industry. which means they probably have beenexposed to the pmi. they may have been exposed to the cmmi atsome point in their lifetime, especially if

they're in the software development community.but it's also possible that they haven't. so not knowing exactly where the level ofadoption is with regard to best practices, i'll answer you this way. it can be an expensive process. it dependson the size of your organization and the scope of an appraisal should you decide to go foran appraisal. so let me give you some tangible examples. if an organization wants one project to beappraised within a particular bureau, agency, division, etc., that level of effort to getthat one project appraised, could in actuality, almost be the same amount of effort to raisethe capability to an organizational level

and have it implemented by multiple entities,multiple projects. we kind of call that cmmi in stealth modeor cmmi at level 3 at cmmi at level 2 with level 3 in mind. so you're basically lookingahead with your vision and saying, we want to demonstrate a level 3 capability but wejust don't have the manpower to do it right now. you may choose to take the bigger chunk andsay let's go ahead and put the infrastructure in place for level 3, meanwhile we'll alsobe doing activities at level 2. and at some juncture, milestone, intersection point, we'llbring it all together holistically for the organization.

if you choose to just do one division, lineof business or entity within an agency, and there are six projects there, they all dosimilar work, they have similar costs and fees but they don't communicate well. their intersections with, you know, interfacesor perhaps stakeholder involvement or core you may find it advantageous to take someof the level 2 and, you know, project management activities and monitoring and control andstakeholder involvement activities and hone in on those. and just focus on those and come to a commonunderstanding across those projects without ever going for a maturity level rating. therewill still be a level of effort and costs

incurred but it's different. yes. we have to pay for an outside lead appraiseras an organization, for that person's time, plus the past time spent for that period inan appraisal and yes, that's a cost to the at icf we have another layer which is we investquality and process improvement activities in the performance of the work. so we havea function and a team of people who are responsible for these activities, which do not bill toour clients. so it's an investment that the company hasmade, in ensuring these best practices are available and in place. so hopefully thatanswered your questions; maybe more than you needed.

>> elizabeth mertinko: no. that sounded perfect.thank you. what is the key limitation for government in approaching cmmi? and relatedto that, what is the key effort required for cmmi implementation? >> audra stewart: elizabeth, could you repeatthe first part of that a little bit slower for me? i had a little connection problem.sorry. >> elizabeth mertinko: sure. what is the keylimitation for government in approaching cmmi? and then secondly, what is the key effortrequired for cmmi implementation? >> audra stewart: i'm not sure what key reallymeans in this case so i'll try to read into that a little bit. there are potential governmentlimitations in the areas of sponsorship.

maybe you believe it's a good idea becauseyou've been exposed to the model and worked with companies that really understand whythey're leveraging the model. and so you want your program managers or your cios or your,you know, to adopt the philosophy or in other words, drink the kool-aid. that is a stakeholder involvement sponsorshiphurdle to overcome; i believe to be the most critical element. and there are ways in whichyou can bring them along. one is to clearly identify the problems within your organizationin a bulleted way, something very simple. tie it to business value. what is lacking in the organization becausewe performed poorly in this area? and what

would be your to be state and what could bethe business value associated with that to be state, if you were to implement specificactivities based on the guidelines of the cmmi? that package right there is informationalto your senior management and sponsorship is critical. so that could be a key limitationbecause there may be some folks who believe that the cmmi is archaic, shouldn't exist,isn't relevant, etc. and i understand why, because there are implementationsof it in the industry, that make you feel that way. so it's all about how you employit. the second part of your question, elizabeth could you repeat that please?

>> elizabeth mertinko: let me see if we stillhave it here. >> audra stewart: key limitations... >> elizabeth mertinko: yeah. and what is thekey effort requirement required for cmmi implementation? >> audra stewart: okay. so again, you're notimplementing cmmi. you're implementing process improvement practices in an organization.therefore, it's a hard question to answer because depending upon how you big your organizationis; how many of your processes you want to improve over time, can dictate costs. so to make it tangible for you, for example,we've implemented cmmi level 3 for over the last at least 15 years, in a holistic way.we've invested in over eight appraisals during

that time at level 2 and level 3, as wellas having a staff internally that manages the process and performs an auditing function. so that our processes do not become stale,we're constantly taking feedback from the users of the process and improving upon it,to meet the needs of our customers better. so an exact key element here is right sizeit. if you want to achieve a maturity level ratingand that's critical, then yes, there will be an investment, a level of effort and costsassociated. to give you some perspective, when we did it, we had two people developingthe processes initially. we created a software engineering processgroup of 20 constituent that it was not their

full time job. and we had them go out to theprojects and grab the best of the best, the cream of the crop examples of what they weredoing. we came together as a working group, evaluatedthem against the cmmi in the particular process area, and determined what's the right sizeof this for our organization; we developed it and then we piloted it; and then we rolledit out organizationally. >> elizabeth mertinko: thank you. >> audra stewart: hopefully that gives yousome insight. >> elizabeth mertinko: lisa, do we have anyquestions on the phone? >> coordinator: at this time i still showno questions.

>> elizabeth mertinko: okay. we have severalmore online. so one of our online participants says i can only read pmboks materials if ijoin pmi. but can i read cmmi content on the cmmi site without having to pay? >> audra stewart: the answer to your questionis yes. >> elizabeth mertinko: excellent. >> audra stewart: the cmmi model for developmentservices, etc. are available on the cmmi institute. one caveat to that is this - it used to besponsored by the federal government, i.e. dod. it is now a for-profit organization.so it's important to note that while it was free, i'm not sure it will always be free.

>> elizabeth mertinko: so go now. okay. >> audra stewart: i'm sorry? >> elizabeth mertinko: i said, so go now,would be the take home message. go get it now while it's still free. so the next question- how can i verify that a vendor has actually achieved the level that they may have statedin response to an rfp? >> audra stewart: okay. there is a web sitethrough the cmmi institute - i can provide the url and it can be sent to you later - whereyou can go out and see the appraisal disclosure statement for a particular vendor. and i highly recommend that you do that ifyou're trying to do a contract or an acquisition

with a vendor who says they are. you shouldmost definitely verify that it exists. it will list the scope of the appraisal. it will list - it will list the projects involved,the entity or the specific organization that was appraised, the percentage of the organizationthat was included in the appraisal and what level and what cmmi model they're using. >> elizabeth mertinko: okay. perfect. nextquestion - now that carnegie-mellon owns, operates and supports, what do you think thefuture is for cmmi? >> audra stewart: well that's an excellentquestion. i was just doing some research on that as a matter of fact, in preparation forthis webinar. and they are expanding how they

view the world, because they realize the worldis changing in front of them. so what they're trying to do and they reallyhave a president who is interested in meeting industry needs, not just the dod specificneeds of the past. but today we have these agile environments. we have the - the needto be more like a digital strategy approach. and so what the cmmi is doing is looking atthe specific and i'll talk specifically about development, they're looking at the developmentmodel and they're thinking of ways that they could remove some of the less valuable practicesthat are included in the model. and they want industry to feed them that informationby going out to their web site and submitting change requests so that they can note specificallywhat is valuable to you within your organization

and what isn't. they're also wanting to change the way inwhich appraisals are performed and conducted minimizing costs to industry so that moreand more folks will continue to have appraisals and be recognized for that standard. they're modifying the model which they didin their last release of it, version 1.3, by including specific language related toif you're an agile implementation, here are some case studies and here are some modificationsin the way an organization may do the work. and - and one way to think about it is wherewe used to have binders on the shelf for everything, we now have web sites and web enabled tools,how we perform our job. so how do we show

evidence that we're doing something? so they'reworking towards improvements in that area as well. they're also evaluating their appraisal approachso that they can respond to some of the feedback they've been receiving over the last few years. >> elizabeth mertinko: thank you. lisa, dowe have any questions on the phone? >> coordinator: at this time i show no questionsor comments. >> elizabeth mertinko: okay. all right. wellthen i'm going to go ahead and wrap things up for today. audra, i just want to thankyou again. i know you put a great deal of time and thought and effort into this presentation.and i think you shared some really valuable

information with us. so thank you again for agreeing to be ourparticipant today and bringing us all a little bit further in our understanding of cmmi. >> audra stewart: thank you so much. it wasmy pleasure. >> elizabeth mertinko: okay. so to our audience,i hope that the information we shared with you today was informative and valuable. iwant to remind you all to register for the june webinar once that announcement is released. and that will go out over several of the children'sbureau listserves, so keep your eyes peeled for that.

if you have any questions regarding today'stopic, that you didn't get a chance to ask or you'd like any information about any ofour scheduled webinars or particularly if you would like to either suggest a topic orvolunteer to be a presenter, please contact me at the email thats listed above. that's elizabethmertinko@icfi.com. i alsowant to remind you that this webinar has been recorded and we'll make it available onlineat the children's bureau's dss web site. when it's complete and posted, we also will notifyyou via the sacwis managers listserve, with the link. and with that, i'd like to close things upfor today and thank you all for your time

and participation. end