Kevin Ferrara, AFSO21
Assessment versus Evaluation
Often, individuals confuse assessment and evaluation as being the same however, it’s important to understand that these two functions are quite different. Within emergency services, assessments are a means of gathering, reviewing, and implementing data with the goal of increasing the performance or processes of an organization. We often find assessments being used when dealing with emergency response times. With each response, specific times are recorded which allows an organization’s leadership to go back and assess if their responders are getting out the door and on scene in a timely manner. If they are not, this data is the basis of an investigation of sorts to determine why they are not. Essentially, the process is being scrutinized to determine where improvement can be made.
Evaluation on the other hand is how an individual or organization, a product per se, is looked upon based on a specific set of standards. In emergency services where capabilities of service are considered, we often speak about Levels of Service (LoS). LoS ranges from optimal to inadequate. For example, a fire chief may specify that for their department to provide an optimal level of service to the community, their department’s specified Key Performance Indicators (KPIs) must be at or above 90%. Should those KPIs fall below 90%, then the LoS would be considered a reduced level, and so on. Similarly, when evaluating recruits, a product of the organization, specific standards such as height and weight may be considered to determine if they meet the requirements for the position.
Community Risk Assessment and Standard of Cover (CRA-SOC)
Everything you do in life can be analyzed using some type of data assessment process. How that data is collected, analyzed, and used may determine what type of job you are qualified for, what type of education you may receive, what size home you may purchase, what might be your next meal, and as we have seen on television commercials, data sometimes determines who you may fall in love with.
The fact is, data can be manipulated to display a desired result. However, when it comes to emergency response, data must be based on facts, not assumptions and emotions. When the latter influences data, the outcome, despite appearing to be the desired result, tends to cause more problems than solutions. When this occurs, organizations find themselves wasting time and money to review the data again and create solutions in response to it, time and money being commodities that are precious and limited, especially in a volunteer emergency service organization.
Knowing that no two locations in the world are identical which includes landscape, infrastructure, resources, and especially residents, you and your agency should explain and document the following:
- Identify unique characteristics of your community
- Apply methodologies to perform an all-hazards risk assessment
- Determine your response strategies relative to your community’s unique hazards and risks
- Assess the historical quality of emergency response performance in your organization
- Develop a Gap Analysis to identify specifically where quality performance exists and where quality of performance is challenged
- Establish plans for quality maintenance or improvement as necessary
Additionally, there may come a time when a single organization is faced with dissolving due to the inability to provide the level of service the community requires. One manner to avoid an organization from becoming dissolved is to combine with another organization that provides similar or identical service. We’ve seen consolidation occur with 9-1-1 call centers and Emergency Operations Centers (EOCs). Not only does this reduce physical footprints in terms of buildings with the community, but reduces costs as well.
Response Times
Time is critical when responding to an emergency. Every emergency response begins at the moment the emergency notification is received. Many urban response areas are protected by paid responders who staff their agencies 24/7; therefore, residents can rely on emergency responders to arrive rather quickly, sometimes within a few minutes. However, in rural or even remote areas such as those in the mountains of central Pennsylvania or the remote highways cutting through the deserts of Arizona and Nevada, response times may take longer.
Not all emergency responders are paid. In 2013, the US Fire Administration reported that of the 30,052 fire departments across America, 19,807 are volunteer consisting of 786,150 volunteer members. Unfortunately, the number of volunteer firefighters is declining each year, thus putting the communities they serve at greater risk. Because many volunteers are employed during the day with many commuting outside their local area, response times may be extreme. As a result, the primary response agency may be delayed or not even respond due to the lack of manpower.
Apparatus & Equipment
An emergency response agency can be fully staffed and possess the highest level of training and certifications possible for each role, but could be considered ineffective if the apparatus and equipment necessary to perform their functions is inadequate.
When evaluating your Emergency Response Capability (ERC) it’s important to ask the following:
- Does your agency have the proper equipment to support your community?
- Is your apparatus effectively covering your response area?
- Does your agency have a replacement plan for apparatus and equipment?
- Does the future state of the community require additional assets to ensure coverage and if so, how will those assets be acquired and paid for?
Budgeting
Let’s be honest, if money grew on trees, we’d all have orchards. The reality, it takes money, capital to keep an organization operational and when that capital starts to decline, organizations have to take a real hard look at the root cause of why the trend line is going down instead of up.
Budgeting is not a science but it does require discipline. In the simplest way to describe proper budgeting is to not spend more than you receive. If you can stick to that simplest methodology, you can stay afloat. But then come those unexpected expenses that eat up the reserves that were budgeted for. Are you prepared or will you have to seek out sources of revenue to compensate the loss of funds?
Some budgetary actions and reviews each emergency service organization should be conducting include:
- Budget development and maintenance
- Personnel costs
- Capital expenses
- Apparatus expenses to include replacement options
- Sources of revenue
- Personal protective Equipment (PPE)
- Prioritization of resources
As mentioned previously, every organization requires capital to operate. Your organization may be fully staffed with the best trained individuals, and the newest apparatus may be siting in the apparatus bays waiting for the next emergency, but if there is no money to pay your staff, put fuel in the apparatus, or keep the power on to operate your computers and charge your portable radios and gas monitoring equipment, then somewhere, somehow, opportunities were missed and priorities were mixed.
Staffing
No matter how much equipment your agency possesses, it’s useless if your staffing is not adequate to operate it effectively. You must work with your agency leadership and the Authority Having Jurisdiction (AHJ) in your response area to identify where improvements in staffing might need to be made. In some communities, the AHJ may be city council board, a Mayor, a township Supervisor, etc.
Some questions to consider when evaluating your staffing capabilities include:
- Does your agency have adequate staffing to support the need of providing effective coverage?
- At what point does your agency become a risk when staffing is reduced?
- Does the future state of your organization necessitate additional or reduction of staffing?
- Who determines your staffing levels and how are they monitored?
Of great importance is to collect your agency’s staffing historical data and compare it with the needs of the area that you support, determining when, where and why a change in staffing was made and if it was necessary. Many times, staffing is reduced because of the lack of incentives, especially within volunteer agencies. Because volunteers are not paid to perform the identical functions their paid counterparts are compensated for, the question many volunteers ask themselves is “why am I risking myself for free?” This is your opportunity to describe the value of their service, in both tangible and intangible terms.
Performance Assessments and Evaluations
Staff performance assessments and evaluations serve several purposes for the organization and employee. Performance assessments identify standards and expectations set forth by the employer. It is important for the employer to be clear with their staff when identifying expectations therefore time must be well spent to develop an effective performance assessment system. With performance assessments, a fire chief or other officer may want to know if a particular firefighter is performing to the expectations he or she established such as putting on their PPE in a specific amount of time, how well their instructors are presenting and students are receiving information, or how effective the department’s fire prevention program is.
Performance evaluations provide a means for reliable, long-term, cumulative record keeping of performance and promotion potential. For example, evaluations may involve documenting a staff members attitude, positive and or negative and how it affects the rest of the organization. Evaluations may involve looking at how well an individual meets or exceed at their key duties, tasks, and responsibilities. Evaluations may include looking at how well an individual performs to accomplish an organization’s mission to include team building, mentorship, communication skills, compliance with standards, etc. Creating and maintaining a thorough evaluation record will enable leadership to make strategic decisions in a more efficient manner. In other words, performance evaluations help to determine if the employee is worth retaining or promoting.
Performance assessments and evaluations can document information regarding professional development accomplishments and opportunities which is important when seeking the best candidates for crew assignments, training, promotions, or other personnel management decisions.
Understanding the structure and culture of your organization is a must. Knowing this puts into context the type of environment the organization operates in and how effectively staff performance evaluations must be accomplished. Staff performance evaluations do not need to be complex however, they should be structured in a manner that captures the needs of the organization, the needs of the staff member, and the needs of the community, the stakeholders, to which both the organization and staff member serve. If your staff are under-performing in the non-emergency situation, how well can you expect them to perform during an emergency situation?
Every organization has at least one staff member that requires some type of encouragement at times, sometimes more often than not. You may have staff members who are amazing while engaged in activities they enjoy but when it comes to performing mediocre tasks like mopping the floor, taking out the trash, or washing the hose or apparatus after an emergency response, they may be a bit laid back; pushing the task off to someone else, say the new recruit. This is where staff performance evaluations are necessary. Each staff member no matter the scale of the organization or their role within it, has value to the organization. However, if that value is not held to a specific standard, any decline could impact the overall organization thus causing stakeholders to perceive their return of investment (ROI) is not what they expect.
