BOARD OF SUPERVISORS

BUDGET ANALYST

1390 Market Street, Suite 1025 San Francisco, CA 94102 (415) 554-7642

FAX (415) 252-0461

April 23, 2007

TO: Supervisor Ammiano

FROM: Budget Analyst

SUBJECT: Updated Status Report on Implementation of the Recommendations in the Management Audit of the Office of Emergency Services

At the request of the Board of Supervisors (Motion No. 05-119), on May 15, 2006, the Budget Analyst completed a Management Audit of the Office of Emergency Services, now known as the Division of Emergency Services (DES)[1]. The purpose of this Management Audit was to evaluate the efficiency, economy and effectiveness of DES' programs, activities and functions. Specifically, the Budget Analyst reviewed DES' activities related to legislative authority, communication and coordination, strategic planning, grants and budget, financial management, emergency plans, pre-disaster mitigation and recovery planning, emergency response, training, exercises, equipment, as well as management, organization and staffing. This Management Audit contained 72 recommendations for improvement in the above described areas.

The Board of Supervisors Budget and Finance Committee scheduled an initial hearing on the Office of Emergency Services Management Audit on May 17, 2006. Following the initial hearing, the Government Audit and Oversight Committee held a hearing on the implementation of the Budget Analyst's Management Audit recommendations on September 11, 2006. At this September 11, 2006 meeting, the Government Audit and Oversight Committee requested that the Budget Analyst report back regarding the implementation status of the Management Audit recommendations.

This report summarizes the implementation status of the 72 recommendations contained in the Budget Analyst's Management Audit of the Office of Emergency Services. The findings in this report are based on information provided by the Division of Emergency Services, in response to the Budget Analyst's request that the Division of Emergency Services provide documentation demonstrating the implementation status of each recommendation. Other than follow-up discussions with select City department and Division of Emergency Services staff, the Budget Analyst did not conduct additional independent research into the efficiency, economy and effectiveness of the Division.

The Budget Analyst notes that this report is a snapshot of the implementation status of the recommendations, as of February and March of 2007, or approximately ten months after the issuance of the Management Audit. However, many of the recommendations contained in the audit, such as maintaining effective communication with departments or issuing advance notices of meetings, require on-going efforts to successfully implement. Therefore, the implementation status of many of the recommendations in the Management Audit will continue to change.

As previously stated, the Budget Analyst's Management Audit of the Office of Emergency Services contained 72 recommendations for improving the efficiency, economy and effectiveness of the Office. The Management Audit assigned each of the 72 recommendations a priority of either 1, 2 or 3 based on the following criteria:

The Management Audit included 34 Priority 1 (47 percent) recommendations, to be initiated in the first six months following the audit, 20 Priority 2 (28 percent) recommendations, to be implemented seven months to one year following the audit, and 18 Priority 3 (25 percent) recommendations, to be implemented approximately one year following the audit. As this status report is being completed approximately ten months after the completion of the management audit report, DES should have begun implementation of the Priority 1 and Priority 2 recommendations.

Attachment I contains each of the Budget Analyst recommendations that were included in the May 15, 2006 Management Audit. The Attachment identifies (a) each recommendation by Section number, (b) a brief description of the recommendation, (c) the party responsible for implementation of each recommendation, (d) the assigned Priority of 1, 2 or 3 as discussed above, (e) DES's general response on their status in implementing each recommendation, (f) the supporting evidence that the Budget Analyst used to confirm or verify the status of implementation of each recommendation, and (g) the Budget Analyst's overall assessment of the status of each recommendation. The Budget Analyst's overall assessment identifies each recommendation according to the following six criteria: Disagree with the recommendation (x), No Progress (0), Planning Stages (1), Significant Progress (2), Substantially Complete (3) and Complete (4).

Table 1 below summarizes the implementation status of the 72 recommendations based on the priority of each recommendation and these overall assessments by number of recommendations and percent of all the recommendations.

Table 1:

Implementation Status of Budget Analyst's

Management Audit Recommendations

(Number/%)

As shown above, DES has completed implementation of 18 recommendations or 25 percent of the Budget Analyst's 72 recommendations, and has substantially completed the implementation of another 13 recommendations or another 18 percent of the 72 recommendations. In addition, DES has made significant progress on implementing 9 recommendations or 13 percent and is in the planning stages for another 23 recommendations or another 32 percent of all the recommendations. Overall, DES has therefore at least begun implementation of 63 recommendations or 88 percent of the 72 total recommendations.

As noted above, there is disagreement with portions of seven of the recommendations. Of the two recommendations that DES has not made any progress on implementing, both were identified as Priority 3, or not needing to be implemented until at least one year after completion of the audit, which would not occur until at least May 15, 2007. Therefore, DES has begun implementation on all of the recommendations, that DES agrees with implementing.

The Budget Analyst notes that there are several recommendations that the Budget Analyst included that have still not been completed, substantially completed or made any significant progress. The most significant recommendations in this Priority 1 category address the need for DES to conduct a strategic planning process, as contained in Recommendations 3.1 and 3.2.

Recommendation 3.1: Senior Management of the Division of Emergency Services should conduct a robust strategic planning process. This process should include appropriate stakeholders, such as the Mayor and the members of the Board of Supervisors, leaders of City Departments, and private and non-profit organizations. The Division of Emergency Services should utilize any resources produced by previous administrations in carrying out this process. The strategic plan should address and prioritize planning, response, mitigation, and recovery activities based on the risk and capabilities assessment, as well as organizational goals and capacity. The Office of Emergency Services should review plans for other jurisdictions to help guide this process.

Recommendation 3.2: Senior management of the Division of Emergency Services should move forward with a thorough assessment of the City's emergency services capabilities. The Division of Emergency Services should use the identified gaps in capabilities to help prioritize efforts, such as training, within the strategic plan.

Strategic planning, including the identification of goals, objectives and performance measures, is an essential responsibility of an organizations' senior management, which helps to direct programmatic efforts appropriately, accomplish results, ensure accountability and properly manage financial resources. However, the Management Audit found that the Division of Emergency Services had yet to conduct a robust strategic planning effort that was consistent with generally accepted strategic planning guidance. As such, DES had not identified any goals for emergency services in San Francisco that were independent of the grant goals, nor identified any specific goals for DES itself. Furthermore, DES had not fully identified and prioritized the City's risks and needed response capabilities, and had not sought out input from policy-makers or stakeholders on the long-term goals and objectives of the City's emergency services.

When conducting the Management Audit, the Budget Analyst considered these recommendations addressing the need to undertake a robust strategic planning process to be the most important immediate priority for the Division of Emergency Services and central to the findings in the overall Management Audit report. From the strategic planning process, many of the other Management Audit recommendations, related to performance measures, staffing, planning, training, equipment, and sustainability would follow. In fact, many of the other recommendations were specifically intended to be addressed in the strategic plan, as identified goals, objectives and priorities, which would then establish a course of action for the Division of Emergency Services to follow.

Unfortunately, the commencement of the strategic planning process has been delayed for almost a year, from the time of the issuance of the Management Audit report in May of 2006. Based on a Request for Proposal process that was begun during the Fall of 2006, DES selected the consulting firm of ICF in December of 2006 to lead the strategic planning process at a cost of $750,000. This contract will be fully funded with Federal Homeland Security Grant Funds.

DES advises that due to a change in the leadership in the Department, a loss of several key staff and the City's cumbersome contracting requirements, the contracting process has taken significantly longer than anticipated. DES states that ICF is scheduled to complete the strategic planning work 34 weeks after project initiation, which should begin in early April of 2007. The Strategic Plan should therefore be completed in late November of 2007. As indicated in the audit report, strategic planning is a key component to many of the areas of improvement highlighted through the audit. Because of this, the delay in initiating the strategic planning process remains a significant area of concern.

Other Priority 1 recommendations that are still in the planning stages include: (a) the establishment of performance measures (Recommendation 3.3), (b) the participation of the Department of Emergency Management in SFStat (Recommendation 3.4), (c) expedited grant reimbursements (Recommendation 5.2), (d) annual State of the City's Disaster Preparedness (Recommendation 6.10), (e) emergency recovery efforts (Recommendation 9.1), and (f) corrective actions for future improvements (Recommendation 11.6), as individually discussed below.

Recommendation 3.3: Senior management of the Division of Emergency Services should establish appropriate performance measures. The Division of Emergency Services should seek help in this effort from the Controller's Office, as appropriate, as the City's lead agency for performance measure development. The Division of Emergency Services should use any existing tools, such as After Action Reports, as a foundation for these measures.

The Department of Emergency Management (DEM) has improved its performance measures in response to the audit recommendations. In particular, DEM has included performance measures related to current activities and has worked with the Controller's Office to establish new measures. However, DEM can improve these measures further by addressing the audits recommendation to use more outcome-oriented rather than output-oriented measures. In some areas, DEM made an effort to do this. For example, DEM combined the measures of Assessment of training program quality from attendee's perspective and Number of training courses to enumerate the number of courses and understand the impact of these courses. However, many performance measures are still output-oriented. For example, DEM uses Number of functional exercises conducted to measure Emergency Response Capabilities. DEM can take further steps to improve its overall approach to performance measurement by extending an outcome-oriented approach to all DES goals. As noted above, DES officials intend for ICF to enhance the existing performance measures through the strategic planning process by developing a solid capabilities assessment, and formally linking exercise evaluation guidelines to regularly assess the same identified capabilities.

Recommendation 3.4: The Division of Emergency Services should participate in the SFstat process. As part of its participation, the Division of Emergency Services should establish performance measures to help hold other City Departments accountable for carrying out emergency preparedness activities.

As indicated in the Management Audit report, DES' participation in the SFstat program would help to ensure continual improvement and public accountability of the Division's efforts. Although this had not previously occurred, the Mayor's Office has invited the Department of Emergency Management to participate in SFstat beginning on April 16, 2007. The Department will be included in SFstat as part of the Public Safety group.

Recommendation 5.2: The Grants Division of the Office of Emergency Services should develop and follow clear financial policies and procedures to ensure expedited grant reimbursement for future expenditures. In doing so, the Grants Division of the Office of Emergency Services should claim reimbursement-eligible expenditures as frequently as possible, but at least more frequently than on a quarterly basis.

The Management Audit found numerous problems with the Division of Emergency Services grant reimbursement procedures, resulting in over $7 million of unclaimed General Fund reimbursements and over $1,000 per day losses of interest earnings to the City's General Fund. To partially address this issue, the Management Audit recommended the implementation of new financial management controls. In addition, the Management Audit cited examples of other jurisdictions that claim reimbursements on a more frequent basis than San Francisco's quarterly claims for reimbursement.

DES advises that FAMIS (the City's Financial Accounting and Management Information System) is now set up to correspond to the specific project and solution areas that are funded and approved by the State. Therefore, DES advises that they can identify and reconcile expenditure information more easily and prepare reimbursement claims more quickly because FAMIS and the claim template have the same format. In addition, DES reports submitting a total of 47 claims for grant reimbursements on nine grants in calendar year 2006. While the Budget Analyst acknowledges that DES has implemented a new system in FAMIS to facilitate the claiming process, and has submitted numerous claims reimbursement requests, without a detailed comparison with data identifying when the expenditures were incurred by the City, the Budget Analyst cannot verify that the City's General Fund is not continuing to lose revenues due to delays in the preparation and submittal of such claims reimbursements.

Recommendation 6.10: The Division of Emergency Services should report on the status of functional, hazard-specific and City Department Emergency Plans during the annual Board of Supervisors State of the City's Disaster Preparedness Hearing.

In accordance with the City's Administrative Code Section 7.19, the Division of Emergency Services is to make an annual presentation to the Board of Supervisors on the State of the City's Disaster Preparedness. Mayoral Directive recommendation 5.2 issued in May of 2006 directed the Division of Emergency Services to make such an annual presentation to the Board of Supervisors on the status of emergency services in San Francisco. This hearing (File 06-0504) was introduced in April of 2006, but was never calendared. In conjunction with a hearing on the proposed update on the status of the Management Audit, the Board of Supervisors should calendar a hearing on the overall status of emergency planning in San Francisco, in accordance with Section 7.19 of the Administrative Code.

Recommendation 9.1: As required by the Mayor's January 9, 2004 Directive for Emergency Preparedness, the Division of Emergency Services should develop plans to mobilize key City departments such as Public Health, Human Services and Building Inspection for Phase II recovery of critical infrastructure and services.

On December 15, 2006, the City's Administrative Officer, Planning Department and DES convened a Task Force on Updating Seismic Safety Standards and Updating the Community Safety Element of the City's Master Plan. The updated Community Safety Element is intended to provide guidelines for rebuilding the City. This Plan is currently under review, which is expected to be completed by May 1, 2007, with environmental review completed by June 1, 2007. The Planning Department anticipates submitting the final revised Community Safety Element of the City's Master Plan to the Board of Supervisors for adoption in July of 2007. DES advises they plan to hire a consultant to prepare a recovery plan beginning in May of 2007.

Recommendation 11.6: Following corrective action, City Departments tasked with making improvements should report back on their progress in a public forum so that all Departments can be aware of changes and improvements City-wide.

DES's role is to facilitate and coordinate various departments in disaster planning, response, training, exercises and recovery. As part of this role, DES is responsible for evaluating emergency responses, providing feedback to other City departments and for keeping track of corrective actions related to emergency services. While some issues are insignificant, requiring minor adjustments, others may require more attention. DES is asking City departments to report back on their progress on corrective actions during the Disaster Preparedness Coordinators meetings. DES advises they are also currently using the areas identified as needing improvement for attention in future training and/or exercises. The Budget Analyst notes that follow-up by DES on such corrective action reports and trainings may also be necessary.

Regarding the remaining six recommendations, DES disagrees with implementing five of these recommendations and the Department of Human Resources (DHR) disagrees with implementing one recommendation. Each of these six recommendations, including the five recommendations which DES disagrees with and the one recommendation which DHR disagrees with are individually discussed below.

Recommendation 1.3: The Disaster Council should review each City emergency plan, annex, mutual-aid agreement, or report and should determine which plan, annexes, agreements, and reports to forward to the Board of Supervisors for public hearing and implementation.

Section 7.4 of the City's Administrative Code provides that the Disaster Council is responsible for developing plans for meeting any emergency, with the effective mobilization of all public and private resources and for recommending appropriate legislation to the Board of Supervisors, as necessary to implement such plans. The Disaster Council, which meets quarterly, is chaired by the Mayor and composed of numerous City department representatives and other private and nonprofit representatives. While numerous emergency plans are being developed by City departments, outside consultants and the Division of Emergency Services, our review during the audit found that other than Emergency Operations Plan 1, the Disaster Council was not reviewing or approving any emergency plans for the City. Furthermore, we found that the Disaster Council had not forwarded any ordinances, resolutions, rules or regulations to the Board of Supervisors for adoption.

During our follow-up status review, the Budget Analyst found that the process for reviewing and approving plans and protocols remains unclear. The City's Administrative Code has not been amended and therefore continues to require that plans and protocols be approved by the Disaster Council. However, a DES memo dated September 14, 2006 addressing the preparation and review of emergency plans did not include a review or approval role for the Disaster Council. Yet contradictory to this memo, the Disaster Council specifically approved the Earthquake Plan on December 6, 2006.

DES advises that taking each item to the Disaster Council, which is chaired by the Mayor and meets once per quarter, is not practical nor is it efficient. Instead, DES advises that using the Disaster Preparedness Coordinators and the Homeland Security Executive Committee, comprised of representatives from ten major City departments, who also are members of the Disaster Council, would be a more appropriate review body for such emergency plans. In addition, DES advises that there is a need to review this issue, in conjunction with the overall composition, role and responsibilities of the Disaster Council. DES advises that such a review is planned to begin in May of 2007. This review is anticipated to result in recommendations for amendments to the City's Administrative Code, which would then be brought to the Mayor and the Board of Supervisors for approval.

Recommendation 5.3: The Grants Division of the Office of Emergency Services should implement policies and procedures for claiming advanced reimbursement on encumbered funds as appropriate.

Although almost all of the emergency grant funds are paid by the grantor on a reimbursement basis, the Federal Urban Area Security Initiative grant guidelines for FY 2005-2006 allowed cities to request reimbursement on encumbered funds, if the City would expend such funds within 150 days of the claim for reimbursement. During the Management Audit, the Budget Analyst found that DES did not claim any such advance reimbursements. As a result, while waiting for reimbursements, the City loses potential interest earnings on General Fund monies.

However, in order to utilize this process, DES advises that the advanced funds would have to be placed in an interest-bearing account, the interest earnings tracked daily, and such interest earnings remitted to the grantor at least quarterly. Additionally, DES was concerned that any delays in the purchasing process could entail loss of the advanced grant funds plus any interest earnings. DES determined that this process was too cumbersome and time-consuming and that DES had insufficient staff to manage this process as well as manage the expenditures of the funds within the grant deadlines. In conclusion, DES disagreed with this recommendation because, according to DES, the long-term risks to the grant funding outweighed the possible short-term gains.

The Budget Analyst made this recommendation in direct reaction to the significant delays on the part of DES to claim grant reimbursements. Recommendation 5.2 in the Management Audit specifically directed the Grants Division in DES to develop and follow clear grant financial policies and procedures to ensure expedited reimbursements after the expenditures were completed. In doing so, the Budget Analyst recommended that the Grants Division of DES claim reimbursement-eligible expenditures as frequently as possible, but at least more frequently than on a quarterly basis.

Our follow-up review found that while DES has implemented a new system in FAMIS to facilitate the grant claiming process, and is submitting frequent claims for reimbursement, DES did not provide data to enable a direct comparison of when the City actually incurred costs relative to when DES submitted the claim for reimbursement. Therefore, the Board of Supervisors should continue to be concerned with claims reimbursements and potential losses of interest earnings to the City's General Fund. The immediate need to implement this recommendation (Recommendation 5.3) to claim advanced reimbursements on encumbered funds could be modified, if DES were to demonstrate expeditious and continuing practices of reimbursement claims post-expenditure relative to when the actual expenditures were incurred by each City department, as recommended in Section 5.2 of the Management Audit.

Recommendation 8.5: The Division of Emergency Services should equip all Department Emergency Operations Centers with the emergency supplies necessary to meet the City's 72-hour personal preparedness standard. In doing so, the Division of Emergency Services should have a plan for care and shelter of response personnel that, at a minimum, meets the City's 72-hour personal preparedness standard.

The Division of Emergency Services public outreach campaign urges all citizens to be prepared for 72 hours after an emergency without government or outside assistance. However, the Budget Analyst found that the City Department Emergency Operations Centers lack supplies required to meet the City's emergency responders' basic needs for food and shelter for the initial 72 hours. While the Budget Analyst acknowledges that the City's Department Emergency Operations Centers may not have sufficient space for all on-site supply caches, the City needs a system for the care and feeding of emergency personnel during the first 72 hours of an emergency.

According to DES, compliance with Recommendation 8.5 is not possible because supplies, such as food and water, are ineligible for grant funding. Therefore, DES's policy is that City departments should meet the City's 72-hour preparedness standard through individual departmental Continuity of Operations plans contained in a section of each City departments' emergency plans. In the meantime, DES advises that the Logistics Work Group, comprised of representatives from various City departments, will explore ways to house and feed the City's Disaster Service Workers, designated first responders and members of mutual aid contingencies.

The following three recommendations addressed the need to develop and implement performance measures for different aspects of emergency services.

Recommendation 8.1: The Division of Emergency Services should work with City departments to develop outcome oriented performance measures that will measure City departments' abilities to implement response protocols.

Recommendation 10.3: The Division of Emergency Services should develop performance measures that evaluate emergency responders' ability to perform Federal target capabilities and identified local capabilities and should pre- and post-test training participants.

Recommendation 11.7: The Division of Emergency Services should develop a system, such as performance measures, for making City departments accountable for improvement. To increase accountability, the Division of Emergency Services should report on these performance measures during the annual Board of Supervisors State of Disaster Preparedness hearing.

During the Management Audit, the Budget Analyst found a lack of consistent outcome-oriented performance measures to evaluate the success or failure of emergency services in the City. Furthermore, DES did not memorialize or distribute the lessons that had been learned from prior monthly tabletop exercises conducted at Disaster Forums. Similarly, DES had not provided timely feedback to departments following functional and field exercises nor provided timely reports regarding what actions City departments should take to improve upon weaknesses identified in such emergency exercises. While Federal guidelines required each exercise be evaluated and that a plan for improvement be drafted, DES had only produced a single evaluation and improvement plan for five exercises conducted in 2004-2005.

In response, the Division of Emergency Services advises that they have since developed over 30 performance measures to measure various aspects of City preparedness. In addition, DES has included the development of additional outcome-oriented performance measures as part of the strategic planning process. Our follow-up review also found that DES prepared individual After Action Reports to assess performance for the April 19, 2006 earthquake exercise and the October 19, 2006 Golden Guardian regional exercise within the required deadlines.

However, the three above-noted recommendations were included in the Management Audit because we also found that, although the Division of Emergency Services was funding technological and communications equipment, in addition to conducting numerous trainings and exercises for various City departments, the evaluations of these exercises were not identifying improvements in the City's response to emergencies. The Division of Emergency Services partially disagrees with implementing the three above-noted recommendations because of concerns regarding accountability and authority. DES advises that in order to be tasked with the accountability of these performance measures, DES would need to have the authority to require implementation.

However, DES advises that such implementation authority rests with each department head, not with DES. Therefore, because DES does not have authority to directly implement these performance measures within individual City departments, DES believes they cannot be held accountable for the resulting emergency response performance. The Budget Analyst acknowledges that the responsibility for implementing specific protocols lies with individual City departments. However, as the coordinating body for emergency response in San Francisco, the Division of Emergency Services must work closely with individual City departments and hold City departments accountable for implementing and improving their emergency responses.

Recommendation 13.6: The Department of Human Resources should establish specific policies and guidelines regarding the amount of time that City employees may spend attending conferences, classes or other outside training and professional development activities, while continuing to receive full compensation from the City.

During the Management Audit, the Budget Analyst found that the previous Executive Director of the Office of Emergency Services was pursuing an 18-month Master's Degree program at the Naval Postgraduate School in Monterey, California, which required on-site attendance two weeks each quarter and an additional 10-15 hours per week of supplementary coursework. While providing invaluable education and networking opportunities with emergency management professionals from throughout the country, the Budget Analyst raised the question of how much time City employees should be permitted to attend outside classes, preparation and on-the-job training, while being fully paid by the City. There are no City-wide policies or requirements for the amount of time a City employee may spend on such outside training, classes, conferences, or other activities while continuing to receive full compensation from the City.

In response, the Department of Human Resources (DHR) advises that such City-wide policies or requirements are not possible because each City department's training requirements will vary significantly, given the wide diversity of operations, businesses and job classifications that exist in the City. For example, DHR cites various mandatory training requirements for police, fire, attorneys, and other professional classifications, which are different from State certification and/or licensing requirements for appraisers, building inspectors and engineers. In addition, DHR notes that occasional, but significant training can be required for all City employees within a specific division or department if a new system is being installed. Therefore, DHR advises that it would be extremely difficult, if not impossible, to set a maximum number of hours a City employee may attend training or other professional development activities. DHR continues to support their existing policy that training hours be managed by each City department, at the discretion of the Appointing Officer in that department, based on the department's needs and operations.

There continue to be areas of concern, especially regarding DES's significant delays in implementing the strategic planning process. In addition to the need to expedite completion of the strategic planning process (Recommendations 3.1 and 3.2), further improvements are needed regarding the six recommendations of concern (Recommendations 3.3, 3.4, 5.2, 6.10, 9.1 and 11.6) discussed above. There is also at least partial disagreement with seven other recommendations.

However, DES has completed implementation of 18 recommendations or 25 percent of the Budget Analyst's 72 recommendations, and has substantially completed the implementation of another 13 recommendations or another 18 percent of the 72 recommendations. Overall, DES has at least begun implementation of 63 recommendations or 88 percent of the 72 total recommendations.

Of the two recommendations that DES has not made any progress on implementing, both were identified as Priority 3, or not needing to be implemented until after May 15, 2007. Therefore, DES has begun implementation on all the recommendations, that DES agrees with implementing. Although further improvements are needed, in the professional judgment of the Budget Analyst, the efficiency, economy, and effectiveness of DES has greatly improved during the time period since the completion of the Management Audit.

Attachment II is a 6-page memorandum dated April 19, 2007 from the Executive Director of the Department of Emergency Management, that responds to the findings contained in this report.

Harvey M. Rose

cc: President Peskin Supervisor Maxwell

Supervisor Daly Supervisor McGoldrick

Supervisor Dufty Clerk of the Board

Supervisor Mirkarimi Controller

Supervisor Sandoval Nani Coloretti

Supervisor Alioto-Pier Cheryl Adams

Supervisor Elsbernd Laura Phillips

Supervisor Jew Vicki Hennessy