· The Administrative Code charges the Controller's Office with establishing an auditing function to monitor the economy and efficiency of departments and agencies of the City and County of San Francisco.

· The Audits Division is not meeting the same level of productivity as other county and city auditors responding to the National Association of Local Government Auditors' Benchmarking and Best Practices Survey, and management does not manage and limit training and time reporting as closely as it should. The Division's Time Distribution Reports show that for fiscal year 2002-2003, Audit staff spent on average 61.6 percent of the time reported on timesheets on audit and project work. Training and professional development accounted for 6.1 percent of the time reported, other general tasks and administration accounted for 14.7 percent of the time reported, and leave from work accounted for an additional 17.6 percent of time reported.

· Further, Audits Division staff received an average credit of 62 hours of training for fiscal years 2001-2002 and 2002-2003. This is 55 percent more than the minimum requirements for training of audit staff. In addition, the Audits Division reported more training hours than required to meet continuing professional education hours. The inclusion of informal training hours increases the Audits Division reporting of training hours to 87 hours per auditor for FY 2002-2003, which is more than double the minimum standard of 40 hours per year for government auditors.

· The Audits Division does not complete all planned audits and tasks included in its Audit Schedule. If unnecessary and obsolete mandates were eliminated from the Audit Plan, then the division could refocus on more valuable audits including Payroll audits.

· At least one respondent to the National Association of Local Government Auditors' Benchmarking and Best Practices Survey reported productivity of 77 percent or more. The Audits Division's current level of productivity is 15.4 percent less than 77 percent. The 15.4 percent difference in productivity is equivalent to 2.65 FTEs for FY 2002-2003. The loss of 2.65 FTE in productivity is equivalent to salary costs of $220,895 based on the average salary for audit staff.

The Audits Division is made up of two groups. The Performance Audits group consists of one 1801 Audit Manager and four 1684 Associate Performance Auditors. Two of the four 1684 auditors are part time, equating to a full time equivalent workforce of approximately 4.2 FTE for the Performance group. The Financial Audits group is made up of three 1686 Audit Managers

and nine 1684 Associate Auditors, or 12.0 FTE. The Audits Division has one 1842 Management Assistant and is managed by one 1690 Audits Director. The mandated work of the Controller's Office Audits Division is discussed in the following section.

Mandates

The Audits Division of the Controller's Office is charged with performing quarterly audits of the City's cash and investments, auditing concessions under which tenants pay the City more than $100,000 in rent per year; franchise fees; 10 percent of contractors on a random sample basis for compliance with the MBE/WBE requirements; City loans, contracts, grants to verify funds have not been used for political activity; Cultural Centers; and charities licensed to solicit donations on the street. In addition, the Controller's Office has contracted with the PUC to conduct contract compliance reviews every six months. The mandated functions of the Controller's Office Audits Division also include evaluating the economy and efficiency of departments and agencies of the City and County of San Francisco and auditing the accounts of any department head or custodian of an account. Appendix 13.1 to this report is a list of all mandates for the Controller's Office Audits Division compiled from the Administrative Code, the Business & Tax Code, the Police Code and contractual agreements. The Audits Division prepares an Audit Schedule each year and subsequently releases an Annual Report describing the accomplishments of the division.

The Controller's Office acknowledges that the Audits Division does not complete all audits and tasks planned for in the Audit Schedule. The Controller's Office has stated that the Audits Division does not have the resources necessary to perform all planned audits. One of the main constraints on the resources of the Audits Division is that the division must perform unnecessary or obsolete functions. For example, the audits of charitable organizations mandated by the Police Code may not provide significant benefit to the City or its citizens. The Controller's Office has reported that these audits are of very little value to the City as the amounts of money audited are very small. However, such audits remain mandated by the Police Code.

The Controller should evaluate the risks and benefits associated with all of the current mandates and develop a prioritization of those mandates. Such an evaluation should have been performed previously and the Controller should have presented to the Board of Supervisors the risks associated with current mandates and requested the elimination of unnecessary and obsolete audit requirements in order to allow the dedication of staff to higher risk audit work.

For example, the Audits Division has included Payroll audits in the audit plan every year, but they have not actually conducted a complete Payroll audit. Payroll audits are not mandated by the current Administrative Code. In addition, Payroll is subject to annual audit by the outside auditors. However, if the Controller's Office was to evaluate the current mandates and find that Payroll audits are of higher value, the Controller could then present it to the Board of Supervisors as a priority function of the Audits Division and request that the codes be updated. If unnecessary and obsolete mandates were eliminated from the Audit Plan, then the division could refocus on more valuable audits including Payroll audits. While the Controller's Office maintains that it is the responsibility of departments to suggest Administrative Code changes, the Budget Analyst believes that the Controller should take the initiative to advise the Board of Supervisors on the relevance of these subject mandates.

Productivity

One standard for productivity is the percentage of total paid hours of an organization that are expended on direct outputs of that organization. Because productive hours are those hours actually expended in the production of the outputs of the office, namely audits, for the measurement of productivity of the Audits Division, we included those hours expended directly on audits, audit planning and follow-up, and those hours expended on Special Projects. We did not include hours expended on general tasks, administration, professional development and leave from work. The hours expended on general tasks were not included because the category does not meet the criteria for productive hours. The Audits Director described it as follows:

This is used to account for time not attributable to audits, authorized projects, or leave. For example, staff are allowed two 15-minute breaks by the MOU with Local 21 and they may charge this time as general time. Staff also use this category to charge downtime when they are unable to perform audit work and spend time on activities such as resolving computer problems, reading and responding to e-mail and mail, emergency office procedures (evacuation drills from City Hall, and any other activities not chargeable as audit or project time).

Hours expended on Special Projects were included in the count for productive hours although it is not entirely clear that all hours counted under Special Projects are expended on direct outputs of the division. In fact, it appears that some of the Special Projects functions may be more appropriately performed by the Controller's Office City Projects Division, or other divisions rather than the Audits Division. The following table shows the percentage of expended hours spent on the direct outputs of the Audits Division and also the percentage of available hours expended on direct time. Table 13.1 includes totals for all work of the division and also for the Audit staff only. Audit staff includes the Audits Director, Audit Manager and Associate Auditor positions, but not the Management Assistant position.

As shown in the following table, Audit staff spent 61.71 percent of total time expended on audit work or other special projects in fiscal year 2001-2002 and 61.61 percent of total time expended on audit work or other special projects in fiscal year 2002-2003. The National Association of Local Government Auditors' (NALGA) Benchmarking and Best Practices Survey for Fiscal Year 2002 stated that the average percentage of total time spent on direct outputs for auditing divisions with 13 to 35 auditors was 65 percent. The NALGA survey average has a standard deviation of 12 percent, which means that some respondents to the survey reported spending at least 77 percent of total time on audit work. Like an average, the standard deviation provides a statistical indicator of relative performance within a sample population. Because detailed survey responses were not available to determine exactly how each jurisdiction reported performance, we used the standard deviation to estimate the degree to which high performers reported productivity above the average.

Table 13.1

Audits Division Productive Hours for All Division Staff and Audit Staff Only FY 2001-2002 and 2002-2003

	FY 01-02		FY 02-03
	Division	Audit Staff	Division	Audit Staff
Direct Time	27,321	26,720	22,157	22,039
Indirect Time	11,416	10,425	8,970	7,439
Available Time	38,737	37,145	31,127	29,478
Benefit Time	6,384	6,152	6,450	6,293
Total Time	45,121	43,297	37,577	35,771
Percent of Direct Time to Available Time	70.53%	71.93%	71.18%	74.76%
Percent of Direct Time to Total Time	60.55%	61.71%	58.97%	61.61%

The Audits Division should endeavor to exceed the NALGA standard for productivity and achieve the same level of productivity as the most productive audit divisions in other counties and cities. The Audits Division expended 74.76 percent of Available Time on division direct output, while the average in the NALGA survey was 76 percent with a standard deviation of 14 percent. When measuring Direct Time to Total Time, the Audits Division's current level of productivity is at least 15.4 percent less than the level of productivity of at least one respondent to the NALGA survey for hours expended on direct outputs. That 15.4 percent difference in productivity is equivalent to 2.65 FTEs for FY 2002-2003. The loss of 2.65 FTE in productivity is equivalent to salary costs of $220,895 based on the average salary for audit staff. It appears that one contributing factor to the division's measures of productive is the division's current level of training, as will be discussed in the following section.

Training Hours

Training is critical for government sector auditors. It is important to maintain skills and to remain familiar with current standards in auditing. Training also allows for the expansion of skill sets to enable an organization to take advantage of new technology that becomes available as tools for auditors. Generally Accepted Government Auditing Standards for training require government sector auditors to complete 80 hours of training every two years with at least 20 hours of training occurring each year. As shown in the following, table auditors completed an average of 62 continuing professional education (CPE) hours per year over the two-year period, which is 55 percent more than the minimum standards for government auditors.

Table 13.2

Continuing Professional Education Hours for Audit Staff

	FY 2001-2002	FY 2002-2003	Mean Hours per year
CPE Training hours	1,036	1,079	1,058
Mean Training hours per auditor	60.9	63.5	62
Median Training hours	64	63

In the preceding table, the mean training hours per auditor are total CPE training hours completed by the audit staff of the Audits Division divided by the number of audit staff reporting hours in that year. The training records included instances of auditors receiving 106, 96, and 89 hours in training in a single year. This means that the auditors were completing more than twice the amount of training than the minimum amount necessary to meet standards. The reported hours for training are conservative, since the division also includes hours for informal training and other professional development activities in their time sheets that are not counted as CPE hours. The division defines Informal Training as:

Informal training generally refers to time spent on activities related to training but does not result in CPE credit. It includes such activities as informal self-study (reading reference materials such as the PPC Guide to Audits of Nonprofits, Sawyer's Internal Auditing, and more recently the AICPA, IIA, and NALGA Peer Review Manuals), review of the division's audit manual, professional reading (bulletins and journals from such professional organizations as the IIA, AICPA, AGA, NALGA) and researching, registering or preparing for professional training.

There are also inconsistencies between the number of CPE hours completed by division staff and the number of hours reported as training in time records. In some instances, the number of hours recorded in time reports far exceeded the number of CPE training hours reported. Some of the difference can be explained by staffing changes. However, the difference is quite large for FY 2001-2002. The following table shows the number of CPE training hours, the number of training hours included in time records and the number of informal training and development/training hours included in time records.

Table 13.3

Training Records for Division Staff

	FY 01-02	FY 02-03
Training hours reported in time sheets	1,944.5	1,141.5
CPE training hours current staff	1,103.0	1,087.0
Difference between credited hours and reported hours	841.5	54.5
Additional Informal Training hours and Training Develop/Records hours	72.8	434.5

The inclusion of informal training hours increases the Audits Division reporting of training hours to an average of 87 hours per auditor for FY 2002-2003, which is more than double the minimum requirement of 40 hours per year for government auditors. The Controller's Office Audits Division should implement management controls to more closely monitor training hours in order to maintain appropriate and necessary training hours and prevent over use of training hours. In addition, the division should assess the value of Informal Training and determine if it is an appropriate use of division time. The hours recorded for training and informal training that are above the minimum requirements for training are equivalent to an average of 0.44 FTE for fiscal years 2001-2002 and 2002-2003, or an equivalent salary cost of $36,677 per year based on the average salary for audit staff. If training hours in excess of required training and informal training hours for fiscal years 2001-2002 and 2002-2003 were instead devoted to the direct work of the division, the productivity of audit staff would improve by an average of 2.47 percent.

Direct Charging

Much of the work of the Audits Division is performed for other departments and agencies of the City and County of San Francisco. Of the 65 audit reports issued in FY 2001-2002, the division completed 22 audits for the Airport, two for the Department of Parking and Traffic, seven for the Port and three for the Recreation and Parks Department. Currently, City departments pay for Audits Division concession audits through workorders. For FY 2003-2004, the Controller's Office has budgeted $530,950 for work order recoveries for the Audits Division. The Audits Division should expand the number of audits for which it directly charges City departments in an effort to recover full costs for audits performed.

Peer Review

The California Counties Audit Chiefs Committee sponsors a Peer Review Program in order to provide quality assurance reviews of county audit functions. The general standard is that a Peer Review takes place every three years. The last peer review of the Audits Division was conducted by the County of Santa Barbara Office of the Auditor-Controller in 1998 and issued in 1999. Since that time, the Controller's Office Audits Division has not had a Peer Review completed, although a Peer Review has recently begun.

The Peer Review issued in 1999 stated that the "Division generally conforms with the Standards for the Professional Practice of Internal Auditing."¹ However, the peer auditors did present several findings and recommendations. The peer auditors found that:

Budgetary constraints, their effect on Divisional staff size, and the primacy given to concession audits adversely affect the Division's ability to perform the required scope of audits to be conducted under the City and County' Charter, as well as that suggested by IIA Standards. This issue was also noted in the Quality Assurance Review dated December 31, 1994.

In addition, the peer auditors found that that the Controller's Office had not conducted a risk assessment since FY 1995-1996 and recommended that the Audits Division "perform an annual risk assessment in order to help in setting divisional priorities and work schedules." As noted previously in this report, the Controller's Office reports that it has discontinued performing risk assessments.

Conclusions

The Administrative Code charges the Controller's Office with establishing an auditing function to monitor the economy and efficiency of departments and agencies of the City and County of San Francisco. The Audits Division is not meeting the same level of productivity as other county and city auditors responding to the National Association of Local Government Auditors' Benchmarking and Best Practices Survey and management does not manage and limit training and time reporting as closely as it should. The Division's Time Distribution Reports show that for fiscal year 2002-2003, Audit staff spent on average 61.6 percent of the time reported on timesheets on audit and project work. Training and professional development accounted for 6.1 percent of the time reported, other general tasks and administration accounted for 14.7 percent of the time reported and leave from work accounted for an additional 17.6 percent of time reported.

Further, Audits Division staff received credit for an average of 62 hours of training for fiscal years 2001-2002 and 2002-2003. This is 55 percent more than minimum requirements for training of audit staff. In addition, the Audits Division reported training hours in addition to continuing professional education hours. The inclusion of informal training hours increases the Audits Division reporting of training hours to 87 hours per auditor for FY 2002-2003, which is more than double the general minimum standard of 40 hours per year for government auditors.

The Audits Division does not complete all audits and tasks planned for in the Audit Schedule. If unnecessary and obsolete mandates were eliminated from the Audit Plan, then the division could refocus on more valuable audits including Payroll audits.

At least one respondent to the National Association of Local Government Auditors' Benchmarking and Best Practices Survey reported productivity of 77 percent or more. The Audits Division's current level of productivity is 15.4 percent less than 77 percent. The 15.4 percent difference in productivity is equivalent to 2.65 FTEs for FY 2002-2003. The loss of 2.65 FTE in productivity is equivalent to salary costs of $220,895, based on the average salary for audit staff.

Recommendations

The Controller should:

13.1 Evaluate the risk associated with the current mandates of the Audits Division;

13.2 Prioritize the mandates and request the Board of Supervisors to eliminate any unnecessary mandates;

13.3 Reduce training to required and/or necessary hours;

13.4 More closely monitor training hours in order to maintain appropriate and necessary training hours and prevent over use of training hours;

13.5 Improve productivity; and,

13.6 Should expand the number of audits for which it directly charges City departments to recover full costs for audits performed.

Costs and Benefits

There are no costs associated with increasing productivity or decreasing training hours to a more reasonable level. Increased efficiency in the Audits Division would result in more audits performed of City Departments, Funds, Vendors and Contractors.

¹ Standards for the Professional Practice of Internal Auditing are developed by the Institute of Internal Auditors.