Current Data Collection Processes and Systems in the Department of Child Safety

Agency Program Evaluation Capacity Analysis

Current Data Collection Processes and Systems in DCS

The Department of Child Safety (DCS) gathers a variety of data related to administrative, financial, and programmatic functioning. Program specific information for both process and summative evaluation is collected from each of the six bureaus to examine implementation and outcomes. Arizona Legislature mandates reports to be made monthly, quarterly, semi-annually, and annually depending on type and purpose. DCS has made these reports available via their website in an effort to promote transparency, accountability, and continuous improvement.

Administrative

 Arizona Legislature acts as the preeminent body for evaluation of management performance at DCS. Based on communication with multiple employees, Director Gregory McKay does not undergo a formal evaluation, nor do the Chief Officers and Deputy Directors. However, Program Coordinators and Program Managers go through an annual evaluation to ensure performance efforts are in accordance with DCS’s current Strategic Plan, and that administrative efforts ultimately contribute to the DCS mission.

 One objective outlined in the 2017-2021 Strategic Plan was to implement standardized procedures for all employees, including supervisors, case specialists, and support staff within their respective programs to ensure all employees are conducting business in a consistent manner (Department of Child Safety, 2017a). Formal evaluation of employee performance is conducted annually based on these standardized measures. Results are synthesized into various reports completed throughout the year. For example, the Monthly Operational Outcomes Report summarizes employees’ performance in areas related to removals, reunification, and child contact rates (DCS, 2018a). Informal evaluations of staff performance, including supervisors, occurs weekly in each unit’s “Huddle Board” meeting where program-specific visual data is displayed to help staff self-identify areas of accomplishments and goals for the upcoming week.

Financial

 DCS has strict and comprehensive guidelines for financial operations. These guidelines are mandated by Arizona Legislature and overseen by internal and external auditing committees. Internally, the Office of Strategic Planning and Budgeting (OSBP) collaborates with the external Joint Legislative Budget Committee (JLBC) to ensure financial liability for the Department.

 Funds are determined and allocated based on a yearly Budget Submittal Request by Director McKay. Appropriated funds include, but are not limited to, temporary assistance to needy families, child care development fund, DCS expenditure authority, and child abuse prevention fund. Expenditure funds include those covered by federal grants like Title IV-B, Title IV-E, Title XIX and Social Services Block Grant. Appropriated and expenditure funds are reviewed semi-annually by the OSBP and JLBC and submitted to the legislature. This Semi-Annual Financial and Program Accountability Report (2018b), identifies financial and program accountability measures for the Department (refer to Appendix C for data specific to financial liability). While DCS has made these reports and others like it available to the public, they are lengthy and difficult to understand.

Programmatic

 Under Director of Support Services, Mike Faust, DCS has adapted a lean management approach that supports continuous improvement. In accordance with this approach, DCS has implemented several evaluation procedures for its multiple programs. For instance, the Arizona Families FIRST Program tracks the number of referrals made, demographics, substance use, reduction of child abuse and neglect, and permanency status. This data is self-reported by clients, collected by staff, and reported annually (refer to Appendix A for specific data). This data is then analyzed by Program Managers for each activity’s effectiveness in reaching program objectives. Data from other bureaus is collected and analyzed in much the same manner and synthesized into the Semi-Annual Child Welfare Report.

Current Data Collection Processes and Systems in AYAP

 The Arizona Young Adult Program (AYAP) uses the Positive Youth Development Model which postulates all children have strengths and future potential and is working towards a theory of change foundation where goals and objectives are outlined with various assumptions about change (Damon, 2004; Organizational Research Services, 2004). This model and theory provide the framework for gathering and evaluating staff and client performance. Federal law authorizes the Children’s Bureau to review AYAP under Title IV-B, IV-E, and Fostering Connections to Success and Increasing Adoptions Act (FCSIA) the primary funding sources for AYAP. This review process is facilitated by the Child and Family Services Review (CFSR) and National Youth in Transition Database (NYTD).

Staff

 According to the latest CFSR Annual Progress and Services Report (2017b), AYAP has made significant progress in staff performance in the targeted areas of transitioning youth to self-sufficiency, educational and employment assistance, preparing for post-secondary education, and providing support through mentorship. Specific data on staff performance includes: number of cases per case specialist, ensuring skills training begins by age 16, number of referrals made for educational assessments, number of referrals made for college readiness classes, number of recipients of the Education and Training Voucher Program (ETV), and number of natural mentors for youth. AYAP Program Coordinator, Megan Conrad reports an emphasis on maintaining monthly child contacts, utilizing an “age of majority” team decision meeting (TDM) and reporting the number of referrals being made. This information is self-reported by AYAP staff and submitted to the Program Coordinator for publication. The number of cases per case specialist, number of youths in congregate care, child contacts, number of youths in ETV, and number of “age of majority” TDMs are used in each unit’s “Huddle Board” meeting for staff to have a visual representation of performance. In theory, all AYAP staff have access to program reports via the website. However, according to an AYAP case specialist, “I have not looked at one AYAP report since I have been employed at DCS…I do like the Huddle Boards, they make sense to me and help me focus on immediate goals” (AYAP case specialist, personal communication, October 5, 2015).

Client

AYAP eligible youth collaborate with Case Specialists, therapists, Independent Living Specialists, Court Appointed Special Advocates, and Community Advisors on an Individualized Service Plan (ISP) to cover the array of services offered through AYAP. The ISP outlines what services are being delivered, are they having the desired results, and are they sufficient. Client data reflected in the ISP for evaluation includes: number of youths enrolled and their ages at enrollment and discharge, number and type of services being utilized, educational status, number and type of placement, and re-entry rates. Conrad maintains an emphasis on tracking the educational status of youth in care and type of placement as it pertains to program objectives of improving educational outcomes and decreasing the number of youths in congregate care.

The NYTD collects data on client outcomes to assist in program evaluation. Cohorts of clients are surveyed at age 17, 19 and 21 on employment, receipt of public assistance, educational attainment, substance use, criminal involvement, marital and familial status, and family of origin connections (Refer to Appendix B for specific data). After the client has been surveyed by AYAP staff, data is entered electronically via the “NYTD federal reporting window.” This information is published in an annual report and accessed via the website. AYAP’s Program Coordinator and Program Manager use this information to assess whether AYAP is meeting program objectives, identify areas for improvement, and develop key initiatives and strategies accordingly.

Externally Required Monitoring Reports and Processes

Agency

 Pursuant to Arizona Revised Statue 8-458, DCS is required to report to external oversight and auditing committees on programs and services (Arizona Revised Statute 8-458, 2015). This form of monitoring and evaluation helps to identify strengths and weaknesses of services and offer recommendations for improvement. DCS has multiple external monitoring sources in place and federally required to comply to receive federal appropriated funds and retain eligibility for Title IV-B, Title IV-E, Title XIX, and Social Services Block Grants. The number of reports required is beyond the scope of this paper. However, one example is the reporting process for the OAG report which is conducted every five years.  The procedure includes planning, gathering information, drafting and finalizing a report. The report is both electronically submitted on a database and hard copies are distributed to DCS Audit Management System. DCS is given six months to respond to recommendations given in the report. The procedure ends with an audit analysis where trends and risks will be identified and presented to the Director.

Program

 Public law 106-169 initiated the John H. Chafee Foster Care Independence Program (CFCIP) of the Social Security Act. The CFCIP is an additional funding source for AYAP that includes the ETV Program. Under this same law, the Administration for Children and Families (ACF) was commissioned to develop a data collection system to assess AYAP’s performance. If AYAP fails to comply with reporting requirements, ACF has authority to deduct between one and five percent of DCS’s annual funding allotment. NYTD is ACFs primary data collection service for evaluation of AYAP outcome objectives (Children’s Bureau, 2012). NYTD publishes an annual report for each state based on the information collected.

As mentioned previously, the CFSR is an additional external monitoring mechanism for multiple programs and functions of DCS including AYAP. The CFSR compiles their review and publishes an Annual Progress and Services Report. The CFSR collects information in the seven service areas of AYAP; as well as, its involvement in the public and private sector, collaboration with community partners, and ETV program. Each section of the review addresses performance measures and offers recommendations. Conrad asserts as the Program Coordinator, that the information provided by NYTD and CFSR are integral to the ongoing functioning of AYAP. She claims the program has a history of struggling with data collection. This is in part due to the inadequacy of CHILDS, the current database system for DCS. She hopes as DCS moves towards integrating the new GUARDIAN database, AYAP and other DCS programs will focus on quality data that is more easily accessible and transferrable.

Internally Required Monitoring Reports and Processes

Agency

 DCS has several internal oversight committees to ensure quality assurance and continuous quality improvement. The DCS Compliance Audit (CA) evaluates operational areas to determine compliance with applicable laws, rules, policies and procedures to identify process improvements. The procedure of the bi-annual CA review entails planning, execution, finalizing a report including an action plan, monitoring the program’s compliance with the action plan quarterly, and conducting an audit analysis. The action plan developed by the CA helps cultivate “compliance consciousness” among DCS staff where compliance is viewed as integrated rather than a separate issue (DCS Policy and Procedure Manual (PPM), 2018).

Program

 In February of 2018, AYAP developed an internal auditing system based on the latest NYTD review. The AYAP Quality Assurance (QA) team developed a review tool specific to 14 to 21-year-olds to capture data on service utilization and outcomes. Conrad reports the AYAP QA has begun collaboration with Audit Management Services to audit an initial 65 files with plans to develop a system that can audit on an on-going basis. This formal internal reporting system will complement the informal “Huddle Board” meetings by adding an additional measure for program evaluation.

Strengths and Weaknesses of Current Evaluation Plan

Strengths

 In four short years as a stand-alone department, DCS has made significant operational improvements as evidenced by a reduction in the number of back-logged cases, decreasing the number of kids in out of home care, and standardizing employee procedures across all bureaus. In response to the most recent OAG Audit (2015), DCS has improved their content, frequency, and accessibility of reports to key stakeholders and the public alike by publishing relevant reports on the DCS website. The Department is working on applying a lean management system and theory of change model for all of its programs, including AYAP. This helps to develop baseline measures and target areas identified for improvement. By extension, the “Huddle Board” meetings appear to be a valuable addition to this new system by providing staff with immediate access to relevant data.

Weaknesses

 It is critical to incorporate continuous evaluation of both the Department and its programs to identify areas of weakness and strategies for improvement. In spite of DCS efforts to promote accountability and transparency, the average citizen is likely to have difficulty navigating, locating, and understanding reports published on the website. This is particularly relevant to those who may be involved in the child welfare system. In addition, these reports do not appear to be readily accessed by DCS employees. While “Huddle Boards” present unit-specific data, they do not give a picture of how the particular program-as-a-whole is reaching objectives. Megan Conrad pinpointed AYAP’s struggle with data collection; a problem exacerbated by an antiquated database system, lack of funding, and considerable change in structure within the Department over the past four years. It is likely this problem is Department-wide. The problem raises the issue of reliability of the current data.

 These weaknesses point towards gaps missing in DCS processes for collection and dissemination of data. In an effort to truly maintain transparency and accountability, the public would benefit from versions of the same report: one written for the Legislature and one for the public. Second, efforts to incorporate program-wide data into “Huddle Board” meetings would educate program staff on program-wide efforts. Lastly, increased efforts for training staff in data collection partnered with an updated database system would improve reliability of data from stage one, thereby improving the way data is reported and data analysis. Taking these measures will ultimately ensure safety, strengthen families and achieve permanency in Arizona.

References

Arizona Revised Statute § 8-458. (2015). Title 8-Chapter 4: 8-458. Inspections bureau;

Monitoring; quality Assurance Process. Retrieved on October 12, 2018 from

https://www.azleg.gov/viewdocument/?docName=https://www.azleg.gov/ars/8/00458.h

m

Chapin Hall. (2015). Arizona Department of Child Safety Independent Review. Chicago, IL: University of Chicago.

Children’s Bureau. (2012). About NYTD. Retrieved on October 12, 2018 form

 https://www.acf.hhs.gov/cb/resource/about-NYTD.

Damon, W. (2004). What is positive youth development? The ANNALS of the American

Academy of Political and Social Science, 59(1), 13-24

Department of Child Safety (2017a). 2017-2021 Five Year Strategic Plan. Retrieved on October

12, 2018 from file:///Users/carenabloom/Downloads/FY-2017-DCS_5_YrStratPlan.pdf

Department of Child Safety (2017b). Child and Family Services: Annual Progress and Services

Review for FFY 2018. Retrieved on October 12, 2018 from file:///Users/carenabloom/Downloads/FFY%202018%20Arizona%20APSR%20Update%202017_09_05%20Final%20(2).pdf

Department of Child Safety (2018a). Monthly Operational Outcomes Report. Retrieved on

October 12, 2018 from https://dcs.az.gov/reports-data/dcs-reports

Department of Child Safety (2018b). Semi-Annual Financial and Program Accountability

Report. Retrieved on October 12, 2018 from file:///Users/carenabloom/Downloads/DCS%20Financial%20and%20PRogram%20Accountability%20Report_August%202018%20(3).pdf

Department of Child Safety Policy and Procedure Manual. (n.d.). Administrative Policy 07-11:

Compliance Audit Authority. Retrieved on October 12, 2018 from https://extranet.azdcs.gov/DCSPolicy/#administrative/policies/DCS%2007-11%20Compliance%20Audit%20Authority%20Policy%202-8-2016.pdf%3FTocPath%3DAdministrative%2520Policy%2520%7C07%2520Audit%2520Management%2520Services%7C_____5

Organizational Research Services. (2004). Theory of Change: A Practical Tool for Action,

Results and Learning. Retrieved on October 12, 2018 from

http://www.aecf.org/m/resource doc/aecf-theoryofchange-2004.pdf

Appendix A

Arizona Families FIRST Findings from Annual Evaluation Report 2017

Appendix A Continued

Appendix B

NYTD Data on AYAP

Appendix B Continued

Appendix C

OSB & JLBC Semi-Annual Financial and Program Accountability Report