Safer Communities Project - Grant Information

Frequently Asked Questions

• Project Evaluation - What is a program evaluation?

  Program evaluation is a process of gathering information about a program's
  implementation, outcomes, and impact to make it more effective and understand the
  outcomes for its participants, communities, partners, and the organization. A program
  evaluation can answer questions such as:
  1. What impact is the program having on clients/participants, staff, or communities?
  2. Which program activities lead to what impacts?
  3. Who is benefiting the most or the least from participating in the program? Why
  are we seeing these results?

  Back to top

• Project Evaluation - What do we mean by outcome, impact, or result?

  Outcomes, impacts, and results are changes in people, organizations, or systems. At the
  individual level, these changes can be in people’s behavior, knowledge, skills, attitudes,
  values, beliefs, social relations, or perceptions. At the organizational or systems levels,
  these changes could be in values, social conditions, policies, influence levels, the visibility
  of an issue, community norms, partnerships, service practices, or the availability of
  resources.

  Back to top

• Project Evaluation - What is the purpose of the grant and the program evaluation?

  The purpose of the grant is twofold:
  1. Identify the outcomes from community-based violence prevention
  and intervention organizations to expand the Medicaid criteria to include services provided by these organizations as Medicaid reimbursable.
  2. Assess each grantee’s capacity to document and report these outcomes in the
  future to make recommendations for evaluation capacity building and training that will
  ensure organizations can meet new criteria and take advantage of Medicaid
  reimbursement.

  
  Expanding the Medicaid criteria requires documented evidence of the outcomes from community-based programs, including which services create measurable
  results. The Program Evaluation will identify, prioritize, and document the outcomes from
  a community perspective. The evaluation capacity assessment will document the needs
  of each grant-funded organization related to collecting and reporting outcome data
  needed for Medicaid reimbursement in the future and provide a set of recommendations
  to the State for building the capacity of organizations to meet these reporting
  requirements.

  Back to top

• Project Evaluation - Who is doing the evaluation and what are their qualifications?

  The evaluation is being led by Kelly Feltault, PhD, Principal and Executive Director of
  Cultural Crossings Consulting, LLC. She has over 20 years of experience as a program
  evaluator specializing in multi-sited, community evaluations and evaluation capacity
  building.
  Tasha Parker, PhD, LSCSW, MPA is co-leading the evaluation and has been a program
  evaluator for 4 years and a clinician for 13. She has over 17 years of experience in program
  development and implementation, specializing in trauma and resiliency, mental and
  behavioral health treatment, prevention, program development, and systems change.
  Two other team members will assist with the evaluation:
  Sarah Williams Leng, MA, CAPM, has 15 years of experience as a program evaluator and researcher working with mission driven organizations to assess and develop their evaluation capacity and infrastructure.
  Scot Seitz, PhD, is a licensed psychologist and applied researcher who has worked in
  community psychology, clinical psychology, and public health for over 15 years.

  Back to top

• Project Evaluation - How will the grant-funded organizations be involved in the evaluation?

  The project and the evaluation teams believe that the community-based organizations are the experts of their programs. To center their expertise, the project will use a collaborative method called Collaborative Outcomes Reporting (COR) to identify and document the outcomes from the community organizations and their participants. The collaborative process begins with the Evaluation Design:
  1. Applicants will name an Evaluation Liaison in their proposal. The Evaluation Liaison
  should be a staff person interested in learning how to assess program results, has
  experience with the organization’s existing information systems and processes, and
  understands the program that is the focus of the grant. Their role will be to
  collaborate with the evaluation team on several key activities, including developing
  the evaluation and the evaluation capacity assessment, and data collection. This
  should be considered a professional development opportunity for the named staff
  person.
  2. Grantees will review a 1.5 hour recorded Grantee Convening video once the awards
  have been made.
  3. Grantees will participate in a one-day Outcomes Scoping Workshop held in-person
  to collaboratively develop the key evaluation questions and jointly identify the
  outcomes common across their organizations and the services or practices they
  feel produce those outcomes.
  The grantee organizations will be involved in Data Collection:
  • Document and Data Review: Evaluation Liaisons will gather existing program data
  available from the grantee organization, and share it with the evaluation team. For
  example: past reports, program materials and descriptions, number of clients
  served, a roster of client and staff contact information, demographics of past
  clients or participants, past surveys or anecdotal data if available, contact
  information for partners, and other programmatic data that might be available.
  • Communication: Evaluation Liaisons will help to notify clients and staff of the
  evaluation and the importance of participating. The evaluation team will provide
  materials about the evaluation to share.
  • Interviews: The evaluation team will select interviewees from grantee rosters and contact lists using a sampling method. Evaluation Liaisons and grantees will provide suggestions for locations
  and logistics to conduct interviews or focus groups in their region. When needed,
  they will connect the evaluation team to translation or other resources if needed.
  • Evaluation Capacity Assessment: Evaluation Liaisons and grantees will actively take
  part in the capacity assessment to map current policies, procedures and historical
  data related to data collection and reporting on the program. This could be past
  surveys, reports, and requirements from funders. We will also conduct interviews
  with key personnel and front-line staff to determine the expectations, current
  practices around data collection and reporting, challenges and lessons learned, and
  resource needs around program evaluation. Part of this process will be to
  catalogue the systems used to collect, store, and analyze the data. All the
  information gathered will be analyzed and summarized, culminating in a
  Monitoring, Evaluation, and Learning (MEL) Capacity Assessment report. Grantees
  will co-shape the recommendations for future evaluation capacity building to the
  State.
  Grantees will also contribute to interpretation of findings:
  • A one-day Summit Workshop will be held in-person. The workshop will review
  and validate the preliminary evaluation findings from the program document
  review and interviews. The group will prioritize findings, provide feedback, and further explore how their programs contribute to outcomes to inform the survey.
  • Final Report: grantees will receive a copy of the final report.

  Back to top

• Project Evaluation - How will the clients, or participants, of the community-based organizations be involved?

  Program participants will be involved in multiple ways:
  1. Some past and current participants will be invited to take part in interviews to share their experiences on the impact of the community organizations’ programs.
  2. Grantees will have the option to invite a program participant to participate in the
  Outcomes Scoping Workshop and the Summit Workshop (described above).

  Back to top

• Project Evaluation - What kind of information will be collected for the evaluation?

  Different types of data from a variety of sources will be collected, including:
  • From grantees: existing information about the community organization and its
  programs, such as documents describing the programs and services, organizational
  policies and practices, partner lists, results from past surveys or interviews, and
  number of people served over the years.
  • From grantees: participant information, such as participant rosters, demographics,
  services received, frequency of participation, and contact information.
  • From grantees: Organizational capacity data related to measuring and reporting
  outcomes, such as data collection methods, systems, platforms, data
  management systems, capacity to generate reports, copies of past surveys,
  challenges and needs, evaluation and data management policies, staff assignments
  and skills and other resources related to evaluation.
  • From program participants: Interviews will collect original data to
  answer the evaluation questions related to the experiences of program participants
  and staff members and their engagement with the grantee’s programs.

  Back to top

• Project Evaluation - What if we don’t have this data or have limited evaluation capacity?

  There are no evaluation capacity requirements to be a grantee. At a minimum,
  organizations should have a client or participant roster with contact information
  stretching back at least one year. Grantees should be ready and willing to actively
  participate in the evaluation as a collaborative partner as described above. 

  Back to top

• Project Evaluation - How long will this take to complete?

  We anticipate the final report being completed by September 2025.

  Back to top

• Project Evaluation - How will the results be used and by whom?

  The outcomes identified in this evaluation will be used by the community-based organizations
  and the state of Pennsylvania.
  1. Pennsylvania will aim to use the outcomes to expand the Medicaid criteria to
  include services provided by community-based organizations as Medicaid
  reimbursable. The state can use the evaluation capacity assessment to develop
  training to build organizational capacity to collect and report the data, and to
  develop reporting systems.
  2. The grant-funded community-based organizations can use the evaluation results to tell
  their story to potential funders, their clients, and partners. They can use the
  capacity assessment to adjust strategic plans, prepare for the new criteria, and
  prioritize evaluation infrastructure.

  Back to top

• Project Evaluation - Who will own the data once it is collected?

  People participating in interviews will receive a copy of their transcript and will own
  that data. Ownership of existing data provided by the grantees will
  remain with grantees.

  Back to top

• Project Evaluation - How will the information collected for the evaluation remain confidential?

  Interview and survey data will be kept in an encrypted, password protected cloud storage system. Client and participant rosters shared with the evaluation team by grantees will be kept in the same protected cloud storage system. Only the members of the evaluation team will have access to the data. The names of clients, participants, and staff taking part in interviews will be replaced by unique identity numbers generated by the evaluation team. Personal identifiers and unique identity codes will not be published or presented in reports. Individual data will be combined with other responses for the analysis. Grantees, the State, and Millersville University will not have access to the lists of interviewees. Participant rosters will be destroyed by the evaluation team at the completion of the project.

  Back to top

• Project Evaluation - Will the evaluators be using a trauma-informed approach when working with grantees and participants?

  Yes, the project and the evaluation teams recognize that many program participants, staff, and others taking part in the evaluation will have experienced trauma related to gun violence. Further,
  the teams acknowledge that most people have experienced some form of trauma in their lives. As a result, the team of evaluators will integrate trauma-informed evaluation practices by consistently considering the impact of current and past experiences on individuals, fostering a safe and supportive evaluation environment, and tailoring assessment methods to be sensitive to the potential effects of trauma. This approach ensures a comprehensive understanding of participants' experiences and well-being throughout the evaluation process.

  Back to top

• Monitoring, Evaluation & Learning (MEL) Capacity Assessment - What is a Monitoring, Evaluation & Learning (MEL) Framework?

  MEL framework is a roadmap that helps organizations, projects and programs keep
  track of what they are doing (monitoring), how well they are doing it (evaluation), and what
  they are learning along the way (learning). This roadmap allows an organization to keep on the
  determined path, to make improvements and adjustments, and learn from the collective
  experience. A MEL framework is a valuable tool to keep on course and continuously
  improve the work.

  Back to top

• Monitoring, Evaluation & Learning (MEL) Capacity Assessment - What is a MEL Capacity Assessment?

  MEL Capacity Assessment is like taking a self-check to see how well an organization
  or team is equipped to monitor, evaluate, and learn from your activities. This specific
  assessment will focus on the organizational capacity and needs related to collecting and
  reporting on program data as required for Medicaid reimbursement. The project will include a data
  mapping activity during which each organizational representative will map out the
  processes and systems used to currently collect participant and outcome data.

  Back to top

• Monitoring, Evaluation & Learning (MEL) Capacity Assessment - Is this part of the grant we received?

  Yes, the MEL Capacity Assessment is part of the grant and participation is required.

  Back to top

• Monitoring, Evaluation & Learning (MEL) Capacity Assessment - What happens during a MEL Capacity Assessment?

  Each organization will select a staff member to be the Evaluation Liaison for that
  organization. The evaluation team will collaborate with that person to collect and review
  existing policies, procedures and historical data related to data collection and reporting on the program. This could be past surveys, reports, requirements from funders, and
  organizational policies. We will also conduct interviews with key personnel, including
  program managers, IT staff, and front-line staff to determine the expectations and current
  practices around data collection and reporting. Part of this process will be to “map”
  current processes for collecting program data and catalogue the systems used to collect,
  store, and analyze the data. All the information gathered will be analyzed and
  summarized, culminating in a MEL Capacity Assessment report. 

  Back to top

• Monitoring, Evaluation & Learning (MEL) Capacity Assessment - How will the data findings be used?

  Based on the analysis, recommendations will be made to OMHSAS for building the MEL
  capacity of community organizations to improve their readiness to collect and report
  outcomes for Medicaid reimbursement. The recommendations could include providing
  training to organizations or additional resources to support MEL capacity building. The
  Assessment also serves as a benchmark for the state and grantees on progress toward
  MEL capacity building. Individual organizations can use the findings for strategic planning.
  The results of the MEL Capacity Assessment will not be used to penalize organizations in
  any way. The goal is to learn about organizations’ current capacity to identify the type of
  support that organizations may need in the future.

  Back to top