Safer Communities Project - Grant Information

Frequently Asked Questions

• Project Evaluation - What is a program evaluation?

  Program evaluation is a process of gathering information about a program's
  implementation, outcomes, and impact to make it more effective and understand the
  outcomes for its participants, communities, partners, and the organization. A program
  evaluation can answer questions such as:
  1. What impact is the program having on clients/participants, staff, or communities?
  2. Which program activities lead to what impacts?
  3. Who is benefiting the most or the least from participating in the program? Why
  are we seeing these results?

  Back to top

• Project Evaluation - What do we mean by outcome, impact, or result?

  Outcomes, impacts, and results are changes in people, organizations, or systems. At the
  individual level, these changes can be in people’s behavior, knowledge, skills, attitudes,
  values, beliefs, social relations, or perceptions. At the organizational or systems levels,
  these changes could be in values, social conditions, policies, influence levels, the visibility
  of an issue, community norms, partnerships, service practices, or the availability of
  resources.

  Back to top

• Project Evaluation - What is the purpose of the grant and the program evaluation?

  The purpose of the grant is twofold:
  1. Identify the outcomes from non-clinical community-based violence prevention
  health organizations to expand the Medicaid criteria to include services provided by these
  organizations as Medicaid reimbursable.
  2. Assess each grantee’s capacity to document and report these outcomes in the
  future to make recommendations for evaluation capacity building and training that will
  ensure organizations can meet new criteria and take advantage of Medicaid
  reimbursement.
  Expanding the Medicaid criteria requires documented evidence of the outcomes from
  non-clinical community-based programs, including which services create measurable
  results. The Program Evaluation will identify, prioritize, and document the outcomes from
  a community perspective. The evaluation capacity assessment will document the needs
  of each grant-funded organization related to collecting and reporting outcome data
  needed for Medicaid reimbursement in the future and provide a set of recommendations
  to the State for building the capacity of organizations to meet these reporting
  requirements.

  Back to top

• Project Evaluation - Who is doing the evaluation and what are their qualifications?

  The evaluation is being led by Kelly Feltault, PhD, Principal and Executive Director of
  Cultural Crossings Consulting, LLC. She has over 20 years of experience as a program
  evaluator specializing in multi-sited, community evaluations and evaluation capacity
  building.
  Tasha Parker, PhD, LSCSW, MPA is co-leading the evaluation and has been a program
  evaluator for 4 years and a clinician for 13. She has over 17 years of experience in program
  development and implementation, specializing in trauma and resiliency, mental and
  behavioral health treatment, prevention, program development, and systems change.
  Two other team members will assist with the evaluation:
  Sarah Williams Leng, MA, CAPM, has 15 years of experience as a program evaluator and researcher working with mission driven organizations to assess and develop their evaluation capacity and infrastructure.
  Scot Seitz, PhD, is a licensed psychologist and applied researcher who has worked in
  community psychology, clinical psychology, and public health for over 15 years.

  Back to top

• Project Evaluation - How will the grant-funded community organizations be involved in the evaluation?

  The evaluation team believes that the organizations are the experts of their programs.
  To center their expertise, we will use a collaborative method called Collaborative
  Outcomes Reporting (COR) to identify and document the outcomes from the community
  organizations and their participants. The collaborative process begins with the Evaluation
  Design:
  1. Applicants will name an Evaluation Liaison in their proposal. The Evaluation Liaison
  should be a staff person interested in learning how to assess program results, has
  experience with the organization’s existing information systems and processes, and
  understands the program that is the focus of the grant. Their role will be to
  collaborate with the evaluation team on several key activities, including developing
  the evaluation and the evaluation capacity assessment, and data collection. This
  should be considered a professional development opportunity for the named staff
  person. Include a brief resume of the person selected in the grant application.
  2. Grantees will participate in a two-hour virtual Grantee Convening once the awards
  are made. The Convening will introduce and begin the evaluation process.
  3. Grantees will participate in a one-day Outcomes Scoping Workshop held in-person
  to collaboratively develop the key evaluation questions and jointly identify the
  outcomes common across their organizations and the services or practices they
  feel produce those outcomes.
  The grantee organizations will be involved in Data Collection:
  • Document and Data Review: Evaluation Liaisons will gather existing program data
  available from the grantee organization, and share it with the evaluation team. For
  example: past reports, program materials and descriptions, number of clients
  served, a roster of client and staff contact information, demographics of past
  clients or participants, past surveys or anecdotal data if available, contact
  information for partners, and other programmatic data that might be available.
  • Communication: Evaluation Liaisons will help to notify clients and staff of the
  evaluation and the importance of participating. The evaluation team will provide
  materials about the evaluation to share.
  • Interviews and Focus Groups: The evaluation team will select interviewees and
  focus group participants from grantee rosters and contact lists using a sampling
  method. Evaluation Liaisons and grantees will provide suggestions for locations
  and logistics to conduct interviews or focus groups in their region. When needed,
  they will connect the evaluation team to translation or other resources if needed.
  • Survey: Evaluation Liaisons will review and help disseminate a survey to all current
  and past participants once outcomes are refined through interviews and focus
  groups. The survey will be distributed online and in-person (pen-and-paper or
  device at point of service).
  • Evaluation Capacity Assessment: Evaluation Liaisons and grantees will actively take
  part in the capacity assessment to map current policies, procedures and historical
  data related to data collection and reporting on the program. This could be past
  surveys, reports, and requirements from funders. We will also conduct interviews
  with key personnel and front-line staff to determine the expectations, current
  practices around data collection and reporting, challenges and lessons learned, and
  resource needs around program evaluation. Part of this process will be to
  catalogue the systems used to collect, store, and analyze the data. All the
  information gathered will be analyzed and summarized, culminating in a
  Monitoring, Evaluation, and Learning (MEL) Capacity Assessment report. Grantees
  will co-shape the recommendations for future evaluation capacity building to the
  State.
  Grantees will also contribute to interpretation of findings:
  • Summit Workshop 1 will be held in-person for one day. The workshop will review
  and validate the preliminary evaluation findings from the program document
  review, interviews, and focus groups. The group will prioritize findings, provide
  feedback, and further explore how their programs contribute to outcomes to
  inform the survey.
  • Summit Workshop 2 will be held virtually for two hours. This workshop will present
  the findings and recommendations from the evaluation and the capacity
  assessment. Grantees will provide feedback included as part of the report.
  • Final Report: grantees will receive a copy of the final report.

  Back to top

• Project Evaluation - How will the clients, or participants, of the community organizations be involved?

  Program participants will be involved in three ways:
  1. Some past and current participants will be invited to take part in interviews or
  focus groups to share their experiences on the impact of the community
  organizations’ programs.
  2. All past and current participants will be invited to take a survey (either paper or
  online) to measure the outcomes identified in the interviews and focus groups.
  3. Grantees will have the option to invite a program participant to participate in the
  Outcomes Scoping Workshop and Summit Workshops (described above).

  Back to top

• Project Evaluation - What kind of information will be collected for the evaluation?

  We will collect different types of data from a variety of sources, including:
  • From grantees: existing information about the community organization and its
  programs, such as documents describing the programs and services, organizational
  policies and practices, partner lists, results from past surveys or interviews, and
  number of people served over the years.
  • From grantees: participant information, such as participant rosters, demographics,
  services received, frequency of participation, and contact information.
  • From grantees: Organizational capacity data related to measuring and reporting
  outcomes, such as data collection methods, systems, platforms, data
  management systems, capacity to generate reports, copies of past surveys,
  challenges and needs, evaluation and data management policies, staff assignments
  and skills and other resources related to evaluation.
  • From program participants: Interviews and focus groups will collect original data to
  answer the evaluation questions related to the experiences of program participants
  and staff members and their engagement with the grantee’s programs

  Back to top

• Project Evaluation - What if we don’t have this data or have limited evaluation capacity?

  There are no evaluation capacity requirements to be a grantee. At a minimum,
  organizations should have a client or participant roster with contact information
  stretching back at least one year. Grantees should be ready and willing to actively
  participate in the evaluation as a collaborative partner as described above. The evaluation
  team recognizes the operational challenges non-profits face—from staffing to funding
  and everything in between. We also recognize that although more funders require
  evaluation data, they do not fund data collection, staff training, or the software and
  systems to manage the data. In our experience, most nonprofits do not have the capacity
  to meet evaluation requirements. That is why we included the capacity assessment as
  part of this grant.

  Back to top

• Project Evaluation - How long will this take to complete?

  We anticipate collaboration with the grant-funded organizations for the project evaluation to begin in June 2024 and the final report being completed by January 2026.

  Back to top

• Project Evaluation - How will the results be used and by who?

  The outcomes identified in this evaluation will be used by the community organizations
  and the state of Pennsylvania.
  1. Pennsylvania will aim to use the outcomes to expand the Medicaid criteria to
  include services provided by community-based organizations as Medicaid
  reimbursable. The state can use the evaluation capacity assessment to develop
  training to build organizational capacity to collect and report the data, and to
  develop reporting systems.
  2. The grant-funded community organizations can use the evaluation results to tell
  their story to potential funders, their clients, and partners. They can use the
  capacity assessment to adjust strategic plans, prepare for the new criteria, and
  prioritize evaluation infrastructure.

  Back to top

• Project Evaluation - Who will own the data once it is collected?

  People participating in interviews will receive a copy of their transcript and will own
  that data. People participating in focus groups will receive a summary of the focus group
  discussion, but to protect the confidentiality of other participants, we cannot provide full
  transcripts of the focus groups. Ownership of existing data provided by the grantees will
  remain with grantees.

  Back to top

• Project Evaluation - How will the information collected for the evaluation remain confidential?

  Interview, survey, and focus group data will be kept in an encrypted, password
  protected cloud storage system. Client and participant rosters shared with the evaluation
  team by grantees will be kept in the same protected cloud storage system. Only the 4
  members of the evaluation team will have access to the data. The names of clients,
  participants, and staff taking part in interviews or focus groups will be replaced by unique
  identity numbers generated by the evaluation team. Personal identifiers and unique
  identity codes will not be published or presented in reports. Individual data will be
  combined with other responses for the analysis. Grantees, the State, and Millersville
  University will not have access to the lists of interviewees or focus group participants.
  Participant rosters will be destroyed by the evaluation team at the completion of the
  project.

  Back to top

• Project Evaluation - Will the evaluators be using a trauma-informed approach when working with grantees and participants?

  Yes. The evaluation team recognizes that many program participants, staff, and others
  taking part in the evaluation will have experienced trauma related to gun violence. Further,
  the team acknowledges that most of us have experienced some form of trauma in our
  lives. As a result, the team of evaluators will integrate trauma-informed evaluation
  practices by consistently considering the impact of current and past experiences on
  individuals, fostering a safe and supportive evaluation environment, and tailoring
  assessment methods to be sensitive to the potential effects of trauma. This approach
  ensures a comprehensive understanding of participants' experiences and well-being
  throughout the evaluation process

  Back to top

• Monitoring, Evaluation & Learning (MEL) Capacity Assessment - What is a Monitoring, Evaluation & Learning (MEL) Framework?

  MEL framework is a roadmap that helps organizations, projects and programs keep
  track of what they are doing (monitoring), how well they are doing it (evaluation), and what
  they are learning along the way (learning). This roadmap allows you to keep on the
  determined path, to make improvements and adjustments, and learn from the collective
  experience. A MEL framework is a valuable tool to keep on course and continuously
  improve your work.

  Back to top

• Monitoring, Evaluation & Learning (MEL) Capacity Assessment - What is a MEL Capacity Assessment?

  MEL Capacity Assessment is like taking a self-check to see how well your organization
  or team is equipped to monitor, evaluate, and learn from your activities. This specific
  assessment will focus on the organizational capacity and needs related to collecting and
  reporting on program data as required for Medicaid reimbursement. We will have a data
  mapping activity during which each organizational representative will map out the
  processes and systems used to currently collect participant and outcome data.

  Back to top

• Monitoring, Evaluation & Learning (MEL) Capacity Assessment - Is this part of the grant we received?

  Yes, the MEL Capacity Assessment is part of the grant and participation is required.

  Back to top

• Monitoring, Evaluation & Learning (MEL) Capacity Assessment - What happens during a MEL Capacity Assessment?

  Each organization will select a staff member to be the Data Steward for that
  organization. The evaluation team will collaborate with that person to collect and review
  existing policies, procedures and historical data related to data collection and reporting on the program. This could be past surveys, reports, requirements from funders, and
  organizational policies. We will also conduct interviews with key personnel, including
  program managers, IT staff and front-line staff to determine the expectations and current
  practices around data collection and reporting. Part of this process will be to “map”
  current processes for collecting program data and catalogue the systems used to collect,
  store, and analyze the data. All the information gathered will be analyzed and
  summarized, culminating in a MEL Capacity Assessment report. 

  Back to top

• Monitoring, Evaluation & Learning (MEL) Capacity Assessment - How will the data findings be used?

  Based on the analysis, recommendations will be made to the State for building the MEL
  capacity of community organizations to improve their readiness to collect and report
  outcomes for Medicaid reimbursement. The recommendations could include providing
  training to organizations or additional resources to support MEL capacity building. The
  Assessment also serves as a benchmark for the state and grantees on progress toward
  MEL capacity building. Individual organizations can use the findings for strategic planning.
  The results of the MEL Capacity Assessment will not be used to penalize organizations in
  any way. The goal is to learn about organizations’ current capacity to identify the type of
  support that organizations may need in the future.

  Back to top

• Application Content and Process - What organizations are eligible to apply for the Safer Communities Project RFA?

  To be eligible, Applicants must meet the following criteria:
  

  Be a 501c3 organization in good standing in the State of Pennsylvania.

  Be a current provider of nonclinical, community-based peer-to-peer individualized and/or small group mental health support services for survivors of gun violence that will be implemented continuously throughout the grant period.

  Be able to allocate staff for project management, fiscal management, data stewardship, and program implementation.

  Be ready and willing to actively participate in the evaluation as a collaborative partner and to submit timely, accurate programmatic and financial reports.

  Be able to provide contact information for current and past staff, past clients, and partners of the program dating back at least one year.

  Please note, programs implemented by staff who are already able to bill for Medicaid are not eligible.

  Back to top

• Application Content and Process - Is this the first time Millersville University is administering this grant?

  This is the first grant that Millersville University is administering on behalf of the Pennsylvania Department of Human Services and for community gun violence prevention and intervention programming. The University has administered federal and state grants in other areas.

  Back to top

• Application Content and Process - Is a nonprofit hospital with a peer-to-peer mental health support services program eligible for this funding?

  DHS specified the purpose of this grant is to evaluate peer-to-peer mental health support service programs implemented by community-based organizations. Hospitals are not eligible for this grant.

  Back to top

• Application Content and Process - Can an organization apply and subcontract the mental health support services?

  To be eligible an applicant must be a current provider of nonclinical, community-based peer-to-peer individual and/or small group mental health support services for survivors of gun violence that will be implemented continuously throughout the grant period. These services cannot be performed by a subcontractor. However, a potential grantee may partner with a subcontractor in this project. If an applicant proposes to work with a subcontractor, the subcontractor must meet the eligible criteria prescribed in the RFA including their program must not be eligible for Medicaid reimbursement. The subcontractor’s roles and expenses (if applicable) must be clearly described in the application. Documentation of the subcontractor’s 501c3 status must also be included in the application.

  Back to top

• Application Content and Process - What is meant by the term "nonclinical" in the RFA?

  "Nonclinical" means programs that are implemented by people with lived experience of gun violence supporting people who recently experienced gun violence. This project will evaluate programs implemented by peers. The funding is not available for programs implemented by licensed clinicians (e.g., licensed clinical social workers, counselors, psychologists, etc.). Please keep in mind one of the primary purposes of this project is to provide the Pennsylvania Department of Human Services with recommendations for expanding Medicaid billable services to include peer-to-peer programs.   

  Back to top

• Application Content and Process - Our organization implements a program (e.g., exercise programs, leadership development programs, etc.) that has a mental health benefit for our clients who have been impacted by gun violence. Is our program eligible?

  To maintain equity to access of information throughout the RFA process, we are not able to provide responses about an individual organization’s programs. Please note, to be eligible an applicant must be a current provider of nonclinical, community-based peer-to-peer individual and/or small group mental health support services for survivors of gun violence that will be implemented continuously throughout the grant period. Programs that will be funded through this initiative have peer-to-peer workers who have experienced gun violence meeting with other people who have recently experienced gun violence. The peer-to-peer outreach worker provides services to individuals and/or small groups, services such as screening, crisis support, conflict mediation, and helping clients get connected with licensed mental health professionals. Programs must have a mental health focus. Programs like an exercise program or a leadership development type of program may have a mental health benefit but are not specifically mental health-focused and thus are not eligible for this funding.

  Back to top

• Application Content and Process - Can the clients served by an organization include adolescents?

  Organizations serving adolescents (as well adult) clients are eligible for this grant. For evaluation purposes, the evaluators would talk with the adolescents’ parents. The evaluation protocol would include parental assent if there were a need for any sort of involvement with adolescents directly.

  Back to top

• Application Content and Process - When the evaluation is completed, will an individual grantee be able to use the finished evaluation results to show the efficacy of their own programs?

  The evaluation will not be evaluating each organization’s program individually. The evaluation design will be evaluating the outcomes across programs. Part of the capacity assessment will include equipping organizations to collect data and document the impact of their programs to then use in applying for future funding.

  Back to top

• Application Content and Process - Will a grantee be able to use the completed data to improve credibility when applying for future funding?

  An organization will be able to speak to the outcomes that are identified and validated through this project, have a better understanding of what the outcomes are, and going forward be better equipped to measure the outcomes. An aspect of this project is to provide the State with recommendations about what the State can do to help community-based organizations build their capacity to document outcomes and pursue new funding because at the conclusion of this project the community-based organizations will be better equipped to conduct the evaluation funders are requiring. Demonstrating outcomes for funders can be a challenge and we are hopeful that this project will equip community-based organizations to be able to access future funding.  

  Back to top

• Application Content and Process - Does 30% - 40% of the evaluation liaison's time needs to go to working with the evaluation team?

  A grantee should plan to have 30% to 40% of the evaluation liaison’s time dedicated to the Safer Communities Project. This position’s time will not be devoted to exclusively working directly with the evaluation team. Some of the position’s time for this project will include collecting data and other tasks.

  Back to top

• Application Content and Process - How many organizations will be awarded and what is the anticipated funding amount?

  The University anticipates awarding grants to 10 to 15 nonprofit organizations throughout Pennsylvania. The University anticipates executing contracts in June 2024 with the term of the first year of the agreement expiring at the end of the 12-month period. An amendment to extend the term may be viable subject to the availability of funds. An applicant may apply for a total budget of up to $250,000 for the entire project, but may not exceed $150,000 in total expenses in the first 12-month period.

  Back to top

• Application Content and Process - Will Millersville University select programs and organizations throughout Pennsylvania or does the University have certain cities in mind? Is there a potential that more than one organization from the same city will be awarded?

  There are not certain cities preselected for this project. The intended distribution of grantees is across the four DHS regions (as shown on a slide in the webinar and the Appendix in RFA). There may be more than one organization from a particular city awarded. Millersville University is working with a directive from DHS that the University not just funnel the money to the two urban centers of Pittsburgh and Philadelphia because there are many dollars and multiple programs that are already directed at gun violence prevention and intervention services in these regions. DHS wanted to make sure that there was more of an emphasis on areas that may be left behind in other funding sources. There is no requirement prohibiting two organizations within the same city from being awarded. The priority is across the four DHS regions and in communities outside of Philadelphia and Allegheny counties.

  Back to top

• Application Content and Process - Are organizations in Philadelphia (and Alleghany) counties eligible for this grant?

  The Pennsylvania Department of Human Services has specified parameters for this grant for prioritizing organizations that are outside of Philadelphia and Allegheny counties. Millersville University will follow the guidance established by DHS. The guidance does not prohibit organizations in these counties from applying and being awarded, but the prioritization is for community-based organizations in other areas of the state. With the anticipation of there being 10 to 15 grantees, there is a reasonable chance that an organization from these counties may be awarded. It depends on a number of factors including the number of eligible applications submitted as well as the geographic distribution of applicants across the state, taking into account the four different DHS regions (as shown on a slide in the webinar and the Appendix in the RFA).

  Back to top

• Application Content and Process - What is the total dollar amount available to grantees for the total project period?

  The total dollar amount available to grantees for the total project period is: $2,907,200.

  Back to top

• Application Content and Process - Is the funding for peer-to-peer mental health support services program? Is a grantee responsible for budgeting for the evaluation? Can an applicant include 30% to 40% of the evaluation liaison's salary in the budget section of the application?

  Yes, the funding is to support the peer-to-peer mental health support services program. A grantee is not responsible for budgeting for the evaluation. The grantee can budget for 30-40% of the evaluation liaison’s position. The organization is responsible for funding the evaluation, just the grantee’s staff time (project manager, financial manager, evaluation liaison, and peer-to-peer worker). As the grantees, the organizations are being paid to deliver their programs and ensure the organization’s evaluation liaison is able to work with the evaluators.

  Back to top

• Application Content and Process - With regard to the four positions specified in the RFA (Project Manager, Financial Manager, Evaluation Liaison, and Peer-to-Peer Support Worker(s)) must these positions be currently filled at the time of the proposal submission or can they be vacant and filled after the proposal submission?

  To be eligible, an applicant must be a current provider of nonclinical, community-based peer-to-peer individualized and/or small group mental health support services for survivors of gun violence that will be implemented continuously throughout the grant period. Consequently, each position must be filled by a person with the relevant program knowledge and experience for the position to ensure expertise and continuity of program implementation. For example, a position could be filled by an individual who is familiar with and has been doing the work on a voluntary basis. Applicants should name the individuals who will be filling the four positions in the Staffing Plan portion of the application and provide a description of how each individual will ensure their position's responsibilities for this project will be fulfilled. The people must also be available to attend June and July 2024 meetings as specified in the RFA. 

  Back to top

• Application Content and Process - May an applicant include in their proposed budget non-cash incentives (eg gift cards) for participants (current and former clients, not staff) who complete surveys and/or participate in interviews or focus groups?

  Yes, an applicant may include in their proposed budget non-cash incentives such as gift cards for up to $30 per client for completing surveys, interviews, and/or focus groups. Please keep in mind each focus group will include five to seven former clients.  

  Back to top

• Application Content and Process - Is there a maximum amount of funding that can be proposed for indirect expenses in the budget?

  While there is not a specific maximum amount allowable for indirect expenses, the proposed amount should be a reasonable amount of the overall budget and should follow the organization's indirect policy. Applicants must include budget justification figures demonstrating how the proposed indirect expenses were calculated. Millersville University may require a potential grantee to reduce proposed indirect amounts if a grantee is selected through the competitive application process, but the proposed indirect amount is not justified and/or reasonable. 

  Back to top

• Application Content and Process - What are examples of indirect costs?

  Indirect Costs are expenses that have been incurred for purposes common to a number of specific programs, but which cannot be identified and charged directly to such programs relatively easily. Examples of expenses that can be covered by indirect include photocopying, heating, lighting, and janitorial services. Typically an organization calculates indirect based on a percentage of FTEs paid for by a project. 

  Back to top

• Application Content and Process - Is it permissible to include licensing fees in the proposed budget?

  No, the project will not pay for licensing fees. 

  Back to top

• Application Content and Process - Does this grant require a match?

  No, the grant does not require a match. 

  Back to top

• Application Content and Process - When is the proposal due and where should it be submitted?

  Applications must be received by 4:00pm EDT on Monday, April 15, 2024 at safer.communities@millersville.edu.

  Back to top