For the more than 30 years, the Network for Teaching Entrepreneurship (NFTE)—a public services organization—has brought the power of entrepreneurship to young people in low-income communities. It has served more than one million youth around the world. As the organization looked to improve its curriculum, it turned to smart technology to help it do so.
Although its highly trained teachers are tasked with creating a thoughtful and engaging curriculum, measuring the actual impact of its programs was difficult. Administrators were using traditional ways of assessing academic performance but weren’t able to evaluate non-cognitive skills. The IT department needed to find a solution and turned to experience management solutions from SAP as the answer.
We sat down with NFTE’s President and CEO, Dr. J.D. LaRock, and Director of Information Technology Paul Dimayuga, to understand how Qualtrics helped the organization better serve its growing student base. Both worked on developing the Entrepreneurial Mindset Index (EMI), which was a project aimed at delivering a comprehensive student assessment with results in real time. The EMI project received a 2020 SAP Innovation Award and has allowed NFTE to continue to fine-tune its program.
Sharon: Can you tell us about your role at NFTE and your involvement with the EMI project?
J.D.: I’ve been CEO for the last nine months. My involvement in the EMI project has been to expand its use and utility for our organization. The EMI is a very important instrument that we use to demonstrate the efficacy of our educational programs in public schools. Students take the EMI at the beginning of a NFTE class and they take it at the end of a NFTE class. It’s a short but very powerful assessment that allows us to know how a student has increased their skill set and the entrepreneurial mindset domains of communication, problem-solving, critical thinking, teamwork, future orientation, risk assessments, and a number of other domains.
What we do at NFTE is help underserved learners find a business idea; develop that idea; and hopefully create a business plan that they can bring out into the world. Helping students do that by developing their entrepreneurial mindset is as important as the business idea they have. And so in my role as CEO, I oversee our organization’s efforts to make sure that the EMI is being effectively used in the context of our model so that we can produce young people who have developed a greater entrepreneurial mindset.
Paul: I’m the IT director, and currently my involvement with the EMI is to oversee any changes that have to happen, which include any sort of modifications, testing, and making sure that we're not doing anything that's going to break the system.
Going forward, I am charged with maintaining the system, but my initial involvement with the EMI project was when we first chose Qualtrics as solution.
Sharon: What was your landscape before you adopted Qualtrics? How were you measuring and analyzing performance?
Paul: EMI was fairly new to us at the time. We weren’t really measuring anything like that. There were a lot of studies around the EMI assessment and how it could help by using things like personality to place someone in the right entrepreneurship category. And that become the goal.
Traditionally, we were just doing summative and diagnostic assessment. It was based on cognitive tests which were done either within the classroom on paper sheets or through Excel spreadsheets. So, it was a very manual process to import those into our database. It just wasn't sustainable, and we knew we needed to look for another solution.
Sharon: How did you decide on SAP for your experience management solution? Why Qualtrics?
Paul: We looked at a few different products, but we ultimately chose Qualtrics because it met our needs for providing assessments based on questions that were non-cognitive, as well as being able to display that data in a dashboard in a meaningful way.
Qualtrics was the only out-of-the-box solution that provided us with inline responses. As soon as the student was done with the assessment, then they could see the results of which EMI domain they belong in.
Aside from that, one of the things that was important to me was to make sure that we also spoke with other folks who were using Qualtrics. We learned very quickly that Qualtrics was really embedded within a lot of other educational institutions. I had a chance to speak with three or four other customers and got a candid view of what their pain points were; how they were using the platform; and why they chose Qualtrics. I found that their stories were very similar to ours and that the solution worked for them. That was one of the big buy-ins from the IT perspective.
I was also looking for scalability. I learned that too many of the other competitive products begin to fall apart once you increase the number of participants. I didn’t want something that was just going to meet our needs today, but not work down the road when we start to scale up. The overall support we receive from Qualtrics is peace of mind.
J.D.: To add on that, having an assessment that is good substantively is only the first step. The quality and flexibility of the platform, coupled with its utility with key audiences such as teachers and with internal audiences such as our researchers are all incredibly important factors.
These assessments are consequential to how we then use data to improve the NFTE program. And it was very clear to us as we investigated the different options that Qualtrics was, by far, the platform that was going to be highly usable, very flexible and, very useful for our needs.
Sharon: Let’s talk about the scope of the project. How long did it take from start to finish and what was involved in implementing the solution?
Paul: The project itself took us about six months from the time we finally chose Qualtrics to the time we implemented it. A lot of what was involved was making sure we had the assistance to complete the project. Qualtrics is a large product between the assessment features, to the form building, API integrations, the scripting, and everything else around it.
We knew that we didn't have a lot of time and so we needed to get help. We worked with Qualtrics to bring in professional services. They helped us build a solution rather quickly. We implemented the solution in a development environment, where we tested for three months. From there, training took a couple of months, and then we went live.
Sharon: What, if any, challenges did you encounter throughout the project and how did you address them?
Paul: The biggest issue that we had was trying to figure out how we could set up the dashboards for the different folks who need to view the data. We have different user types in different school districts, and so we had to make sure that we are compliant with who gets to see which data.
Frankly, that was also part of the reason that we chose Qualtrics. Out of all the vendors that I looked at, Qualtrics was the only one that was FedRAMP-compliant, which is really the highest tier of compliance.
Sharon: Did you have to address change management?
Paul: You know, the only thing we really needed to address was to get everyone’s mind set around the new solution and to train folks internally. I had to make sure to set up a development environment that everyone could use to make any changes and make sure they understood the reason why they had to go there. They needed to understand that changes were going to happen within the same day. So, getting that discipline in place took some time. But as soon as people understood, as soon as we broke a few eggs, we started having that methodology in place. I think right now everyone is in lockstep and we all understand the reasons why we do these things. I always feel the difficulties are more with the people involved rather than the technology itself.
Sharon: How has adopting Qualtrics affected NFTE’s overall objectives? What types of insights are you gleaning from the data you’re collecting?
J.D.: First of all, I think the flexibility and ability to scale using the Qualtrics platform allows us to analyze differences across schools and school districts in different parts of the country on the types of mindset domains that students are intending to make progress. NFTE operates internationally. Our programs are offered in 25 states in the U.S. and 10 countries around the world. So, as you might imagine, there are variations from place to place—not so much on how the NFTE program is delivered, but the types of mindset domains that are developed.
What using the EMI undergirded by the Qualtrics platform allows us to do is understand how the NFTE program and a student's larger school experience connects together. It helps us understand the ways in which those two things play out differently from region to region and country to country, and then allows us to work with our partners to make adjustments, if needed, to how we deliver the program.
Sharon: Do you plan on doing more with Qualtrics in the future?
Paul: We've got a number of research studies in mind where we’ll use Qualtrics data to help us look at the long-term value proposition of NFTE programs. At this point in our organizational research, we've largely focused on the near-term impacts of the NFTE experience. What the scale and scope of EMI powered by Qualtrics helps us do is facilitate longer-term research into longer-term outcomes of NFTE learners. For example, we can now look at the students who had extremely strong, short-term mindset domain growth in different areas and go back to our alumni and assess whether those mindset domain strengths persist. We can analyze whether there are other mindset domains that come to shore as students go on to college, become business owners, or just go out into the larger world.
There is a lot of interest right now in the longer-term impacts of educational interventions. What we have now with the EMI, powered by Qualtrics, is useful in helping us to do a much deeper investigation of those longer-term issues.
Sharon: What advice would you give to other organizations looking to implement Qualtrics?
Paul: Do your homework. Do the comparisons. Make sure that you are speaking with other organizations in your vertical market that are looking for a solution. Even though Qualtrics seems to be one of the more expensive solutions, if you look at all the feature sets it provides, it was actually the only solution that we saw that provided both the forum-building survey assessments with a very rich dashboard. You really have to look at your own requirements and your own needs, but make sure that you're comparing those as a whole against the other platforms as well.
To learn more about Qualtrics, register for an ASUG on-demand webcast, “Redefining Customer Experience with SAP Qualtrics Offerings."