For the more than 30 years, the Net­work for Teach­ing Entre­pre­neur­ship (NFTE) — a pub­lic ser­vices orga­ni­za­tion — has brought the pow­er of entre­pre­neur­ship to young peo­ple in low-income com­mu­ni­ties. It has served more than one mil­lion youth around the world. As the orga­ni­za­tion looked to improve its cur­ricu­lum, it turned to smart tech­nol­o­gy to help it do so. 

Although its high­ly trained teach­ers are tasked with cre­at­ing a thought­ful and engag­ing cur­ricu­lum, mea­sur­ing the actu­al impact of its pro­grams was dif­fi­cult. Admin­is­tra­tors were using tra­di­tion­al ways of assess­ing aca­d­e­m­ic per­for­mance but weren’t able to eval­u­ate non-cog­ni­tive skills. The IT depart­ment need­ed to find a solu­tion and turned to expe­ri­ence man­age­ment solu­tions from SAP as the answer. 

We sat down with NFTE’s Pres­i­dent and CEO, Dr. J.D. LaRock, and Direc­tor of Infor­ma­tion Tech­nol­o­gy Paul Dimayu­ga, to under­stand how Qualtrics helped the orga­ni­za­tion bet­ter serve its grow­ing stu­dent base. Both worked on devel­op­ing the Entre­pre­neur­ial Mind­set Index (EMI), which was a project aimed at deliv­er­ing a com­pre­hen­sive stu­dent assess­ment with results in real time. The EMI project received a 2020 SAP Inno­va­tion Award and has allowed NFTE to con­tin­ue to fine-tune its program. 

Sharon: Can you tell us about your role at NFTE and your involve­ment with the EMI project? 

J.D.: I’ve been CEO for the last nine months. My involve­ment in the EMI project has been to expand its use and util­i­ty for our orga­ni­za­tion. The EMI is a very impor­tant instru­ment that we use to demon­strate the effi­ca­cy of our edu­ca­tion­al pro­grams in pub­lic schools. Stu­dents take the EMI at the begin­ning of a NFTE class and they take it at the end of a NFTE class. It’s a short but very pow­er­ful assess­ment that allows us to know how a stu­dent has increased their skill set and the entre­pre­neur­ial mind­set domains of com­mu­ni­ca­tion, prob­lem-solv­ing, crit­i­cal think­ing, team­work, future ori­en­ta­tion, risk assess­ments, and a num­ber of oth­er domains.

What we do at NFTE is help under­served learn­ers find a busi­ness idea; devel­op that idea; and hope­ful­ly cre­ate a busi­ness plan that they can bring out into the world. Help­ing stu­dents do that by devel­op­ing their entre­pre­neur­ial mind­set is as impor­tant as the busi­ness idea they have. And so in my role as CEO, I over­see our organization’s efforts to make sure that the EMI is being effec­tive­ly used in the con­text of our mod­el so that we can pro­duce young peo­ple who have devel­oped a greater entre­pre­neur­ial mindset.

Paul: I’m the IT direc­tor, and cur­rent­ly my involve­ment with the EMI is to over­see any changes that have to hap­pen, which include any sort of mod­i­fi­ca­tions, test­ing, and mak­ing sure that we’re not doing any­thing that’s going to break the system. 

Going for­ward, I am charged with main­tain­ing the sys­tem, but my ini­tial involve­ment with the EMI project was when we first chose Qualtrics as solution. 

Sharon: What was your land­scape before you adopt­ed Qualtrics? How were you mea­sur­ing and ana­lyz­ing performance?

Paul: EMI was fair­ly new to us at the time. We weren’t real­ly mea­sur­ing any­thing like that. There were a lot of stud­ies around the EMI assess­ment and how it could help by using things like per­son­al­i­ty to place some­one in the right entre­pre­neur­ship cat­e­go­ry. And that become the goal. 

Tra­di­tion­al­ly, we were just doing sum­ma­tive and diag­nos­tic assess­ment. It was based on cog­ni­tive tests which were done either with­in the class­room on paper sheets or through Excel spread­sheets. So, it was a very man­u­al process to import those into our data­base. It just was­n’t sus­tain­able, and we knew we need­ed to look for anoth­er solution.

Sharon: How did you decide on SAP for your expe­ri­ence man­age­ment solu­tion? Why Qualtrics?

Paul: We looked at a few dif­fer­ent prod­ucts, but we ulti­mate­ly chose Qualtrics because it met our needs for pro­vid­ing assess­ments based on ques­tions that were non-cog­ni­tive, as well as being able to dis­play that data in a dash­board in a mean­ing­ful way. 

Qualtrics was the only out-of-the-box solu­tion that pro­vid­ed us with inline respons­es. As soon as the stu­dent was done with the assess­ment, then they could see the results of which EMI domain they belong in.

Aside from that, one of the things that was impor­tant to me was to make sure that we also spoke with oth­er folks who were using Qualtrics. We learned very quick­ly that Qualtrics was real­ly embed­ded with­in a lot of oth­er edu­ca­tion­al insti­tu­tions. I had a chance to speak with three or four oth­er cus­tomers and got a can­did view of what their pain points were; how they were using the plat­form; and why they chose Qualtrics. I found that their sto­ries were very sim­i­lar to ours and that the solu­tion worked for them. That was one of the big buy-ins from the IT perspective. 

I was also look­ing for scal­a­bil­i­ty. I learned that too many of the oth­er com­pet­i­tive prod­ucts begin to fall apart once you increase the num­ber of par­tic­i­pants. I didn’t want some­thing that was just going to meet our needs today, but not work down the road when we start to scale up. The over­all sup­port we receive from Qualtrics is peace of mind. 

J.D.: To add on that, hav­ing an assess­ment that is good sub­stan­tive­ly is only the first step. The qual­i­ty and flex­i­bil­i­ty of the plat­form, cou­pled with its util­i­ty with key audi­ences such as teach­ers and with inter­nal audi­ences such as our researchers are all incred­i­bly impor­tant factors. 

These assess­ments are con­se­quen­tial to how we then use data to improve the NFTE pro­gram. And it was very clear to us as we inves­ti­gat­ed the dif­fer­ent options that Qualtrics was, by far, the plat­form that was going to be high­ly usable, very flex­i­ble and, very use­ful for our needs.

Sharon: Let’s talk about the scope of the project. How long did it take from start to fin­ish and what was involved in imple­ment­ing the solution? 

Paul: The project itself took us about six months from the time we final­ly chose Qualtrics to the time we imple­ment­ed it. A lot of what was involved was mak­ing sure we had the assis­tance to com­plete the project. Qualtrics is a large prod­uct between the assess­ment fea­tures, to the form build­ing, API inte­gra­tions, the script­ing, and every­thing else around it. 

We knew that we did­n’t have a lot of time and so we need­ed to get help. We worked with Qualtrics to bring in pro­fes­sion­al ser­vices. They helped us build a solu­tion rather quick­ly. We imple­ment­ed the solu­tion in a devel­op­ment envi­ron­ment, where we test­ed for three months. From there, train­ing took a cou­ple of months, and then we went live. 

Sharon: What, if any, chal­lenges did you encounter through­out the project and how did you address them?

Paul: The biggest issue that we had was try­ing to fig­ure out how we could set up the dash­boards for the dif­fer­ent folks who need to view the data. We have dif­fer­ent user types in dif­fer­ent school dis­tricts, and so we had to make sure that we are com­pli­ant with who gets to see which data. 

Frankly, that was also part of the rea­son that we chose Qualtrics. Out of all the ven­dors that I looked at, Qualtrics was the only one that was FedRAMP-com­pli­ant, which is real­ly the high­est tier of compliance. 

Sharon: Did you have to address change management? 

Paul: You know, the only thing we real­ly need­ed to address was to get everyone’s mind set around the new solu­tion and to train folks inter­nal­ly. I had to make sure to set up a devel­op­ment envi­ron­ment that every­one could use to make any changes and make sure they under­stood the rea­son why they had to go there. They need­ed to under­stand that changes were going to hap­pen with­in the same day. So, get­ting that dis­ci­pline in place took some time. But as soon as peo­ple under­stood, as soon as we broke a few eggs, we start­ed hav­ing that method­ol­o­gy in place. I think right now every­one is in lock­step and we all under­stand the rea­sons why we do these things. I always feel the dif­fi­cul­ties are more with the peo­ple involved rather than the tech­nol­o­gy itself.

Sharon: How has adopt­ing Qualtrics affect­ed NFTE’s over­all objec­tives? What types of insights are you glean­ing from the data you’re collecting? 

J.D.: First of all, I think the flex­i­bil­i­ty and abil­i­ty to scale using the Qualtrics plat­form allows us to ana­lyze dif­fer­ences across schools and school dis­tricts in dif­fer­ent parts of the coun­try on the types of mind­set domains that stu­dents are intend­ing to make progress. NFTE oper­ates inter­na­tion­al­ly. Our pro­grams are offered in 25 states in the U.S. and 10 coun­tries around the world. So, as you might imag­ine, there are vari­a­tions from place to place — not so much on how the NFTE pro­gram is deliv­ered, but the types of mind­set domains that are developed. 

What using the EMI under­gird­ed by the Qualtrics plat­form allows us to do is under­stand how the NFTE pro­gram and a stu­den­t’s larg­er school expe­ri­ence con­nects togeth­er. It helps us under­stand the ways in which those two things play out dif­fer­ent­ly from region to region and coun­try to coun­try, and then allows us to work with our part­ners to make adjust­ments, if need­ed, to how we deliv­er the program.

Sharon: Do you plan on doing more with Qualtrics in the future? 

Paul: We’ve got a num­ber of research stud­ies in mind where we’ll use Qualtrics data to help us look at the long-term val­ue propo­si­tion of NFTE pro­grams. At this point in our orga­ni­za­tion­al research, we’ve large­ly focused on the near-term impacts of the NFTE expe­ri­ence. What the scale and scope of EMI pow­ered by Qualtrics helps us do is facil­i­tate longer-term research into longer-term out­comes of NFTE learn­ers. For exam­ple, we can now look at the stu­dents who had extreme­ly strong, short-term mind­set domain growth in dif­fer­ent areas and go back to our alum­ni and assess whether those mind­set domain strengths per­sist. We can ana­lyze whether there are oth­er mind­set domains that come to shore as stu­dents go on to col­lege, become busi­ness own­ers, or just go out into the larg­er world. 

There is a lot of inter­est right now in the longer-term impacts of edu­ca­tion­al inter­ven­tions. What we have now with the EMI, pow­ered by Qualtrics, is use­ful in help­ing us to do a much deep­er inves­ti­ga­tion of those longer-term issues.

Sharon: What advice would you give to oth­er orga­ni­za­tions look­ing to imple­ment Qualtrics? 

Paul: Do your home­work. Do the com­par­isons. Make sure that you are speak­ing with oth­er orga­ni­za­tions in your ver­ti­cal mar­ket that are look­ing for a solu­tion. Even though Qualtrics seems to be one of the more expen­sive solu­tions, if you look at all the fea­ture sets it pro­vides, it was actu­al­ly the only solu­tion that we saw that pro­vid­ed both the forum-build­ing sur­vey assess­ments with a very rich dash­board. You real­ly have to look at your own require­ments and your own needs, but make sure that you’re com­par­ing those as a whole against the oth­er plat­forms as well. 

To learn more about Qualtrics, reg­is­ter for an ASUG on-demand web­cast, ​“Redefin­ing Cus­tomer Expe­ri­ence with SAP Qualtrics Offerings.”