The key elements of gamification in corporate training – The Delphi method

Pre-reading Note:

This paper has received the 2022 Best Paper Award from the journal and is the most downloaded paper of the year.

The original research titled 'The Key Elements of Gamification in Corporate Training – The Delphi Method' is published in the SCI journal Entertainment Computing under the Creative Commons CC BY 4.0 license for open access. In order to facilitate researchers and educators in Taiwan who are interested in instructional gamification for reading, citation, further research, and practical application, the original Chinese version of the paper is made publicly available.

As the blog format may not be as rigorous as a journal, the following content is provided for ease of reading. The official format and content are based on the English version published in the journal. At the end of the article, gamification references are also listed in APA format. Readers are welcome to refer to them for further in-depth research.

Additionally, the author has written an introductory article, 'From Practice to Academia, and Back from Academia to Practice: A Glimpse of Key Research in Educational Training Gamification,' explaining the reason for conducting this research, expressing gratitude, and providing a list of relevant practical experiences and articles on instructional gamification. Readers are also welcome to read it.

Journal Article DOI link: https://doi.org/10.1016/j.entcom.2021.100463

Academic citation reference (APA format): Wang, Y. F., Hsu, Y. F., & Fang, K. (2021). The Key Elements of Gamification in Corporate Training – The Delphi Method. Entertainment Computing, 40, 100463. https://doi.org/10.1016/j.entcom.2021.100463"

Abstract

Gamification of training, the application of gamification to modify training content and methodology, has become a hot topic in corporate training and school education in recent years. However, the relevant key elements have not been identified. Therefore, we aimed to engender and prioritize a list of key elements of effective gamification in the course of corporate training.

Using the Delphi method, this study surveyed 14 corporate training gamification experts to evaluate the key elements of gamification – 12 were identified and categorized into six design principles: integration with training goals, rapid feedback, team competition, clear rules, goal-oriented challenges, and freedom to fail. Points and leaderboards were also named as the keys to effective game mechanics.

The integration of educational objectives with gamification design, and the balancing between game mechanics and design principles were the conception for gamification in corporate training. This study also provided qualitative data, presenting the rich and differentiated views of corporate training experts regarding the key elements of gamification, to provide some practical tips for teachers in the development of training gamification.

Keywords

Adult learningDelphi methodEmployee trainingTraining gamification

1. Introduction

1.1. The concept of gamification

Gamification, defined as “the use of game design elements in non-game contexts” [1], gradually became a hot topic after 2010 [2], [3], [4]. The idea is to turn tasks into game-playing to boost intrinsic motivation [5], entice more participants and to increase their level of engagement in the task [2]. Gamification has since been widely used in different fields such as business operations, marketing, and education [6], [7], [8].

The field of education and training has also generated a great deal of interest in gamification [2], [9], [10]. The process of applying gamification elements to modify training content and methodology is called gamification of learning [11] or training gamification. This has become a common choice in redesigning traditional training [12], and its application in higher education and corporate employee training has also become more prevalent. Moving forward, the gamification market growth estimate is billions of U.S. dollars, and this new trend will likely transform the workplace in the future [13], [14].

Currently, a unified theoretical framework for academic research on gamification is still lacking. Existing studies on gamification are based on quantitative research methods that rely on survey data to infer user behavior; there is very little qualitative research on gamification [15]. Even though there is growing application and literature on gamification for adult education, many issues are unresolved. For example, the effectiveness of gamification is unclear [11], and the key elements of gamification await clarification [16]. From existing literature, it is difficult to decipher the gamification elements or combinations with the most impact [17]. Therefore, this study applied the Delphi method to conduct surveys of expert opinions, targeting corporate training professionals as survey subjects. The study focused on two goals:

(1) Exploration of key elements of successful gamification in corporate training practices. (2) Evaluate key gamification elements and their degree of importance, and how corporate trainers apply such elements in their practices.

2. Gamification in research and practice

2.1. Gamification

As gamification is still in the early stages of development, its definition is often misunderstood [17]. Among related terminologies such as games, serious games, game-based learning and gamification, there are many similarities but also plenty of differences in principles and concepts [11], [17]. A game is a made-up imaginary world with very specific rules, and often played at a specific time and place [18]. Usually for entertainment purposes, a game consists of many elements, including game functions or gameplay [3]. Serious games are also called educational games or learning games – their primary purpose is education and not entertainment [18]. Even though very closely related to games, serious games are often used in different learning environments [11], such as military, government, education, business, and healthcare [19]. For example, “American's Army”, considered the most successful military serious games, is used to attract new recruits into the army [20]. Serious games can also be used as flight simulations to support pilot training [21]. One of the focuses of game-based learning is about learning inducement, while serious games focus on behavior modification of the trainees [22]. Nevertheless, game-based learning is often used interchangeably in the literature with serious games [23].

Different from ordinary games, gamification doesn’t provide an independent experience, and doesn’t have a complete narrative or program [24]. It differs from serious games [11], and has no relation to software applications or computers [25]. The widely accepted definition is “the use of game design elements in non-game contexts” [1], also known simply as the “points, badges, leaderboard” method [5]. The goal is to enhance participants’ motivation through the addition of gaming elements [26], enticing users and encouraging specific types of behavior, creating significant driving force to induce desirable user behavior [27].

2.2. Gamification of training

Gamification can attract the attention of students [16], thus drawing interest towards its implementation in education [2], [9], [10]. Gamification of learning is the process of applying gaming elements to modify training content and methodology [11], to help develop student potential and provide positive encouragement to boost their performance level [28]. Gamification is not about playing games in the classroom. It is the insertion of a few gaming elements into an existing teaching program. It is a type of educational design process, and not a type of teaching method. The user learns from the educational content, they do not learn from gamification [11]. Thus, gamification of training is defined here as applying gamified elements in a training environment. The implementation of gamification elements in the process of learning, teaching and training may elevate students’ interest and motivation to participate in the subject content, increasing their concentration and enjoyment of learning [24], [29]. As students maintain a proactive attitude, improve their learning process and overcome any potential psychological barriers [30], they will be able to learn with efficiency [15].

2.3. Key elements of successful gamification

Common gamification elements are points, scoring, leaderboards [17], [30], [31], progress bars, ranks, rewards or incentives, all of which often appear in the implementation of gamification [32]. The addition of a story or narratives is also part of gamification [33]. Common gamification design principles include: goals and challenges, personalization, rapid feedback, visible feedback, freedom of choice, freedom to fail and social engagement [9]. A review study of educational and learning gamification found that gamification elements may be divided into five different types including: progressive achievement, social interaction, immersion experience, non-digital elements, and others, while the most common elements are points, challenges, badges and leaderboard [34].

Another way to categorize gamification elements is extrinsic motivation vs. intrinsic motivation [35], which may be explained by applying theories of psychology, including the theory of operant conditioning, the expectancy-based theories, the theories of self-regulation and the theory of self-determination [11]. For example, from the perspective of the theory of operant conditioning, points and badges can be considered as reinforcement of operant conditioning, prompting students to respond to teaching expectations and complete specific tasks or goals [36]. The self-regulation theories may be used to explain how to apply the leaderboard to encourage students to attempt more challenging goals [37]. The theory of self-determination may explain how badges can satisfy students’ need for positive feedback on their abilities [38].

2.4. Possible gamification problems

Although past research showed that gamification achieved positive effects in general, some gamification approaches may lead to inconsistent results [15]. For example, the leaderboard and the badges may cause interference with intrinsic motivation [35], or lower satisfaction and reduce the momentum for action [39], or lead to discontentment with the fairness of gamification [16]. If students view the gaming elements as redundant, then the implementation of gamification may be counterproductive [17]. If the focus is on students’ erroneous behavior or attitude, simply making the training program fun will not improve the effect of the training [11]. Students will ultimately lose interest if the implementation fails to adhere to essential gamification design guidelines [24].

3. Method

Originated from the RAND Corporation in 1950 [40], [41], the Delphi method uses an interactive iterative process to seek expert consensus [42], [43], in order to make decisions or evaluations [44], or conduct predictive research [41]. The classical Delphi method includes the following four key features: anonymity, iteration, controlled feedback and statistical summary [42], [44], [45], [46], [47], [48]. When a problem or phenomenon is not fully understood, then the Delphi method is a suitable framework for research [49], [50]. The Delphi research process calls for a pre-round to develop a key elements questionnaire for discussion during subsequent rounds [51]. The Likert five-point scale may be used to evaluate the consistency of expert opinions [51]. In the first round, experts are asked to fill out a questionnaire, which are collected for statistical analysis. Afterwards, experts are invited for a second-round evaluation where they may reference the analysis result from the first round, and revise or expand on their answers in the first round. Next, a similar analysis and evaluation/selection process is used for the third round [42]. At least two rounds of questionnaires are conducted iteratively to obtain expert opinions. For most studies, two to three rounds of iterative discussions using the Delphi methodology is enough to reach a consensus [45].

Even though the concept of consensus is the foundation of many Delphi studies, the meaning of consensus is not yet clearly defined. The most common definition is a certain percentage in agreement or a certain ratio of scores within a designated range [52]. The Inter-quartile range (IQR), the standard deviation (SD) and the frequency distribution are often used as the basis of consensus assessment [45]. An IQR of less than 1 indicates that over 50% of the expert opinions are focused on a certain point. An IQR of zero indicates that the experts are in complete agreement [53]. The SD is an indicator of the level of opinion dispersion. An SD value of less than 1.5 indicates that the experts have reached a consensus [45]. Moreover, if the average score for an element is greater than 4 on the five-point scale, and more than 51% of the experts assigned a score between 4 and 5, it is an indication that experts view the item being evaluated as a key element. When multiple indicators are used for measurement at the same time, it is possible to obtain a more reliable expert consensus [45].

Therefore, this study has set the consensus criteria as IQR≦ 1 and SD≦1.5, which are regarded as the requirements for a consensus of expert opinions. After a consensus has been reached, when an item received an average score greater than 4 in the expert evaluation, and more than 51% of the expert evaluations gave a score of 4 or above, the item is considered a gamification key element.

Research related to educational gamification is still in the early stages [54], thus this study adopted the Delphi method to explore the key elements of training gamification. Fourteen corporate training experts with ample experience in training gamification were invited to participate in three-rounds of expert opinion collections in accordance with the Delphi method, to evaluate the key ingredients for successful gamification of training. To reach consensus among the experts and to select the key elements of gamification, this study conducted at least two rounds of expert-opinion collection, and the Delphi research stopped after the third round or earlier when consensus was reached. The data for this research were collected between September 2019 through March 2020.

3.1. Expert participants

The selection of appropriate experts is essential to the Delphi research method [41]. Suitable experts would be knowledgeable and experienced, have the willingness and time to participate, and be equipped with communication skills [50]. Therefore, experts were chosen purposefully [55], relying on expert recommendations [45] or the snowball method of sampling [42]. If the experts are homogeneous, then 10 to 15 people would be enough to form a representative group [42], [56].

For this research, two gamification training experts served on the nomination team. One of them is a coach specialized in training corporate lecturers, and a published author of several books on educational training and gamification teaching methodology. The other nominating team member is a gamification research scholar.

Furthermore, the following three criteria were considered for the nomination of educational gamification experts: (1) The nominee has utilized common gamification elements such as points, scoring, and leaderboards within their training courses. (2) The nominee has more than three years of experience with educational gamification. (3) The nominee has been a lecturer at one of the top 500 listed companies in Taiwan over the past year.

To ensure the completeness of the gamification expert nomination, three sources were used to collect the list of gamification experts, including: human resource executives of listed companies, executives of management consulting firms that specialize in corporate training, and the recommendation of fellow corporate lecturers. In the end, 27 qualified gamification experts were nominated. Invitations were extended via phone or email, and 16 of the 27 experts agreed to participate in the research, and two of them withdrew in the middle of the process. The remaining 14 experts completed all three rounds of survey questionnaires. The participants included 12 males (86%) and 2 females (14%). Among them, they averaged about 7 years of gamification teaching experience, and most of them (n = 7, 50%) had more than 9 years of gamification experience.

The content of courses taught by these experts included the following fields: presentation skills, innovation and creativity, project management, communication skills, problem analysis and resolution, sales skills, language skills, online marketing, finance and financial report, and train the trainer. Within the last year of the survey, each expert had multiple experiences performing gamification training at some of the top 500 listed companies in Taiwan. The gamification approach used by the experts in their own practices were also collected. A total of 29 gamification elements were used, including points, leaderboards, virtual identities, puzzles, storylines, experiential activities, integration with software apps, and so on. For detail information on the experts who participated in this research, please refer to Table 1.

Table 1. Professional Qualifications of Surveyed Experts.

Code	Gender	Years of Experience	Fields of Educational Expertise	The approaches of gamification that each expert used in their own practice
A	Male	Over 9 years	Communication Skills	Integration with training goal, Game rules, Fairness, Task with challenging goals, Team work, Points, Time pressure, Experiential activities, Competition, Leaderboard, Prize or bonus money, Storyline, Virtual identity
B	Male	6 to 9 years	Problem Analysis and Resolution	Integration with training goal, Team work, Experiential activities
C	Female	3 to 6 years	Sales Skills	Integration with training goal, Fairness, Team work, Points, Competition, Prize or bonus money
D	Male	3 to 6 years	Presentation Skills	Integration with training goal, Game rules, Fairness, Team work, Points, Competition, Prize or bonus money, Board games, Virtual identity
E	Male	3 to 6 years	Innovation and Creativity	Integration with training goal, Game rules, Fairness, Team work, Points, Competition, Prize or bonus money
F	Female	Over 9 years	Language Skills	Integration with training goal, Team work, Points, Time Pressure, Competition, Virtual currency or chips, Integration with software apps
G	Male	3 to 6 years	Presentation Skills	Integration with training goal, Rapid feedback, Task with challenging goals, Team work, Points, Time pressure, Competition, Prize or bonus money, Storyline, Social network, Virtual identity, Board games
H	Male	3 to 6 years	Online Marketing	Integration with training goal, Prize or bonus money, Storyline, Virtual identity
I	Male	Over 9 years	Finance and Financial Report	Integration with training goal, Fairness Team work, Points, Competition, Leaderboard, Prize or bonus money, Virtual currency or chips
J	Male	3 to 6 years	Presentation Skills	Integration with training goal, Game rules, Team work, Points, Competition, Clues, Virtual identity
K	Male	Over 9 years	Project Management	Integration with training goal, Game rules, Task with challenging goals, Team work, Points, Time pressure, Competition, Puzzles, Virtual identity
L	Male	Over 9 years	Project Management	Integration with training goal, Rapid feedback, Game rules, Task with challenging goals, Team work, Points, Time Pressure, Increasing difficulty, Experiential activities, Competition, Leaderboard, Performance status feedback, Storyline, Clues, Social network, Progress bar, Virtual identity.
M	Male	Over 9 years	Training of Lecturers	Integration with training goal, Rapid feedback, Game rules, Fairness, Task with challenging goals, Team work, Points, Time pressure, Increasing difficulty, Competition, Prize or bonus money
N	Male	Over 9 years	Innovation and Creativity	Integration with training goal, Game rules, Team work, Points, Time pressure, Competition, Puzzles, Virtual currency or chips, Storyline, Board games, Virtual identity, Virtual Helper
O†	Male	3 to 6 years	Project Management	Integration with training goal, Fairness, Team work, Points, Competition, Prize or bonus money

Note. † indicated this expert withdrew this study in the second round period.

3.2. Pre-round of Delphi method

For the pre-round of this research, experts were interviewed via email and telephone to explore their practical experiences of training gamification, and the following three questions were asked: (1) training gamification implementation examples, (2) the key elements of training gamification, and (3) training gamification challenges and recommendations. After the expert’s answers were transcribed, consideration was given to key gamification elements of education and learning based on a review of empirical literature by Majuri et al. (2018) and a review of related gamification research [3], [9], [31], [32], [33], [55], [56]. Comparisons were made of the research reviews and practical gamification experience described by experts during the interviews, and the data were categorized into 35 key elements of training gamification. Out of the 35 items, 19 items (59%) were recognized by both the literature and the experts interviewed, 11 items (31%) were proposed by the literature and 5 items (14%) were derived from the experts’ opinions,this would serve as the basis for subsequent expert evaluation of key elements in training gamification.

3.3. The three Delphi rounds

For the first round, the experts were asked via email to evaluate 35 training gamification key elements using the Likert five-point scale, where points 1 through 5 indicated not at all important, not important, neutral, important and very important, respectively. The order of the questions in each expert questionnaire had been randomized to avoid ordering bias [41]. At the end of the questionnaire, the experts were invited to write in key elements that should have been on the list. The consensus criteria were: IQR≦ 1 and SD≦ 1.5 [45].

At the first round, 24 items met the criteria of consensus in terms of IQR, SD. For the first round, no experts had added any key element items. Since the Delphi research method requires at least two iterative rounds to obtain expert opinions [42], thus all gamification key elements evaluated were kept for the second round of Delphi research.

One of the special features of the Delphi method is to allow participants to reference other people’s opinions anonymously, to iteratively refine their perspectives or revise their own ideas [42], [45]. For the second round, participating experts were again asked to evaluate the 35 gamification elements, and each item’s descriptive statistical data of expert opinions in the previous round were also available to them, including the mean, SD, and IQR. If an expert’s opinion exceeded the upper or lower IQR, that expert was asked to provide an explanation. At the end of the second round, the experts reached consensus for 33 items, which was 9 more than the first round. For the third round, the experts were asked to evaluate only the remaining 2 items without consensus. In the end, the experts met the criteria and arrived at consensus on all 35 items, thus ending the Delphi method investigation. The situation was in line with most Delphi method research implementations of 2 to 3 rounds of surveys [45]. Table 2 and Table 3 shows the three rounds of iterative process.

Table 2. The result of the first and the second round of expert opinion collection.

Empty Cell	Round 1 (n = 15)					Round 2 (n = 14)
Elements	Mean	SD	IQR	≧4 Ratio	Rank	Mean	SD	IQR	≧4 Ratio	Rank
Integration with training goal	5.0	0.00	0.0	100%	1	5.0	0.0	0.0	100%	1
Rapid feedback	4.8	0.40	0.0	100%	4	5.0	0.0	0.0	100%	2
Game rules	4.8	0.40	0.0	100%	3	4.9	0.26	0.0	100%	3
Fairness	4.8	0.34	0.0	100%	2	4.7	0.59	0.0	93%	4
Task with challenging goals	4.6	0.49	1.0	100%	6	4.5	0.49	1.0	100%	5
Team work	4.5	0.73	1.0	86%	9	4.4	0.63	1.0	93%	6
Points or scoring	4.1	0.98	1.5*	73%	16	4.4	0.62	1.0	93%	7
Time Pressure	4.8	0.44	0.5	100%	5	4.3	0.45	0.75	100%	8
Increasing difficulty	4.5	0.63	1.0	93%	8	4.3	0.48	1.0	100%	9
Experiential activities	4.4	0.73	1.0	86%	11	4.3	0.48	1.0	100%	10
Competition	4.6	0.60	0.5	93%	7	4.3	0.62	1.0	93%	11
Freedom to fail	4.5	0.62	1.0	93%	10	4.1	0.77	1.0	79%	12
Educational tools or props	4.4	0.47	1.0	100%	12	3.8	0.59	0.0	79%	13
Leaderboard	4.2	0.91	2.0*	67%	13	3.8	0.67	0.0	79%	14
Prize or bonus money	3.9	1.15	1.0	80%	14	3.8	0.88	1.0	57%	15
Adaptation to difficulty	4.0	0.85	1.0	73%	15	3.7	0.86	1.0	64%	16
Performance status feedback	3.7	1.05	2.0*	60%	21	3.6	0.73	1.0	64%	17
Virtual currency or chips	3.7	1.25	2.0*	40%	17	3.5	0.97	1.0	43%	18
Storyline	3.5	1.25	2.0*	60%	23	3.4	0.62	1.0	50%	19
Peer rating	3.7	0.70	1.0	67%	18	3.3	0.59	1	36%	20
Clues	3.5	0.71	1.0	60%	22	3.3	0.72	1	36%	21
Social network	3.7	1.05	1.5*	67%	20	3.3	0.91	1.75*	36%	22
Customization or personalization	3.3	0.77	1.0	47%	24	3.2	0.7	0.75	29%	23
Puzzles	3.6	0.71	1.0	47%	19	3.1	0.70	0.75	29%	25
Badges	3.0	0.97	1.0	27%	27	3.0	0.64	0.75	14%	24
Board games	2.7	0.83	1.0	20%	31	2.9	0.67	0	7%	26
Progress bar	3.2	1.00	1.0	33%	25	2.9	0.74	0	14%	27
Integration with software apps	2.5	0.71	0.5	0%	32	2.8	0.72	1	7%	28
Virtual identity	3.0	1.10	1.5*	33%	29	2.7	0.82	1	14%	29
Virtual Helper	2.9	1.06	0.0	13%	28	2.7	0.91	1	14%	30
Virtual treasure	3.0	0.82	0.5	20%	26	2.4	0.72	1	0%	31
Hierarchy	2.8	1.00	1.5*	27%	30	2.3	0.73	1	7%	32
Virtual reality	2.5	1.15	1.5*	13%	33	2.3	0.94	1.75*	7%	33
Health score	1.8	0.93	1.5*	7%	34	1.8	0.77	1	7%	34
Check in	1.8	0.91	2.0*	0%	35	1.6	0.62	1	0%	35

Note. Total 15 experts participated in the first round, while the 14 experts participated in the second round. At the first round, there 24 items met the criteria of consensus and 11 items did not meet. At the second round, there 33 items met the criteria of consensus and 2 items did not meet. * shown that did not meet the criteria of consensus.

SD: standard deviation; IQR: inter-quartile range; ≧4 Ratio: The numbers was 4 points and above / total numbers × 100%.

Table 3. The result of the third round of expert opinion collection.

Elements	Mean	SD	Q1	Q3	IQR	≧4 No	≧4 Ratio
Social network	3.10	0.86	3.00	4.00	1.00	5	36%
Virtual reality	2.20	0.69	2.00	3.00	1.00	0	0%

Note. Total 14 experts participated in the third round. Here, the 2 items which did not meet at the second round were be met the criteria of consensus at the third round.

SD: standard deviation; Q1: first quartile; Q3: third quartile; IQR: inter-quartile range; ≧4 No: The numbers was 4 points and above; ≧4 Ratio: The numbers was 4 points and above / total numbers × 100%.

4. Results and discussions

This study obtained consensus through three rounds of expert surveys, and 35 gamification key elements were ranked based on the importance mean value. Among them, 12 items (34%) had a score of four or more in mean importance value, 13 items (37%) had a mean score between 3.00 and 3.90, and 10 items (29%) had a mean score of less than three. A closer examination of items with a mean score greater than 4 shows that the ratio of experts who assigned a score of 4 or greater all exceeded 51% (the range was 79% ∼ 100%). In the end, the experts reached consensus on 12 items which the mean score ≧4 as the key elements of training gamification.

Currently, there isn’t a common method to classify gamification key elements. For example, badges may be considered as game interface design model [3], or gaming mechanics [9], [57], [58]. Dicheva el al. (2015) conducted a comprehensive analysis research of educational gamification by incorporating the gamification element definition of Deterding et al. (2011), and proposed two major categories of gamification elements: gamification design principles and game mechanics [[3], [9]]. Based on these two major categories, this study further explores effective gamification elements obtained from the surveys via the method of collection and organization.

4.1. Category I: Gamification design principles

Based on the expert evaluation score of four or more points (indicating important or very important), see Table 2, this study gathered expert opinions and organized them into six major gamification design principles.

(1) Integration with Training Goals

The experts unanimously agreed that this “Integration with training goals” was the most important key element of training gamification, with an expert scoring mean value of 5.00, SD = 0.00 and IQR = 0.00. The ratio of evaluation importance value of 4 points or above was 100% (see Table 2). The experts stated that:

N: “The purpose of games is to serve the course material. Games are the means, not the end goal. The connection between games and the course content must be very strong. Do not gamify for the sake of gamification.”

I: “Do not gamify just for gamification, it should be related to the subject that is being taught.”

G: “Without clear goals, two situations may result: (1) The learner does not know what to do. (2) After everyone experiences the high of game-playing, there is a letdown. Let the learners understand clearly what this activity is for, and what they would gain by doing a good job.”

Training instructors are advised to first determine the knowledge, skill set, or performance goals of the training, and diligently choose the elements of gamification [17]. Just as the phrase - “Do not gamify for the sake of gamification” - frequently mentioned by many experts in the study, careful consideration of the course objectives is recommended when gamification is used. Before starting the process of training gamification, our experts suggested a re-evaluation or redesign of the curriculum using the instructional design method or the ADDIE (Analysis, Design, Development, Implementation, Evaluation) model, to integrate gamification and training goals while enhancing learning effectiveness [11].

(2) Rapid Feedback

“Rapid Feedback” is a key element highly valued by the experts (mean = 5.00, SD = 0.00, IQR = 0.00, 100% of experts viewed it as important or very important, see Table 2). The experts stated that:

G: “Whether it’s a race to answer questions, presentation or simulation, immediate feedback is very important.”

D: “If you answer, you get extra points.” Strengthen the students’ “familiarity and approval of the scoring rules” so that they are even more willing to answer questions.

Ample feedback would enhance the participants’ experience [35], [59]. Points, badges and leaderboards are all key elements of feedback that encourage users’ participation and their level of engagement [51]. For this study, the participating experts mostly used points and leaderboards to provide feedback in their practices, which will be discussed further in the section on gamification element categories.

(3) Team-based Competition:

“Competition” is an important key element of successful gamification (mean = 4.30, SD = 0.62, IQR = 1.00, 93% of the experts viewed it is important or very important, see Table 2), and the unit of competition is “team work” (mean = 4.40, SD = 0.63, IQR = 1.00, 93% of experts viewed it as important or very important, see Table 2). Through team-based competitions, learners become invested in the challenges and maintain their participation and degree of engagement in the training course. Experts stated that:

J: “Set up a team-based competition, let the students compete to answer questions. …this measure brings about healthy competition and establishes a positive, autonomous cycle of team motivation.”

G: “Competition, especially team competition, can stimulate learners’ degree of participation and enhance concentration. Individual competition may evolve into a situation where some people become very active while others slowly step back into the role of “observers”.

This type of social oriented ability is also a category often seen in gamification [34]. Our experts found that a gamification design principle based on the concept of team competition would contribute to gamification of training in a corporate setting.

(4) Clear and fair game rules

It is evident from the experts’ feedback that clear and simple “game rules” is recommended (mean = 4.90, SD = 0.26, IQR = 0.00, 100% of experts viewed it as important or very important, see Table 2). “Fairness” of the game (mean = 4.70, SD = 0.59, IQR = 0.00, 93% of experts viewed it as important or very important, see Table 2) is one of the keys of training gamification implementation. Experts stated that:

I: “The design should consider fairness, timeliness, transparency, and clear rules for the scoring system”

J: “The game rules must be very easy to understand, and any games requiring too much time for explanation are really inappropriate.”

J:“When the fairness of the game is undermined, it is easy for some learners to choose to ‘give up participation’ in the process.”

Without rules and limitations, games are just simple enjoyment [60]. For training gamification, rules are part of the key elements [61], and fairness is an important link of game rules, as unfair games will cause participants to become dissatisfied [16]. Our experts suggested that one of the key elements of gamification was to determine how to motivate learner participation through good rule design, taking fairness into consideration while encouraging certain desirable behaviors from the students.

(5) Increasing level of difficulty in challenging goals

“Task with challenging goals” (mean = 4.50, SD = 0.49, IQR = 1.00, 100% of experts viewed it as important or very important, see Table 2) is an important element in training gamification design. During the process of training gamification, it is possible to design different tasks within the training course to let learners participate in the resolution. Expert opinions also believed that “increasing difficulty” (mean = 4.30, SD = 0.48, IQR = 1.00, 100% of experts viewed it as important or very important, see Table 2) and “time pressure” (mean = 4.30, SD = 0.45, IQR = 0.75, 100% of experts viewed it as important or very important, see Table 2) is an important element. This indicates that as the lessons move forward, the task’s level of difficulty may gradually increase and the learners may be asked to complete the tasks within the time limit. Experts stated that:

K: “Give learners challenges and ask them to come up with a solution within a limited timeframe, and encourage team competition.”

I: “Gamification must have layers of difficulties, from the easiest to the hardest.”

G: “Effectively control the time from the perspectives of curriculum design, adjustment of level of difficulty, and rule descriptions.”

Difficult yet concrete goals can motivate people to take actions, stimulate their performance level in pursuit of strategies for improvement, which may be explained by the theories of self-regulation [19], [11]. Our experts also suggested that as the skill sets of learners increased, a gradual increment of task difficulties helped trigger a state of flow and creative happiness for the learners, optimizing user experience by matching the challenges with upgraded abilities of the users.

(6) Experiential activities and freedom to fail

While conducting training gamification, experts also introduced “experiential activities” to replace traditional descriptive teaching (mean = 4.30, SD = 0.48, IQR = 1.00, 100% of experts viewed it as important or very important, see Table 2). Also, “freedom to fail” is an important element to success (mean = 4.10, SD = 0.77, IQR = 1.00, 79% of experts viewed it as important or very important, see Table 2). Experts stated that:

M: “Within the context of a game, experience the journey from non-cooperation to cooperation.”

G: “Allow learners to try their best in a relatively safe environment – thus it is very important to have low risk of failure. Design questions without standard answers so that everyone can speak freely. Otherwise, the trainer’s attitude and situational control skills must be excellent such that students are not embarrassed when they answer incorrectly.”

Through training gamification design, learners may look upon failures as opportunities; when they do not succeed at first, they feel free to make multiple attempts and practice repeatedly [63]. Our experts suggested that the integration of experiential activities with curriculum goals, the provision of meaningful feedback, and a risk-free learning environment would be the key elements of gamification in corporate training.

4.2. Category II: Game mechanics

Regarding gamification elements, many prior studies considered points, badges, leaderboard as the most common game mechanics [9], [17], [31], [34]. For this study, which conducted interviews with experts of corporate training, some key elements identified by the experts echoed previous research results, while other expert-identified elements differed from prior studies.

(1) Points

The first element is points. Experts in this study had a high degree of support for using gamification elements such as “points or scoring” in the training course (mean = 4.40, SD = 0.62, IQR = 1.00, 93% of experts viewed it as important or very important, see Table 2). The experts stated that:

M: “Each unit has points, which may be points for an interactive unit, or discussions or case exercises. Each of these tasks would have different points.”

E: “What I encourage is participation. What I encourage is not correctness. Wrong answers also get 100 points, but right answers get 300 points. My approach is to reward participation as well as correct answers.”

This echoes previous research conclusions. Points may build a connection between efforts, performance and results, thereby strengthening related behaviors [62]. Through attendance, participation in tasks, students receive points which may further affect the students’ level of engagement [16], [25] and enhance their learning results [17]. Care should be taken that the points themselves are not rewards, but serve as the guidelines and reminders to achieving the training goals [35]. Students should consider points or other gamification elements as valuable or having psychological significance, otherwise such measures would not increase student participation [17]. Our experts suggested that the use of a scoring mechanism would play an important role in training gamification by enhancing learning effects while heightening student reactions and expectations.

(2) Badges

Even though prior studies have identified badges as a part of gamification mechanism, the importance of “badges” were not supported by corporate training experts. After two rounds of evaluations by the experts, only 14% of the experts considered it important (mean = 3.0, SD = 0.64, IQR = 0.75, 14% of experts viewed it as important or very important, see Table 2). Most experts in this study believed that badges were not an important element to training gamification. After the completion of three rounds of Delphi method research, experts were invited to provide supplementary comments about the application of badges. The experts stated that:

N: “Many games use badges, but I think they are more suitable for educational games that are larger in size and more time-consuming, in order to provide a driving force like “stamp collection” so that players will continue to play and add to their collections. There are very few large-size and continuous educational games like that in a training environment.”

D: “Badges are usually for games where the users may upgrade their ranks. For short-term courses, to explain clearly the rules for upgrading ranks and badges to the learners may end up confusing them instead.

F: “The essence of badges is the same as bonus points, which is really a bonus point mechanism that has been exquisitely packaged and looks nicer. Perhaps it would attract teenagers or younger students.

Since the cost of implementing badges is relatively low, trainers may be particularly attracted to them [17]. Our experts found that the use of badges in short-term training courses (e.g. duration of 1–3 days) would not be effective gamification as the impact to adult learners would be insignificant. In addition, badges used frequently in software supplemented gamification systems would not be easily applicable to most corporate trainings conducted with face-to-face teaching. This is one area for further research in the future.

(3) Leaderboard

The level of importance of the “leaderboard” changed in an interesting manner through the course of this study. At the first round of expert evaluation, even though it received a mean value of 4.20 with 67% of the experts viewing it as important or very important, it did not meet the consensus standard criteria (SD = 0.91, IQR = 2.00, see Table 2). At the second round, consensus was reached (SD = 0.67, IQR = 0.00, see Table 2), but its importance mean value dropped to 3.8, and the ratio of experts viewing it as important or very important grew to nearly 80% (see Table 2). The survey process showed that even though many experts supported the use of leaderboard and believed it can strengthen competitiveness and promote student participation, some experts felt that excessive emphasis on competition would adversely affect learning. Experts stated their very different views on the importance of leaderboard as follows:

A: “The score board is right in the front. Using the team scoring method, the scores are announced and updated during each intermission.”

L: “I personally do not believe in the importance of competition. In some classes, I would even purposely minimize competition: we would not have any rankings, and no reward mechanism either, not even verbal complements.”

E: “The use of competition, is that actually good or bad? In order to score points, we may even lose a little of the essence of learning. Of course, we want to keep learning as the main objective, and not have scoring points as the main task.”

Past research showed that in the process of training gamification, the points and leaderboard mechanism can operate appropriate competitiveness, but the effects of the leaderboard were mixed [25]. If competition is over-emphasized, it could lower participants’ motivation [24], or make students uncomfortable [16]. Our experts found that appropriate application of the element of competition while emphasizing the essence of learning without excessive competition would be important to training gamification.

(4) Other findings

For corporate educational training, storylines or plots may be interwoven in the process, but the importance of stories varied with our experts. Some experts used stories as introductions prior to the beginning of the teaching process or during case discussions. Other experts, meanwhile, believed that stories were an important component to attracting the students’ attention. Regarding the importance of a storyline, the divergent views of the experts may be due to their individual application experience and different focus in terms of course planning. Since this study utilized the Delphi method, the number of expert samples was relatively small. If the expert samples came from different groups with different teaching experiences, the importance of a storyline may change. This aspect may require further research in the future. Moreover, the experts in this study seldom relied on the assistance of information software systems or apps, possibly because the use of software could increase teaching variables, distracting the attention of the students or increasing the level of difficulties. The experiences of the experts in the study seemed to indicate that the use of software was not necessary during the gamification of the teaching process – these experiences may offer a point of reference for other instructors.

5. Conclusion

Many studies have shown that training gamification effectively enhances learners’ interest and participation. However, ambiguous gamification directions may lead to erroneous results. This study used the Delphi method to evaluate the importance of key gamification elements according to 14 corporate training experts. Expert interviews and applications of gamification design principles/mechanics were also presented as a reference for future gamification research or application.

According to our experts, some gamification design principles would enhance learner participation and learning results in corporate training. The principles include: integration of gamification with the curriculum, provision of rapid feedback when learners exhibit desirable behavior, a team-based competition mechanism, and a gradual increase in the difficulties of tasks to be completed by the teams in a limited timeframe. Also, immersion learning would result given the following: clear and fair game rules, risk-free experiential activities allowing the freedom to fail, and a training environment tolerant of trials and errors.

Secondly, our experts often relied on points and leaderboards as game mechanics, planning different scoring components in the course. When the learners exhibited desirable behaviors, such as answering questions, participating in discussions, or taking on difficult challenges, they would score points for accuracy as well as participation. Furthermore, leaderboards were used to display cumulative team scores, which may boost participation by displaying the progress of the learners and their peers.

Thirdly, our experts found that the gamification of training was not about playing games in the class, and did not require the use of software applications or board games as props. Several experts also mentioned: “Do not gamify for the sake of gamification”. Thus, it is important to focus on the design of key gamification elements to enhance learning effectiveness for research and application in the future.

5.1. Limitations and future research

There are some limitations in this study. First, the number of participating experts was limited, thus further research is required to see whether these opinions are sufficient to represent a wide range of educational gamification. Moreover, most of the gamification course material were used by enterprise employees. When the intended target users are students in an academic setting or elsewhere, different effects may result. Also, corporate training sessions are often special seminars or short-term classes, unlike school classes which may last a semester or more. With different teaching timeframe, should gamification be implemented differently? Furthermore, experts that implement training gamification may not fully understand the underlying gamification theories and research, which could limit their gamification design or usage of elements. The individual course subject or practical experience of each expert may reflect their preference of different instructional methods for training gamification. For example, some experts preferred the use of rewards or storylines, while others viewed these mechanisms as less important. Such different perspectives in the qualitative results could be due to the selection and recruitment process to find expert participants for this study. Even with such research limitations, this study integrated the practical experiences of the experts as well as academic research, in hopes of offering some valuable references in the research and application of training gamification.

Funding

The authors did not receive any funding for this study.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements and Disclosures

None.

References:

[1] S. Deterding, M. Sicart, L. Nacke, K. O’Hara, D. Dixon, Gamification: using gamedesign elements in non-gaming contexts, in: Proceedings of the 2011 CHI Conference on Human Factors in Computing, ACM, Vancouver, BC, Canada, 2011, pp. 2425-2428.

[2] K. Huotari, J. Hamari, Defining gamification: a service marketing perspective, in: Proceedings of the 16th International Academic MindTrek Conference, ACM, Tampere, Finland, 2012, pp. 17-22.

[3] S. Deterding, D. Dixon, R. Khaled, L. Nacke, From game design elements to gamefulness: defining gamification, in: Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, ACM, Tampere, Finland, 2011, pp. 9-–15.

[4] J. Hamari, V. Lehdonvirta, Game design as marketing: How game mechanics create demand for virtual goods, Int. J. of Bus. Sci. & Appl. Manage. 5 (1) (2010) 14-29, http://ssrn.com/abstract=1443907.

[5] M. Vesa, J.T. Harviainen, Gamification: concepts, consequences, and critiques, J. Manage. Inquiry. 28 (2) (2019) 128–130, https://doi.org/10.1177/1056492618790911.

[6] L.C. Stanculescu, A. Bozzon, R.-J. Sips, G.-J. Houben, Work and play: an experiment in enterprise gamification, in: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work and Social Computing, ACM, San Francisco, CA, USA, 2016, pp. 346-358.

[7] R.N. Landers, Developing a theory of gamified learning: Linking serious games and gamification of learning, Simul. & Gaming. 45 (6) (2014) 752–768, https://doi.org/10.1177/1046878114563660.

[8] A. Anderson, D. Huttenlocher, J. Kleinberg, J. Leskovec, Steering user behavior with badges, in: Proceedings of the 22nd International Conference on World Wide Web, ACM, Rio de Janeiro, Brazil, 2013, pp. 95-106.

[9] D. Dicheva, C. Dichev, G. Agre, G. Angelova, Gamification in education: a systematic mapping study, Educ. Technol. Soc. 18 (3) (2015) 75–88. https://www.jstor.org/stable/10.2307/jeductechsoci.18.3.75.

[10] S. De Sousa Borges, V.H. Durelli, H.M. Reis, S. Isotani, A systematic mapping on gamification applied to education, in: Proceedings of the 29th Annual ACM symposium on Applied Computing, 2014, pp. 216–222.

[11] R.N. Landers, E.M. Auer, A. Helms, S. Marin, M.B. Armstrong, Gamification of adult learning: gamifying employee training and development, in: R.N. Lamders (Ed.), Cambridge Handbook of Technology and Employee Behavior, Cambridge University Press, Cambridge, 2019, pp. 271–295.

[12] P. Denny, The effect of virtual achievements on student engagement, in: Proceedings of 2013 Association for Computing Machinery’s Special Interest Group on Computer Human Interaction Conference, Paris, France, 2013, pp. 763-772.

[13] S. Smith-Robbins, This game sucks: how to improve the gamification of education. Educau. Rev. 46 (1) (2011) 58–59, https://er.educause.edu/-/media/files/articledownloads/erm1117.pdf.

[14] M. Burke, T. Hiltbrand, How gamification will change business intelligence. Bus. Intell. J. 16 (2) (2011) 8–16, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.476.6771&rep=rep1&type=pdf.

[15] J. Hamari, J. Koivisto, H. Sarsa, Does gamification work? a literature review of empirical studies on gamification, in: Proceedings of the 47th Hawaii International Conference on System Sciences, Waikoloa, HI, USA, 2014, pp. 3025-3034.

[16] L.A. Ribeiro, T.L. da Silva, A.Q. Mussi, Gamification: a methodology to motivate engagement and participation in a higher education environment, Inter. J. of Educ. & Res. 6 (4) (2018) 249-264, https://www.ijern.com/journal/2018/April-2018/21.pdf.

[17] M.B. Armstrong, R.N. Landers, Gamification of employee training and development, Int. J. Train. & Dev. 22 (2) (2018) 162–169, https://doi.org/10.1111/ijtd.12124.

[18] D.R. Michael, S.L. Chen, Serious Games: Games that Educate, Train, and Inform, Muska & Lipman/Premier-Trade, New York, 2005.

[19] R. Garris, R. Ahlers, J.E. Driskell, Games, motivation, and learning: a research and practice model, Simul. Gaming. 33 (4) (2002) 441–467, https://doi.org/10.1177/1046878102238607.

[20] D. Nieborg, America‟ s Army: more than a game, in: Proceedings of the 35th Conference of the International Simulation and Gaming Association, Munich, Germany, 2004.

[21] C. Perryer, B. Scott-Ladd, C. Leighton, Gamification: implications for workplace intrinsic motivation in the 21st century, Asian Forum on Bus. Educ. J. 5 (3) (2012)371–381. https://doi.org/10.14456/afbe.2012.13.

[22] B. Sawyer, P. Smith, Serious games taxonomy, in: Proceedings of the Serious Games Summit at the Game Developers Conference, San Francisco, CA, USA, 2008. [23] T.M. Connolly, E.A. Boyle, E. MacArthur, T. Hainey, J.M. Boyle, A systematic

literature review of empirical evidence on computer games and serious games, Comput. Educ. 59 (2) (2012) 661–686. https://doi:10.1016/j.compedu.2012.03.004.

[24] I. Furdu, C. Tomozei, U. Kose, Pros and cons gamification and gaming in classroom, Broad Res. Arti Intell. Neurosci. 8 (2017) 56–62. https://doi:10.5281/zenodo.1045364.

[25] T.A. Papp, A. Theresa, Gamification effects on motivation and learning: Application to primary and college students, Int. J. Cross-Discip. Subj. in Educ. 8(3) (2017) 3193–3201, http://doi.org/10.20533/ijcdse.2042.6364.2017.0428.

[26] B.M. Sørensen, S. Spoelstra, Play at work: continuation, intervention and usurpation, Organ. 19 (1) (2012) 81–97, https://doi.org/10.1177/1350508411407369.

[27] G. Barata, S. Gama, J. Jorge, D. Gonçalves, Studying student differentiation in gamified education: a long-term study, Comput. Hum. Behav. 71 (2017) 550–585, https://doi.org/10.1016/j.chb.2016.08.049.

[28] J. Hamari, J. Koivisto, Why do people use gamification services? Int. J. Inf.Manage. 35 (4) (2015) 419–431, https://doi.org/10.1016/j.ijinfomgt.2015.04.006.

[29] R.N. Landers, M.B. Armstrong, A.B. Collmus, How to use game elements to enhance learning: applications of the theory of gamified learning, in: M. Ma, A. Oikonomou, L.C. Jain (Eds.), Serious Games and Edutainment Applications, Springer, New York, 2017, pp. 457–483.

[30] R.N. Landers, K.N. Bauer, R.C. Callan, M.B. Armstrong, in: Psychological theory and the gamification of learning, Gamification in Education and Business, Springer, New York, 2015, pp. 165–186.

[31] K. Seaborn, D.I. Fels, Gamification in theory and action: a survey, Int. J. Hum. Comput. Stud. 74 (2015) 14–31, https://doi.org/10.1016/j.ijhcs.2014.09.006.

[32] F.F.-H. Nah, Q. Zeng, V.R. Telaprolu, A.P. Ayyappa, and B. Eschenbrenner, Gamification of education: a review of literature, in: Proceedings of the International Conference on Human–Computer Interaction in Business, Springer, Heraklion, Crete, Greece, 2014. pp. 401-409.

[33] M.B. Armstrong, R.N. Landers, An evaluation of gamified training: using narrative to improve reactions and learning, Simul. Gaming 48 (4) (2017) 513–538, https://doi.org/10.1177/1046878117703749.

[34] J. Majuri, J. Koivisto, J. Hamari, Gamification of education and learning: a review of empirical literature, in: Proceedings of the 2nd International GamiFIN Conference, GamiFIN, Pori, Finland, 2018.

[35] C. Perryer, N.A. Celestine, B. Scott-Ladd, C. Leighton, Enhancing workplace motivation through gamification: transferrable lessons from pedagogy, Int. J.Manage. Educ. 14 (3) (2016) 327–335, https://doi.org/10.1016/j.ijme.2016.07.001.

[36] K.M. Kapp, The Gamification of Learning and Instruction, John Wiley & Sons Inc, San Francisco, 2012.

[37] R.N. Landers, K.N. Bauer, R.C. Callan, Gamification of task performance with leaderboards: a goal setting experiment, Comput. Hum. Behav. 71 (2017) 508–515, https://doi.org/10.1016/j.chb.2015.08.008.

[38] R.M. Ryan, C.S. Rigby, A. Przybylski, The motivational pull of video games: a selfdetermination theory approach, Motiv. & Emot. 30 (4) (2006) 344–360, https://doi.org/10.1007/s11031-006-9051-8.

[39] M.D. Hanus, J. Fox, Assessing the effects of gamification in the classroom: a longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance, Comput. & Educ. 80 (2015) 152–161, https://doi.org/10.1016/j.compedu.2014.08.019. 

[40] N. Dalkey, O. Helmer, An experimental application of the Delphi method to the use of experts, Manag. Sci. 9 (3) (1963) 458–467, https://doi.org/10.1287/mnsc.9.3.458.

[41] C. Okoli, S.D. Pawlowski, The Delphi method as a research tool: an example, design considerations and applications, Inf. & Manage. 42 (1) (2004) 15–29, https://doi.org/10.1016/j.im.2003.11.002.

[42] G.J. Skulmoski, F.T. Hartman, J. Krahn, The Delphi method for graduate research, J. Inf. Technol. Educ. Res. 6 (1) (2007) 1–21. https://doi.org/10.28945/199.

[43] W. Vernon, The Delphi method: a review, Int. J. Ther. Rehab. 16 (2) (2009) 69–76.https://doi.org/10.12968/ijtr.2009.16.2.38892.

[44] G. Rowe, G. Wright, The Delphi technique as a forecasting tool: issues and analysis, Inter. J. Forecast. 15 (4) (1999) 353–375, https://doi.org/10.1016/S0169-2070(99)00018-7.

[45] L. Giannarou, E. Zervas, Using Delphi technique to build consensus in practice. Inter. J. of Bus. Sci. & Appl. Manage. 9 (2) (2014) 65-82, https://doi.org/10.5430/jha.v3n4p1.

[46] J. Landeta, Current validity of the Delphi method in social sciences, Technol. Forecast. & Soc. Change. 73 (5) (2006) 467–482, https://doi.org/10.1016/j.techfore.2005.09.002.

[47] T. Shields, G. Silcock, H. Donegan, Y. Bell, Methodological problems associated with the use of the Delphi technique, Fire Technol. 23 (3) (1987) 175–185, https://doi.org/10.1007/BF01036934.

[48] N.C. Dalkey, The Delphi method: an experimental study of group opinion, in: D.R. Lewis, D. Snyder (Eds.), Studies in the Quality of Life: Delphi and DecisionMaking, Lexington Books, Lexington, 1972, pp. 13–54.

[49] P. Kittell-Limerick, Perceived Barriers to Completion of the Academic Doctorate: A Delphi Study, Texas A and M University Commerce, Bryan, 2005.

[50] M. Adler, E. Ziglio, Gazing into the Oracle: The Delphi Method and its Application to Social Policy and Public Health, Jessica Kingsley Publishers, London, 1996.

[51] J. Greatorex, T. Dexter, An accessible analytical approach for investigating what happens between the rounds of a Delphi study, J. Adv. Nurs. 32 (4) (2000)1016–1024, https://doi.org/10.1046/j.1365-2648.2000.t01-1-01569.x.

[52] I.R. Diamond, R.C. Grant, B.M. Feldman, P.B. Pencharz, S.C. Ling, A.M. Moore, P.W. Wales, Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies, J. of Cli. Epide. 67 (4) (2014) 401–409, https://doi.org/10.1016/j.jclinepi.2013.12.002.

[53] D. Ramos, P. Arezes, P. Afonso, Application of the Delphi Method for the inclusion of externalities in occupational safety and health analysis, DYNA 83 (196) (2016)14–20. https://dx.doi.org/10.15446/dyna.v83n196.56603.

[54] L.E. Nacke, C.S. Deterding, The maturing of gamification research, Comput. Hum Behav. (2017) 450–454, https://doi.org/10.1016/j.chb.2016.11.062.

[55] A. Fink, J. Kosecoff, How to Conduct Surveys: A Step by Step Guide, Sage Publications Inc., Newbury Park, 2015.

[56] A.L. Delbecq, A.H. Van de Ven, D.H. Gustafson, Group techniques for program planning: a guide to nominal group and Delphi processes, Scott Foresman & Co., Glenview, 1975.

[57] G. Zichermann, C. Cunningham, Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps, O’Reilly Media Inc, Sebastopol, 2011.

[58] J. Antin, E.F. Churchill, Badges in social media: a social psychological perspective, in: Proceedings of the 2011 CHI Conference on Human Factors in Computing Systems, ACM, Vancouver, BC, Canada, 2011, pp. 1-4.

[59] J. Juul, A Casual Revolution: Reinventing Video Games and Their Players, MIT Press, Cambridge, 2010.

[60] T. Fullerton, Game Design Workshop: A Playcentric Approach to Creating Innovative Games, CRC Press, Boca Raton, 2014.

[61] R.W. Helms, R. Barneveld, F. Dalpiaz, A method for the design of gamified trainings, in: Proceedings of the Pacific Asia Conference on Information Systems, Singapore, 2015, pp. 59.

[62] L. Von Ahn, L. Dabbish, Designing games with a purpose, Commun. ACM. 51 (8) (2008) 58–67, https://doi.org/10.1145/1378704.1378719.

[63] Kiryakova, G., Angelova, N., Yordanova, L. Gamification in education, in: Proceedings of the 9th International Balkan Education and Science ConferenceTurkey, 2014.