Not just another focus group: making the case for the nominal group technique in criminology
© Vander Laenen; licensee Springer. 2015
Received: 2 June 2014
Accepted: 23 December 2014
Published: 4 March 2015
The Nominal Group Technique (NGT) is seldom applied in criminological research. This article explores the potential of NGT as a tool for criminological research.
NGT is a highly structured technique combining characteristics of an individual survey and a focus group.
It offers various benefits: 1) it limits researcher influence and influence from group dynamics; 2) increases the likelihood of equal participation for all group members; 3) affords equal influence to (conflicting) values and ideas; 4) can be used in an exploratory (phase of a) study as well as to generate hypotheses about topics that are unfamiliar to the researcher; and 5) is useful for determining the ideas of a research population that is socially or culturally different from that of the researcher.
NGT is particularly relevant in applied research as a decision-making tool and as a consensus method. It also holds promise as an online tool for criminological research focused on sensitive topics where participants take part anonymously.
Introduction: group techniques in social sciences and criminology
In qualitative research, there are various reasons to use group techniques – e.g. the practice of allowing individual subjects to participate together as a group – to collect data. Interactions that take place using group techniques add depth to the obtained information and allow shared meanings to emerge, which is not possible in individual interviews (Macphail, 2001). Group techniques also tend to stimulate the expression of new, bias-free ideas. They have shown to be particularly effective in research on vulnerable groups and young people. By allowing people to participate as a group, their influence is increased. Group techniques also counterbalance the direct impact of the researcher on the discussions and partially neutralises the power differences between participants and decision-makers or professionals (Wilkinson, 1999; Madriz, 2003). Group techniques allow access to research participants who may find one-to-one interaction frightening or intimidating. Such participants tend to be more willing to talk to peers than researchers about sensitive topics (Large & Beheshti, 2001; Madriz, 2003).
The most well-known and frequently-applied group technique is the focus group. aSince the end of the 1980’s, focus groups have become common in the social sciences. Focus groups involve a small number of participants discussing topics raised by a moderator, who guides the interview process. The method is based on group dynamics; interaction between group members is the defining characteristic. This interaction distinguishes a focus group from a group interview (where the focus is on the interaction between the researcher and the participants, MacDougall & Fudge, 2001). Group discussions tend to produce lively and timely images of the motives, interests, questions and ideas of the participants involved (Merton et al. 1990). Focus group research is regularly applied in qualitative criminological research. For example, it is used to study the ideas and perceptions of professionals working in the criminological field (see, among others, Douglas & Cuskelly, 2012; Kaye et al. 2014), ideas and perceptions of the general public and young people on crime and the justice system (Boda & Szabo, 2011; Dirikx et al. 2012), and perceptions of professionals on sensitive topics such as police peer retaliation (Cancino & Enriquez, 2004) and the use of force by the police (Klukkert et al. 2009). It has been regularly applied to study sensitive topics such as abuse, violence, drug use and sex work and the populations – victims, offenders and prisoners – involved (see among others Wachholz & Miedema, 2000; Surratt et al. 2004; Garcia & Lane 2012).
However, group techniques – and focus groups in particular – have disadvantages as well. The most frequently mentioned disadvantages are associated with group dynamics. Group discussions may be dominated by one or more individuals (Macphail, 2001). Participants may feel pressured to conform with peers and/or dominant individuals or ideas in the group. The group dynamic may influence the attitudes of participating individuals (Bristol & Fern, 2003). Groups can impede individual reactions resulting in ‘groupthink’. In most group techniques, it is difficult to guarantee equal participation among all group members (Macphail, 2001). Moreover, the central role of the researcher can fundamentally influence a group discussion (Campbell, 2002).
In sum, the epistemological value of data generated through methods involving group dynamics notwithstanding, these dynamics pose a challenge for researchers. In this respect, it is worthwhile to discuss the Nominal Group Technique (NGT) in detail. bNGT is a highly structured group technique that uses face-to-face meetings. NGT combines both individual and group phases. The purpose of the structure and individual phases is to limit group dynamics and dynamics of social power. The technique prevents dominant people from controlling the group and limits the interplay of the researcher in the generation of ideas (Aspinal et al., 2006). Overall, it is aimed at giving each person an equal opportunity to generate ideas, speak and vote (Macphail, 2001). Group discussion and group interaction carry equal weight since individual judgements are combined. NGT proceeds along a fixed format consisting of four phases, with individual and group phases. cThe individual stages enhance the opportunity for all participants to contribute ideas and influence decision-making equally because each person has the same power in voting. It limits the influence of dominant, more confident or outspoken group members, or those with perceived high status (Gallagher et al., 1993; Gaskin, 2003; Jones, 2004). The individual stages also enhance the opportunity for minority voices and votes to become part of the generated data. Moreover, the ideas that are proposed, discussed and voted on all come from the participants and not from the researcher, thus limiting the influence of the researcher. Apart from NGT’s individual phases, the whole process takes place in a group setting, which allows access to research participants who may find one-to-one interaction frightening or intimidating.
NGT has been regularly applied in research on education and health (services). dFor example, it is used in needs assessment research (Keatinge et al., 2000) to study unmet needs (Martinez & Carter-Pokras, 2006; Drennan et al., 2007), among other things. In criminological research, NGT is rarely ever applied. eWe encountered very few publications that applied NGT, even within the broader field of criminology. In 1973, Zastrow used NGT to study young peoples’ and professionals’ goals and priorities for curbing youth delinquency. In (Jordan 1992) studied alternative policing strategies in the U.S. for meeting the needs of urban communities by the year 2001 with the help of a panel of law enforcement executives and community leaders. In a 2007 U.S. study (Government Accountability Office GAO 2007), an expert panel discussed challenges and alternative strategies for monitoring and evaluating the results of international projects fighting human trafficking. NGT has been used as a group problem-solving technique by community police officers to establish a relationship between themselves and community members and to assist the community in developing action plans to confront its problems (Wiatrowski & Campoverde, 1996). In drug research, NGT has been applied in a piece of participatory research concerning harm reduction treatment with clients and to develop culturally relevant outcomes to measure progress (Ruefli & Rogers, 2004). Among these, only two studies can be called recent. One applied NGT in a group of hotel security experts to reach consensus on a baseline anti-terrorist strategy for a hotel (Paraskevas, 2013). Another used NGT in an exploratory study of young people on the role of software piracy in decisions to adopt video game consoles (Kartas & Goode, 2012). For criminological research, NGT could potentially be used for research aiming to focus research on the unique perspective of a particular research population. NGT presupposes that the members of such a population are considered subjects and not research objects. At the same time, it is recognized that no ‘right’ or ‘true’ or general meaning exists (Weber, 2003). For research based on other theoretical approaches, NGT can be particularly relevant in applied research as a decision-making tool and as a consensus method.
While focus groups are used widely in criminological research, NGT has been mostly neglected. In what follows, we will discuss and reflect on the strengths, weaknesses and potential applications of NGT. We also give an example of NGT in criminological research by reporting on the use of NGT in a study on the needs assessment of drug (prevention) policy (Vander Laenen, 2011). Since NGT is scarcely used in criminological research, it is worth describing the sampling, format and process of decision-making and the data analysis.
The nominal group technique: a brief manual
NGT is a single-purpose technique (Delbecq et al. 1975) and only a limited number of topics and issues can be covered. The question posed at the start of the NG meeting itself is critical. It will determine the quality of the generated ideas. NGT’s success depends on the unambiguous formulation of the question (Tuffrey-Wijne et al., 2007). A research question has to be concrete, in a language that is appropriate for the participants, and sufficiently refined. As a rule of thumb, avoid asking too general a question. For instance, do not ask ‘what administrative problems do you expect in the development of an integrated drug policy?’ because respondents will resort to generalities such as ‘poor communication’ or ‘insufficient motivation’. It is more useful to ask the respondents to list critical incidents, examples or descriptions of behaviour that illustrate the problems they expect (Delbecq et al. 1975). This will promote detailed and realistic responses. To this end, guidelines may be added to the central research question. Still, the question should be sufficiently open-ended for participants with novel solutions. Consequently, sufficient time and effort must be invested in developing the research question. The framing of the starting question is determined by the goal of the NG meetings. It can be developed by the research team or it can be developed in earlier stages of research. In view of its importance, composing and pilot testing more than one question is advised.
The researchers need to bring a sufficient number of flash cards (for the participants to write down their individual ideas during phase 1), a flip-chart (for the researcher to write down the individual ideas during phase 2), two different colored markers (one to write down their ideas, one to tabulate the scores), 3) sufficient pens for the participants; 4) and cardboard for the participants to write down their names. Next, the table should ideally be arranged in a u-shape so that the participants can all see the flip-chart. Before executing a NG meeting, it is also important to develop a road map to increase the structure of the meeting (to be used by the panel chairman) as well as an observation schedule to write down additional information (for the observer). Reflections of the participants generated throughout the NG meeting are written down in the observation schedule. The wording and the expressions used by the participants are also written down. Discussions or disagreements on some of the ideas and the reflections of the participants on the priority list should receive specific attention.
Sampling for qualitative methods differs from sampling for quantitative methods. In the former case, samples are purposive rather than random; they aim to select cases that will provide rich data (MacDougall & Fudge, 2001). The same holds for sampling in a NG. Depending on the type of research, NGT participants will vary. They can be professional experts or any group of people who have expertise in a given subject. This is based on the notion that anyone whose interests are affected by a problem are experts regardless of whether they are professionals (Aspinal et al., 2006). The number of participants for a NGT varies from study to study, ranging between six to twelve participants (Gaskin, 2003). Delbecq et al. (1975) strongly suggest limiting the number of participants to between five and nine. For young people, it is advisable to limit the number of participants to six or seven (Heary & Hennessy, 2002). The number of NG’s to be organised will depend on the type of research. There are no clear guidelines on the number of NG’s to reach saturation, e.g. the point at which no new information or themes are observed in the data. It is clear though that for a comprehensive assessment of a problem area to emerge “several NGT sessions should be conducted, using target groups whose experience directly relates to the problem area, but from different perspectives” (Delbecq et al. 1975, p. 113). In studies, the number of NG’s varies between a single or a few NG meetings up to ten NG meetings (Ramirez et al., 2000; Aspinal et al., 2006) and more (Vander Laenen, 2009).
The traditional NGT procedure
NGT proceeds along a format consisting of four phases: the silent generation of ideas in writing; round-robin feedback from group members to record each idea; discussion of each recorded idea for clarification and evaluation, and individual voting on priority ideas. At the start of the NG, the purpose and the procedure of the meeting are explained. The researcher can introduce the participants or they can introduce themselves. The ground rules for the interaction and the role of the panel chairman and of the observer are explained. During phase one, the researcher states an open-ended question without allowing discussion and he hands out flash cards to each participant. Each participant spends several minutes in silence individually brainstorming all the possible ideas and writing them down using a brief sentence or some words. During phase two, the ideas are collected by the researcher by sharing them round-robin fashion (one response per person each time), while all ideas/items are recorded on a flipchart using a key term or a short sentence. No comments from participants are allowed, but clarification in response to questions is encouraged. The round-robin technique is used to limit the ownership of the ideas since the group members discuss the individual ideas and the individual who wrote down the idea does not have to clarify nor ‘defend’ it. The items written on the flip chart use the wording of the participants. During this phase, participants are allowed to write down new ideas, and present them when his/her turn occurs in the round-robin (Gallagher et al., 1993). The third phase is aimed at discussing each recorded idea one-by-one for clarification in the group. Similar ideas are joined and identical ideas are discarded. The role of the panel chairman is to ensure that every participant gets a fair chance of clarifying an idea (this can be his own idea or the idea of another participant). In this phase, the role of the chairman is similar to the role of the moderator during a focus group. During phase four, each participant evaluates the ideas and individually and anonymously votes for the best ideas. Variations to the voting system are applied in research (Frankel, 1987), e.g. the best idea gets five points, the next-best four, and so on. Each participant can score each idea from one to five; each participant can freely attribute six points to the ideas, etc. The votes are then shared in the group and tabulated so it becomes visually clear which ideas received the most points and are prioritized. It is possible to add a fifth phase when consensus is required. To this end, after the results from the vote in phase four have become clear, a discussion in the group of the preliminary vote is held and a re-vote takes place. A NGT lasts on average between 60 and 75 minutes. Some authors who use NGT to reach consensus say that two hours may be necessary (Gaskin, 2003).
The modified NGT
In the literature, several authors have reported modifications to NGT since its introduction by Delbecq et al. NGT is regularly used in combination with other qualitative (group) techniques, such as interviews and focus groups. NGT has been applied as part of a focus group (Gaber & Gaber, 2002; Rosemann & Vessey, 2008) and NGT’s and focus groups have been used as separate techniques within a single study (Ramirez et al., 2000; Ginsburg et al., 2002). Other variations are possible. In some NG’s evidence is used alongside expert opinion before voting (Kapiriri et al. 2003; Ayuso-Mateos et al., 2013). Aspinal et al. (2006) prepared a meeting by identifying issues and ranking them before attending the NG. It is also possible to allow more discussion during phase three of the NG, consistent with a focus group format (Fein et al., 1997).
In recent years, online applications of group techniques are on the rise. These can take the form of real-time synchronous or asynchronous applications. In the synchronous form, participants are online simultaneously at a prearranged time and group discussion is interactive and in real time. The asynchronous form refers to a website where participants log in during a set research period (for instance one week) and participate at a time that is convenient for them without live interaction taking place between the participants (Tates et al., 2009). As is the case for traditional real life group discussions, online group techniques have been applied as a standalone technique. They are applied following qualitative interviews to identify expert opinions. In these examples, NGT is used for the voting on these opinions (Lossius et al., 2013). In a study we are currently executing, f39 harm reduction strategies were identified during interviews with professionals and during focus groups with problem drug users. These strategies are then voted on by 120 professionals via a modified asynchronous online NGT and by 30 problems drug users via paper.
NGT combines quantitative and qualitative data collection methods and also yields data that can be analysed both quantitatively and qualitatively. Qualitatively, a content analysis of the individual items can be carried out. This enables themes to be constructed out of the individual items. Moreover, the ‘discussion’ sections of the procedure can be analysed by means of coding and using standard Computer Aided Qualitative Data Analysis software. Note that this necessitates the use of recording equipment, which is not always allowed in some settings (e.g. in prison) or may be refused by some participants.
NGT enables data to be generated in such a way as to limit the amount of analysis required after the group session (Aspinal et al., 2006). The data processing and analyses of NGT results is relatively simple. After a NGT, the researcher has already compiled an overview of the ideas for each group, the ideas with the highest score and the number of participants that scored a specific idea. In the event that different NG’s respond to the same research question, further analysis is needed, as each group generates different lists of items and these are difficult to compare. Some studies collate all group scores to give an overall priority of items from the different groups (Dening et al. 2012). Other studies have developed a standard set of priority items derived from all the group meetings that can be voted on by all the participants from the NG’s (Vander Laenen, 2009) or by larger groups of participants (Tuffrey-Wijne et al., 2007). Quantitatively, it is possible to compare results across individuals belonging to several categories, e.g. based on background characteristics such as age, gender and ethnicity, provided that participants complete a questionnaire (Ramirez et al., 2000; Ginsburg et al., 2002).
A case study: NGT in drug research
Study design: rationale for using NGT
Study design and data collection
Research build up
Nr. of groups
Develop central study question
What should good drug prevention be like?
What are the most important things: 1) to make sure that young people do not use drugs? 2) to make sure that young people who do use drugs do not get into trouble? 3) to better help young people who get into trouble with drugs anyhow?
What is the most important idea? (each priority 1–5)
What do you want/need to make sure the priorities are put into practice?
Returning to the field
Do you agree with the analysis? If not, what do you want to change/add?
The participants were asked for oral informed consent. Their parents were asked for passive informed consent. Participants were paid for their participation, and their anonymity and confidentiality were guaranteed. No staff members were present during the groups (Balen et al. 2006).
The starting question was developed in the first part of the study during open focus groups. During these focus groups, the researcher asked the young people one broad question – ‘What should good drug prevention be like?’ – in order to develop the central youth-centred research question. The analysis of the focus groups showed that the concept of ‘drug prevention’ could not be used for the central question and that harm minimization and care should be included alongside prevention. These findings led to a three-fold central research question for the NGT, which was worded as follows: “How can we make sure that young people do not use drugs? How can we make sure that young people who do use drugs do not get into trouble? What should be done to better help young people who get into trouble with drugs anyhow?”. Two definitions were added to this question (one on ‘drugs’ and one on ‘getting into trouble’). Next, two general guidelines were added. The first guideline stressed that the researchers were looking for the ideas the participant himself found most important; the second guideline stressed the importance of generating achievable, concrete ideas. The starting question (and the two definitions and guidelines) were pilot tested in one NG meeting. Since the participants did not know whether they should respond to each of the three elements of the question, an additional guideline was added to indicate that this was indeed expected. For the NGT a road map, describing in detail the process of the NGT and an observation schedule for the observer was developedh.
The NGT procedure
In our study, the NGT procedure was mildly modified to accommodate the research population. In phase one (the idea-generation phase), some participants where assisted by the researcher in writing down their ideas, i.e. the participants formulated the idea and the researcher wrote it down for them. In phase two (the round-robin feedback), the researcher read each idea aloud to the group in order to further limit the ownership of the ideas and to avoid embarrassment for participants with limited reading skills. In phase four, the researcher collected all individual votes and shuffled them to guarantee maximal anonymity. During this phase, the researcher asked participants to assist her in tabulating the votes in order to increase the contest element of the technique and to allow participants to expend excess energy. In the study, reaching consensus was not a goal and so the procedure ended when the votes were tabulated. Moreover, the results of the NGT were not the end of the study since the participants added meaning to the results and looked for practically workable solutions to the identified problems during a separate research phase, using focus groups. Two researchers, a panel chairman and an observer executed the NGT. The NGT meetings lasted on average 54 minutes.
Data collection and analysis
The study was executed in seven different settings; in each setting, two NG meetings were organised. The results of the NG meetings were analysed per age group by comparing the different ideas to look for recurring ideas. After four-to-five NG meetings, no new ideas were added to the list of ideas discussed during previous NG meetings. Still, to honour the engagement of the residential settings in the study and out of respect for the participants, each of the parts of the study (see Table 1) was executed in each of the settings. Consequently, seven NG meetings per age group were conducted.
Number of top-5 items and scores divided by age group
Intervention strategies for drug (prevention)
Substance orientated prevention
- Don’t start
- Limit availability
- Can’t be avoided - legalize
- Limit use
Person orientated prevention
- Provide information
- Learn social skills
Environment orientated prevention
- Early intervention
- Harm reduction prevention
Each group gave priority scores only for those solutions generated within their own group. In order to have the population generate a standard prioritization, a survey was developed that included the three ideas with the highest score from each nominal group (NG) meeting. Initially, this resulted in a priority list of 21 items. To optimize the diversity of the survey in line with the Zinberg model, in some cases an idea from a NG session with a low score was added to the priority list. This also allowed more adolescents to be exposed to the creative response. For instance, no idea on early intervention was included in the priority list despite its proven effectiveness, and earlier research stressed the importance of trust for young people in communication. Therefore, the item ‘talk about it with people that can be trusted’ was added. Since the NGT was executed in two age groups, two separate priority lists were developed. The ideas were written as the participants had worded them and placed in random order. Two versions of the surveys were used, listing the ideas in different orders to decrease the likelihood that the order of presentation would affect the responses (Ramirez et al., 2000). The participants in the NGT meetings were asked to rate how each item would prevent (problem) drug use or help problem drug users on a five-point Likert scale. The surveys were self-administered after standard instructions were read.
Strengths and weaknesses of NGT
Online application NGT
- Subject-matter experts sample
- Number of creative ideas generated
- Access to difficult to engage population
- Equal participation (in idea generation and in voting)
- Saves (travel) expenses and time
- Limits researcher influence – participant-led
- Flexibility – convenience (asynchrounous)
- Highly structured
- Participants’ sense of accomplishment and motivation
- Basic data-analysis quick and easy
- Part of mixed-method
- Selection and response bias
- Restrict participation
- Single-topic sessions - research question critical
- Limited computer access or computer-illiteracy
- Group dynamics remain
- Lack nonverbal cues
- Two researchers needed
- Lack of real-time group dynamic
- Physically attend meeting – no anonymity
- (Limited) reading and writing skills required
- Provides only limited depth
NGT methodology: strengths and weaknesses (Table 3)
Strengths of NGT
The primary advantage of NGT over other group techniques is the enhanced opportunity for all participants to contribute ideas and influence decision-making. It limits the influence of dominant, more confident or outspoken group members as well as group members with perceived high status (Gallagher et al., 1993; Gaskin, 2003; Jones, 2004). Focus groups can reflect the perceptions of the most outspoken or opinionated members and can focus on what the group considers most interesting or controversial but not necessarily most important (Fein et al., 1997). A greater number of ideas are generated in NGT’s than in other group processes. Van de Ven and Delbecq (1974) compared the conventional interacting group with NGT in terms of the quantity of ideas generated. They found that NGT was more effective than conventional interaction groups (de Ruyter (1996) and Eisele (2007) found the same results). Because of the individual phases in the process, each idea has equal opportunity to be put on the agenda and each person has the same power in voting. Shedlin & Schreiber (1995) and Tuffrey-Wijne et al. (2007) state that the experience of giving and ranking ideas may be particularly empowering for people who have been traditionally excluded from participating in research. Furthermore, NGT’s structure limits the influence of the researcher (Macphail, 2001). The participants generate and score the priorities. Tellingly, the term participant-led priorities is used in the literature. The role of the researcher is limited; he/she follows a series of pre-stated steps, whereby the procedure will not significantly differ between the different NG meetings. NGT is highly structured, with clearly defined phases and goals. For research with children and young people, the use of a more structured technique rather than more open-ended methods such as focus groups helps to create a non-threatening environment for the participants (Porter, 2012). However, it can also be a disadvantage because participants need to feel comfortable with – and remain within – this structured group process (Jones, 2004). NGT is a technique that is positively evaluated by participants. Some authors state that it gives a greater sense of accomplishment and satisfaction for the participants than focus groups (Jones, 2004). Next, the NG meeting is often one of the first experiences where verbal glibness is not the overriding base of power (Delbecq et al. 1975; Ruefli & Rogers, 2004), which is particularly satisfying for (young) people who experience difficulties expressing themselves. In the case study described above, the visibility of the results of the NG indeed proved rewarding for the participants; their priority list was literally hanging on the wall. Zastrow (1973), p. 111 called this the ‘game mystique’. At the end of a focus group, results can be overly abstract and are not visible. It is relatively easy to interpret the results of NGT meetings, as ideas are generated and voted on/ranked during the session itself. In focus groups, the recorded discussions have to be transcribed ad verbatim and analysed, which is a time-consuming and skilled exercise (Gallagher et al., 1993). However, in order to avoid missing too much rich information, further in-depth analysis of the individual ideas and results of a NG is advisable, so that the time required to analyse NGs is comparable to the time invested in focus group data analysis.
An advantage of online applications of NGT and of group discussions in general is that they offer anonymity, which has been found to stimulate disclosure in sensitive topics (Tates et al., 2009, Stover & Goodman, 2012). Importantly, the online form can enable researchers to gain access to groups that are difficult to engage via others research methods, in particular young people and vulnerable groups (Stewart & Williams, 2005; Yu et al. 2011; Thomas et al. 2013). For researchers, the online application saves (travel) expenses and time for transcribing data since the data are already in writing (Tates et al., 2009). The asynchronicity of online groups increases the flexibility and convenience of logging in and is particularly advantageous for reaching people with a busy schedule (Stewart & Williams, 2005). Next, the asynchronous format gives participants ample time for reflection and allows them to respond at length and change or nuance their opinion (Tates et al., 2009). That being said, in our study on harm reduction strategies we experienced that the response to the open-ended questions was rather limited.
Weaknesses of NGT
As a sampling technique, NGT shares a disadvantage characteristic of all qualitative group techniques: as a rule, purposive sampling is used. As in focus groups, the results of the NGT cannot be generalized to a wider population due to the limited number of participants. Notwithstanding the possibility to test the results of a NGT by using a survey with a representative sample, the initial selection of the NG participants is a key factor and is open to bias due to the specific characteristics of the participants, both in terms of who is selected to attend (a possible selection bias), and who agrees to participate (a potential response bias) (van Teijlingen et al., 2006). In research with professionals aimed at consensus, the level of expertise is crucial to its success and the validity of the data it generates (Harvey & Holmes, 2012). Contrary to focus groups, NGT is a single-purpose technique (Delbecq et al. 1975) and only a limited number of topics and issues can be covered. The starting question in itself is a critical and potentially weak element. NGT’s discussion stages do not guarantee anonymity, which may limit participants’ willingness to express their views, especially on more controversial issues (Fein et al. 1997; Van Teijlingen et al. 2006). Even structured group processes may be influenced by powerful social dynamics, especially in adolescent populations (Ramirez et al., 2000). NGT can be used with participants with limited writing skills on the condition that the researcher assists the participants during individual phases of the procedure. However, to participate, a certain level of verbal ability and a capacity to express ideas is essential. NGT therefore is not suitable for people with severe and profound intellectual disabilities; for this group, for instance asking simple open questions and using communication aids such as symbol cards can facilitate research participation (Beail & Williams, 2014). A major disadvantage of NGT in a qualitative research design is that is does not add sufficient depth to the research results. The requirements for a moderator in a NGT meeting are less demanding than those for a moderator in a focus group meeting since the role of the researcher in a NGT is more limited. However, in order to create a safe and structured environment within the group, ensure that all participants are able to express their views and keep particular personal or professional views from dominating the discussion, a skilled and experienced group facilitator is still essential (Gallagher et al., 1993; Tuffrey-Wijne et al., 2007). Furthermore, the potential for researcher bias remains, particularly when the results are further analysed and a survey with additional questions is developed from the NGT results. During a NGT, ideally two researchers should be present to support participants during the individual phases in reading and writing. One researcher can pay particular attention to individual needs of participants while the other researcher can continue the process with the group. In fact, two researchers are always advisable in group techniques, because much (non)verbal communication escapes the attention of a single moderator (Vander Laenen, 2009).
An important limitation of online applications of NGT – in particular when studying marginalised or older populations – is that limited computer access or computer illiteracy may restrict participation. This is why in our study on harm reduction strategies we had to print out the priority list for the problem drug users. Both synchronous and asynchronous group discussions have the disadvantage that they lack nonverbal cues. A consequence of this is a heightened potential for misinterpretation of the written communication, which might negatively impact the group dynamic. Moreover, the lack of a real-time group dynamic has been criticized, especially in the case of focus groups since interaction is key (Tates et al., 2009). A disadvantage specifically linked to synchronous group discussion is that the discussion can be dominated by the fastest-typing participant (Stover & Goodman, 2012).
Developments and potential applications
Online group techniques are seldom applied in criminological research. An exception to this is the well-known study by Williams (2007) who conducted a virtual focus group with online community members to map the maturation of regulation within an online social setting. It is clear that the rise of (organised) cybercrime will only increase the necessity of the online applications of research techniques if we want to get into contact with respondents from this online community for research purposes. In view of the adaptability of NGT, its use in an online form could be applied in different research designs. Next, online applications are interesting to apply in criminological research on sensitive topics where participants are anonymous. It would allow the study of mixed groups, as we did in our study on harm reduction (with professionals from the police, justice department, treatment sector, local government and problem drug users all prioritizing the same ideas). Why not use an online NGT to study procedural justice, for instance, with both the police and offenders or victims and offenders participating in one NG meeting?
Taking the characteristics of NGT into account, NGT can be used in an exploratory (phase of a) study as well to generate hypotheses about topics that are relatively unfamiliar to the researcher or to uncover ideas that are of importance to a research population socially and culturally different from the researcher. It can also be used to generate content information about a topic unknown to the researcher (Gallagher et al., 1993).
Modifying and mixing methods
A NGT can be used simply as a technique to increase external validity. Although NGT is a qualitative research method, it is also possible to let a representative sample validate the results. This allows for subgroup analyses and the possibility of studying correlations. We want to stress the potential of the NG as a technique that merits further application in particular as part of a mixed-method research design, either as part of a design combining qualitative methods or as part of a design combining qualitative and quantitative methods. We concur with Boda and Szabo (2011) that qualitative and quantitative methods should be viewed as complementary rather than as rival camps given the strengths and weaknesses found in single-method designs (e.g. the lack of key contextual factors regarding why priorities exist in quantitative survey and the selection bias limiting the generalizability of results for purposive sampling, Weathers et al., 2011). Boda and Szabo (2011), p. 339 make a compelling argument for this in their study on the role of the media in shaping attitudes towards crime and the justice system. They state: “We should look for more sophisticated theories to explain the formation of public opinion and the role of the media in it, and, in addition to the quantitative analysis of statistical data, we should move towards audience research and the use of more qualitative methods”.
NGT can also be used as a standalone method. It is particularly relevant in applied research as a decision-making tool and as a consensus method. Its purpose is to gain insight into the problems or issues of importance and identify solutions for these problems and issues. For instance, it can be used to identify areas for hot-spot policing or to study alternative strategies for family violence or radicalisation. If the goal of the technique is to reach consensus, and consensus cannot be reached, the method’s group phases allow researchers to gain insight into differing perspectives, the elements of the topic where (no) consensus can be reached and why this is the case. As a collaborative process, which gives all participants a(n equal) voice, NGT holds great potential for actively involving vulnerable populations in gathering and prioritizing their ideas. It can be used to identify critical problem dimensions in hard-to-reach or hard-to-study populations. It was successfully used in health research where it has been applied to study different cultures (Martinez & Carter-Pokras, 2006; Malpede et al., 2007) and to study young people (Ginsburg et al., 2002). It can be used to involve multidisciplinary and multi-agency experts in particular to increase the likelihood of equal representation for all group members (Harvey & Holmes, 2012). This is the case when there is a (perceived) power disparity among participating experts, which is the case, for instance, for participants from health and social services and participants from the criminal justice system (Vander Laenen, 2014). As an established technique for citizen participation, it can be used to involve citizens in (action) research,research with local communities on prevention strategies for crime and disorder (Junger et al., 2012) or on legitimacy. The study by Wiatrowski and Campoverde (1996), who used NGT as a group problem-solving technique applied by community police officers to establish a relationship between themselves and community members and to assist the community in developing a plan of action to confront its problems, is a good example of this. Moreover, results from this type of research can offer guidance for criminal justice organizations as to how to best adapt strategies and training in a fast-changing and increasingly cosmopolitan world characterized by changing social values (Bottoms & Tankebe, 2012).
NGT is a highly structured technique combining characteristics of an individual survey and a focus group. Its structure limits researcher influence and influence from group dynamics. It increases the likelihood of equal participation for all group members and equal influence of (conflicting) values and ideas. NGT can be used in an exploratory (phase of a) study, can be used to generate hypotheses about topics which are relatively unfamiliar to the researcher, or to become familiar with the ideas found to be relevant to a research population that is socially and culturally different from the researcher. NGT is particularly relevant in applied research as a decision-making tool and as a consensus method. Its purpose is to gain insight into the problems or issues of importance and to identify solutions for these problems and issues. Since NGT limits the influence of group members with perceived high status, the technique is also valuable in research where participants have different levels of power.
Suggestions for further reading
Delbecq, A.L., Van de Ven, A.H., & Gustafson, D.H. (1975). Group techniques for program planning: a guide to nominal group technique and Delphi processes. Glenview, Illinois: Scott Foresman.
As the title suggests, it is a practical guide that explains in detail the NGT. With regard to the NGT, it includes a description of preparatory tasks, of the NGT process, its strengths and weaknesses and responses to frequently asked questions regarding the technique.
Merton, R.K., Fiske, M., & Kendall, P.L. (1990). The focussed interview, A manual of problems and procedures, Second Edition. New York: The Free Press.
The manual does not focus on NGT but it does provide vary useful information for researchers doing group interviews. In particular, it provides practical information on strategies for the moderator of a group to minimize undesirable forms of social interaction.
aA basic WoS search of the term ‘focus group’ resulted in more than 12,000 hits.
bNGT was developed in 1968 by Delbecq et al. and is derived from research in social psychology. NGT was originally developed as a technique to facilitate the involvement of disadvantaged citizens (Eisele, 2007).
cRead on for a detailed discussion.
dA basic WoS search of the term ‘Nominal Group Technique’ generated nearly 500 hits.
eA basic WoS search of the term ‘nominal group’ under the category “criminology and penology” resulted in only one hit: a study by Zastrow (1973).
fIn 2014 we used NGT as part of a three-stage mixed method design study to conduct a needs assessment study on harm reduction strategies in a local community (stage one is a needs assessment via interviews with multi-agency experts and focus groups with problem drug users; stage two is a two-part validation of the identified needs via a modified online NGT; stage three is a feasibility study via focus groups with professionals and drug users).
hThe road map consisted of a detailed description of the wording of the introduction, the goal and the procedure of the meeting, the central research question, the NG-format consisting of four phases, and the closing comments. For each element, the roles of the moderator and observer were outlined as well. In the observation schedule, the number and the characteristics of participants and the general atmosphere during the NG was registered. Next, for each phase of the NG the duration, questions asked by the participants, difficulties experienced by the participants, nonverbal participant interactions and interactions between the researchers and the participants were written down.
iIn total, the 14 NGT meetings resulted in 311 dif-ferent ideas. After discussing and erasing identical ideas, 290 ideas were retained. On average, a group generated 22 ideas. Participants wrote down an average of just under four ideas. To limit the researcher bias, the moderator and the observer independently coded the items. The initial inter-rater-agreement was 85% which was good (>80.0%) compared to other qualitative research. For the classification of the remaining items an agreement was reached between both coders.
- Aspinal, F, Hughes, R, Dunckley, M, & Addington-Hall, J. (2006). What is important to measure in the last months and weeks of life?: A modified nominal group study. International Journal of Nursing Studies, 43(4), 393–403.View ArticleGoogle Scholar
- Ayuso-Mateos, J, Avila, C, Anaya, C, Cieza, A, & Vieta, E. (2013). Development of the international classification of functioning, disability and health core sets for bipolar disorders: results of an international consensus process. Disability and Rehabilitation, 35(25), 2138–2146.View ArticleGoogle Scholar
- Balen, R, Blyth, E, Calabretto, H, Fraser, C, Horrocks, C, & Manby, M. (2006). Involving children in health and social research, ‘Human becomings’ or ‘active beings’? Childhood, A Journal of Child Research, 13, 29–48.View ArticleGoogle Scholar
- Beail, N, & Williams, K. (2014). Using qualitative methods in research with people who have intellectual disabilities. Journal of Applied Research in Intellectual Disabilities, 27(2), 85–96.View ArticleGoogle Scholar
- Boda, Z, & Szabo, G. (2011). The media and attitudes towards crime and the justice system: a qualitative approach. European Journal of Criminology, 8(4), 329–342.View ArticleGoogle Scholar
- Bottoms, A, & Tankebe, J. (2012). Beyond procedural justice: a dialogic approach to legitimacy in criminal justice. The Journal of Criminal Law and Criminology, 102, 119–170.Google Scholar
- Bristol, T, & Fern, EF. (2003). The effects of interaction on consumers’ attitudes in focus groups. Psychology and Marketing, 20(5), 433–454.View ArticleGoogle Scholar
- Campbell, J. (2002). A critical appraisal of participatory methods in development research. International Journal of Social Research Methodology, 5(1), 19–29.View ArticleGoogle Scholar
- Cancino, JM, & Enriquez, R. (2004). A qualitative analysis of officer peer retaliation - preserving the police culture. Poling, An international journal of police strategies and management, 27(3), 320–340.View ArticleGoogle Scholar
- D’Oosterlinck, F, Broekaert, E, De Wilde, J, Bockaert, LF, & Goethals, I. (2006). Characteristics and profile of boys and girls with emotional and behavioural disorders in Flanders mental health institutes: a quantitative study. Child: Care, Health and Development, 32(2), 213–234.Google Scholar
- de Ruyter, K. (1996). Focus versus nominal group interviews: a comparative analysis. Marketing Intelligence & Planning, 14(6), 44–50.View ArticleGoogle Scholar
- Delbecq, AL, Van de Ven, AH, & Gustafson, DH. (1975). Group techniques for program planning: a guide to nominal group technique and Delphi processes. Glenview, Illinois: Scott Foresman.Google Scholar
- Dening, K, Jones, L, & Sampson, E. (2012). Preferences for end-of-life care: a nominal group study of people with dementia and their family carers. Palliative Medicine, 27(5), 409–417.View ArticleGoogle Scholar
- Dirikx, A, Gelders, D, & Parmentier, S. (2012). Police-youth relationships: a qualitative analysis of Flemish adolescents’ attitudes toward the police. European Journal of Criminology, 9(2), 191–205.View ArticleGoogle Scholar
- Douglas, L, & Cuskelly, M. (2012). A focus group study of police officers’ recognition of individuals with intellectual disability. Psychiatry, Psychology and Law, 19(1), 35–44.View ArticleGoogle Scholar
- Drennan, V, Walters, K, Lenihan, P, Cohen, S, Myerson, S, & Iliffe, S. (2007). Priorities in identifying unmet need in older people attending general practice: a nominal group technique study. Family Practice, 24(5), 454–460.View ArticleGoogle Scholar
- Eisele, P. (2007). A field experiment comparing different workgroup interactive techniques. Perceptual and Motor Skills, 104(1), 171–178.View ArticleGoogle Scholar
- Fein, JA, Lavelle, JM, Ginsburg, KR, & Giadino, AP. (1997). A methodology to maximize resident input to improve a pediatric rotation. Archives of Pediatrics & Adolescent Medicine, 151(8), 840–844.View ArticleGoogle Scholar
- Frankel, S. (1987). NGT+MDS: An adaptation of the nominal group technique for ill-structured problems. The Journal of Applied Behavioural Science, 23(4), 543–551.View ArticleGoogle Scholar
- Gaber, J, & Gaber, S. (2002). Using focus and nominal group techniques for a better understanding of the transit disadvantaged needs. Transportation Planning and Technology, 25(2), 103–120.View ArticleGoogle Scholar
- Gallagher, M, Hares, T, Spencer, J, Bradshaw, C, & Webb, I. (1993). The Nominal Group Technique: a research tool for general practice? Family Practice, 10(1), 76–81.View ArticleGoogle Scholar
- Garcia, CA, & Lane, J. (2012). Dealing with the fall-out: identifying and addressing the role that relationship strain plays in the lives of girls in the juvenile justice system. Journal of Criminal Justice, 40(3), 259–267.View ArticleGoogle Scholar
- Gaskin, S. (2003). A guide to Nominal Group Technique (NGT) in focus-group research. Journal of Geography in Higher Education, 27(3), 341–347.View ArticleGoogle Scholar
- Ginsburg, KR, Menapace, AS, & Slap, GB. (1997). Factors affecting the decision to seek health care: the voice of adolescents. Pediatrics, 100(6), 922–930.View ArticleGoogle Scholar
- Ginsburg, KR, Alexander, PM, Hunt, J, Sullivan, M, Zhao, H, & Cnaan, A. (2002). Enhancing their likelihood for a positive future: the perspective of the inner-city youth. Pediatrics, 109(6), 1136–1443.View ArticleGoogle Scholar
- Government Accountability Office (GAO). (2007). Human Trafficking, Monitoring and evaluation of international projects are limited, but experts suggest improvement (p. 59). Washington: United States Government Accountability Office. Retrieved at www.gao.gov/new.items/d071034.pdf (15-06-2008).Google Scholar
- Harvey, N, & Holmes, C. (2012). Nominal group technique: an effective method for obtaining group consensus. International Journal of Nursing Practice, 18(2), 188–194.View ArticleGoogle Scholar
- Heary, CM, & Hennessy, E. (2002). The use of focus group interviews in pediatric health care research. Journal of Pediatric Psychology, 27(1), 47–57.View ArticleGoogle Scholar
- Jones, SC. (2004). Using the nominal group technique to select the most appropriate topics for postgraduate research students’ seminars. Journal of University Teaching and Learning Practice, 1(1), 20–34.Google Scholar
- Jordan, R.E. (1992). Policing Strategies That Address Community Needs in the 21st Century. Commission on Peace Officer Standards & Training: Rockville, USAGoogle Scholar
- Junger, M., Laycock, G., Hartel, P., & Ratcliffe, J. (2012). Crime science: editorial statement, Crime Science, 1(1); 3 pGoogle Scholar
- Kapiriri, L, Robbestad, B, & Norheim, OF. (2003). The relationship between prevention of mother to child transmission of HIV and stakeholder decision making in Uganda: implications for health policy. Health Policy, 66(2), 199–211.View ArticleGoogle Scholar
- Kartas, A, & Goode, S. (2012). Use, perceived deterrence and the role of software piracy in video game console adoption. Information Systems Frontiers, 14(2), 261–277.View ArticleGoogle Scholar
- Kaye, J, Winterdyk, J, & Quarterman, L. (2014). Beyond criminal justice: a case study of responding to human trafficking in Canada. Canadian Journal Of Criminology and Criminal Justice, 56(1), 22–48.View ArticleGoogle Scholar
- Keatinge, DR, Tarren-Sweeney, M, Vimpani, G, Hazel, P, & Callan, K. (2000). Identifying service needs of children with disruptive behaviour problems using a Nominal Group Technique. Nursing and Health Sciences, 2(4), 179–189.View ArticleGoogle Scholar
- Klukkert, A, Ohlemacher, T, & Feltes, T. (2009). Torn between two targets: German police officers talk about the use of force. Crime Law and Social Change, 52(2), 181–206.View ArticleGoogle Scholar
- Large, A, & Beheshti, J. (2001). Focus groups for children, do they work? The Canadian Journal of Information and Library Science, 26(2/3), 77–89.Google Scholar
- Lossius, H., Kruger, A., Ringdal, K., Sollid, S., & Lockey, D. (2013). Developing templates for uniform data documentation and reporting in critical care using a modified nominal group technique. Scandinavian journal of trauma resuscitation & emergency medicine, 21(80); 8 p.Google Scholar
- Macdougall, C, & Fudge, E. (2001). Pearls, pith, and provocation, planning and recruiting the sample for focus groups and in-depth interviews. Qualitative Health Research, 11(1), 117–126.View ArticleGoogle Scholar
- Macphail, A. (2001). Nominal group technique: a useful method for working with young people. British Education Research Journal, 27(2), 161–170.View ArticleGoogle Scholar
- Madriz, E. (2003). Focus groups in feminist research. In N Denzin & Y Lincoln (Eds.), Collecting and Interpreting Qualitative Materials (2nd ed., pp. 363–387). London: Sage.Google Scholar
- Malpede, CZ, Green, LF, Fitzpatrick, SL, Jefferson, WK, Shewchuk, RM, Baskin, ML, & Ard, JD. (2007). Racial influences associated with weight-related beliefs in African American and Caucasian women. Ethnicity & Disease, 17(1), 1–5.Google Scholar
- Martinez, IL, & Carter-Pokras, O. (2006). Assessing health concerns and barriers in a heterogeneous Latino community. Journal of Health Care for the Poor and Underserved, 17(4), 899–909.View ArticleGoogle Scholar
- Merton, RK, Fiske, M, & Kendall, PL. (1990). The focussed interview, a manual of problems and procedures (2nd ed.). New York: The Free Press.Google Scholar
- Paraskevas, A. (2013). Aligning strategy to threat: a baseline anti-terrorism strategy for hotels. International Journal of Contemporary Hospitality Management, 25(1), 140–162.View ArticleGoogle Scholar
- Porter, J. (2012). Methodological issues in collecting children’s views part 2: using nominal group technique to explore children’s views of the difficulties encountered in school. Social and Behavioral Sciences, 47, 1627–1631.Google Scholar
- Ramirez, JI, Gossett, DR, Ginsburg, KR, Taylor, SL, & Slap, GB. (2000). Preventing HIV transmission: the perspectives of inner-city Puerto Rican adolescents. Journal of Adolescent health, 26(4), 258–267.View ArticleGoogle Scholar
- Rich, M, & Ginsburg, R. (1999). The reason and rhyme of qualitative research: why, when, and how to use qualitative methods in the study of adolescent health. Journal of Adolescent Health, 25(6), 371–378.View ArticleGoogle Scholar
- Rosemann, M, & Vessey, I. (2008). Toward improving the relevance of information systems research to pracitce: the role of the applicability checks. MIS Quaterly, 32(1), 1–22.Google Scholar
- Ruefli, T, & Rogers, SJ. (2004). How do drug users define their progress in harm reduction programs? Qualitative research to develop user-generated outcomes. Harm Reduction Journal, 1(8), 13.Google Scholar
- Shedlin, MG, & Schreiber, JM. (1995). Using focus groups in drug abuse and HIV/AIDS research. In EY Lambert, RS Ashery, & RH Needle (Eds.), Qualitative Methods in drugs abuse and HIV research (Research Monograph 157) (pp. 136–145). Rockville: National Institute on Drug Abuse (NIDA), Division of Epidemiology and Prevention Research.Google Scholar
- Stewart, K, & Williams, M. (2005). Researching online populations: the use of online focus groups for social research. Qualitative Research, 5, 395–416.View ArticleGoogle Scholar
- Stover, C, & Goodman, L. (2012). The use of online synchronous focus groups in a sample of lesbian, gay, and bisexual college students. Computers, Informatics, Nursing, 30(8), 395–399.View ArticleGoogle Scholar
- Surratt, HL, Inciardi, JA, Kurtz, SP, & Kiley, MC. (2004). Sex work and drug use in a subculture of violence. Crime & Delinquency, 50(1), 43–59.View ArticleGoogle Scholar
- Tates, K, Zwaanswijk, M, Otten, R, Van Dulmen, S, Hoogerbrugge, P, Kamps, W, & Besning, M. (2009). Online focus groups as a tool to collect data in hard-to-include populations: examples from paediatric oncology. Medical Research Methodology, 9(15), 1–9.Google Scholar
- Thomas, C, Wootten, A, & Robinson, P. (2013). The experiences of gay and bisexual men diagnosed with prostate cancer: results from an online focus group. European Journal of Cancer Care, 22(4), 522–529.View ArticleGoogle Scholar
- Tuffrey-Wijne, I, Bernal, J, Butler, G, Hollins, S, & Curfs, L. (2007). Using Nominal Group Technique to investigate the views of people with intellectual disabilities on end-of-life care provision. Journal of Advanced Nursing, 58(1), 80–89.View ArticleGoogle Scholar
- Van De Ven, AH, & Delbecq, AL. (1974). The effectiveness of nominal, delphi, and interacting group decision making processes. The Academy of Management Journal, 17(4), 605–621.View ArticleGoogle Scholar
- Van Teijlingen, E, Pitchforth, E, Bishop, C, & Russell, E. (2006). Delphi method and nominal group technique in family planning and reproductive health research. Journal of Family Planning and Reproductive Health Care, 32(4), 249–252.View ArticleGoogle Scholar
- Vander Laenen, F. (2009). ‘I don’t trust you, you are going to tell’, Adolescents with emotional and behavioural disorders participating in qualitative research. Child: Care, Health & Development, 35(3), 323–329.Google Scholar
- Vander Laenen, F. (2011). How drug policy should (not) be: institutionalised young people’s analysis. International Journal of Drug Policy, 22(6), 491–497.View ArticleGoogle Scholar
- Vander Laenen, F. (2014). Whose claim is legitimate anyway? Negotiating power in inter-agency collaboration. In N Persak (Ed.), Legitimacy and trust in criminal law, policy and justice (pp. 111–129). United Kingdom: Ashgate publishing.Google Scholar
- Wachholz, S, & Miedema, B. (2000). Risk, fear, harm: Immigrant women’s perceptions of the “policing solution”' to woman abuse. Crime, Law and Social Change, 34(3), 301–317.View ArticleGoogle Scholar
- Weathers, B, Barg, F, Bowman, M, Briggs, V, Delmoor, E, Kumanyika, S, Johnson, J, Purnell, J, Rogers, R, & Halbert, C. (2011). Using a mixed-methods approach to identify health concerns in an African American community. American Journal of Public Health, 101(11), 2087–2092.View ArticleGoogle Scholar
- Weber, M. (2003). The definition of sociology and of social action: methodological considerations. In R Boudon, M Cherkaoui, & P Demeulenaere (Eds.), The European Tradition in Qualitative Research (Vol. III, pp. 363–380). London: Sage Publications.Google Scholar
- Wiatrowski, MD, & Campoverde, C. (1996). Community policing and community organization: assessment and consensus development strategies. Journal of Community Practice, 3(1), 1–18.View ArticleGoogle Scholar
- Wilkinson, S. (1999). Focus groups, a feminist method. Psychology of Women Quarterly, 23(2), 221–244.View ArticleGoogle Scholar
- Williams, M. (2007). Policing and cybersociety: the maturation of regulation within an online community. Policing and Society: An International Journal of Research and Policy, 17(1), 59–82.View ArticleGoogle Scholar
- Yu, J, Taverner, N, & Madden, K. (2011). Young people’s views on sharing health-related stories on the Internet. Health & Social Care in the Community, 19(3), 326–334.View ArticleGoogle Scholar
- Zastrow, C. (1973). Nominal group - new approach to designing programs for curbing delinquency. Canadian Journal of Criminology, 15(1), 109–117.Google Scholar
- Zinberg, NE. (1984). Drug, set and setting: the basis for controlled intoxicant use. New Haven - London: Yale University Press.Google Scholar
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.