Internet Research (PDF, 125 KB)
Prepared by the
The content and views expressed in this document are those of members of this committee, and do not necessarily reflect those of the Interagency Advisory Panel or Secretariat on Research Ethics.
The Panel and Secretariat welcome your comments: email@example.com
Following our assessment of ethics in Internet-based research, we propose that the Interagency Advisory Panel on Research Ethics integrate the following recommendations into its update of the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans.
In its 2004 consultation,1 the Social Sciences and Humanities Research Ethics Special Working Committee (SSHWC)2 identified the failure to consider ethical issues in Internet research as a significant gap in the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS). This document seeks to fill that gap.
Today we are witnessing the proliferation of new communications technologies that significantly affect the world of research. The Internet—a prime example—has many dimensions that can be studied (textual information sites, notices, pictures, archives, online videos, etc.) or used as research tools (email, discussion groups, chat rooms, online questionnaires, online intervention sites, etc). In the wake of these technological developments, numerous debates are underway on ethics in Internet research (American Association for the Advancement of Science, 1999; Association of Internet Researchers 2002; Eysenbach and Till, 2001; Ess and Jones, 2004; Robson and Robson, 2002; Buchanan, 2004; National Committee for Research Ethics in the Social Sciences and the Humanities, Norway, 2003; Sveningsson, 2004; Kitchin, 2003, 2007). This discussion paper focuses on research ethics in connection with the new communications technologies as they pertain to the ethical principles in the TCPS. This question was not raised directly in the TCPS. Kitchin (2003, 398) notes in this regard that the TCPS:
must establish its position, whether definitional, procedural, or both, on the ethics of conducting Internet/cyberspace research ….That cyberspace is being exploited for research is not in question. The question addressed here, rather, is whether related ethics research protocols are articulated or adequately inferred in the TCPS, and whether Canadian researchers are provided guidance by the TCPS when entering the realm of cyberspace for data collection and analysis.
These questions are therefore of direct concern to SSHWC, which proposes the 10 recommendations listed at the beginning of this paper.
Several bodies involved in research ethics have already addressed these issues, without necessarily placing them in a specific section of their codes of ethics. Three stand out:
Kitchin (2007, pp. 13 and following) identifies a typology of Internet-based research that specifies which projects are likely to require an ethical review by a research ethics board. She distinguishes between 1. web-based research (1a. non-intrusive and 1b. engaged) and 2. online research. Only categories 1b and 2 would require an ethics assessment.
Kitchin supports her theories with the works of Eysenbach and Till (2001), among others. She defines non-intrusive web-based research as research on the Internet that does not require the involvement of the researcher (it draws on text data or data from Web sites or discussion groups that the researcher observes). Data from NGs (next generations), listservs, chat rooms, IRC (internet relay chat), and MUDs (multi-user domain/dungeon) are therefore “comparable in terms of accessibility to a television or a newspaper article” (2003, p. 403).
Thus, this information is considered public and so would not require ethics review, as per the TCPS’s guidelines that ethics board review is not necessary for a study based on “publicly available information” (Section 1, Article 1.1c) or for research that “involves no interaction with the person who is the subject of the public records …” (1.1c, comment page 1.2) or “publicly available information or materials, including archival documents…” (Section 3, Article 3.1). To Kitchin, non-intrusive analysis strategies fall under the TCPS categories covering naturalistic observation or public activities, for which the paradigm of the human subject would not apply.
Using that perspective, we can suggest that sites containing text, to the extent that they are public sites, pose a minimum of ethical problems involving privacy and confidentiality of data. Material on the Internet that deals with public persons may be used unless its access is accompanied by specific clauses regarding dissemination. Do these same criteria apply to sites containing photos or videos? Copyright can be a factor in some cases, necessitating permission to reproduce the material.
Nevertheless, when it comes to chat rooms and discussion groups, other ethical issues arise that run counter to this analysis. Even if the researcher is not involved, these spaces are not necessarily public spaces, and may even be considered private spaces. Despite discussion by Kitchin (2007, pp. 60 and following) and others on the criteria for separating the two spheres, consensus has yet to be reached. Whitty (2004, p. 211) raises the following ethical issues regarding this subject:
Even if we conclude that these spaces are public spaces, the anonymity they afford can give the illusion that these are private spaces. Can we, as researchers, ethically take advantage of people’s false sense of privacy and security? Is it ethically justifiable to lurk in these sites and download material without the knowledge or consent of the individuals who inhabit these sites? … It is quite naïve of researchers to simply equate online media with what on first thought might appear to be offline equivalents (such as magazines and videos).
The AoIR (2000) requires that the assessment take into consideration the degree of sensitivity of the information. According to Bruckman (2002), any online information may be […] analyzed as long as 1. it is officially and publicly archived, 2. no password is required to access it, 3. no site regulations prohibit its analysis, and 4. the subject is not too sensitive.
Sveningsson (2004) uses the same public/sensitive criteria to identify several scenarios (p. 55):
However, the concept of sensitivity is not well defined and requires further discussion. For Eysenbach and Till (2001), individuals participating in discussion groups may view the intervention of researchers as an intrusion in their personal and private space. It would therefore be necessary to first determine whether that communication space is public or private. According to these authors, some steps can be taken to analyze the level of privacy: membership or registration required, the real or assumed number of users, and the presentation of standards and specific objectives that indicate the degree of privacy.
Given these ethical issues, should we maintain that all information found on the Internet is public, or should we subject that assessment to certain criteria (public–private continuum, sensitivity of the theme, etc.)?
For other Internet-based research projects (1. engaged web-based research, even with minimal involvement where the researchers participate in communications, and 2. online research [collecting information through interviews, discussion groups, surveys or questionnaires]), ethical assessment would be necessary, since these situations raise problems involving consent, anonymity, confidentiality and risk assessment.
The issue of free and informed consent, a central concern in the TCPS, is different for research involving the Internet, particularly in the case of chat rooms (if we consider them to be in the private domain or dealing with sensitive subjects), discussion groups, interviews, focus groups, online questionnaires and experimental research. Two positions emerge in the literature regarding the ethical issues involved in obtaining free and informed consent on the Internet. The first emphasizes the constraints inherent in this injunction, raises questions and proposes strategies. The second sees Internet consent as less of a problem.
For many researchers, obtaining consent in Internet-based research raises a number of problems. With regard to consent from respondents in chat rooms or discussion groups, Sveningsson (2001) indicates that, given the speed of discussions and the constant change in the individuals participating and the number of online participants, it is difficult and perhaps impossible to obtain consent online and from all participants. Possible mechanisms include announcing that the researcher is on line and attempting to assess participants’ reactions. Such strategies have the drawback of changing the observation context and therefore skewing the data. It is also possible to ask a site’s webmaster to post a banner indicating that a researcher is observing.
Bruckman (2002) proposes that researchers create their own chat rooms and provide information on the objectives of the research and obtain informed consent from participants. As Sveningsson (2001) indicates, this strategy also raises several questions: Would people join such a chat room? Who would sign up, and why? Would these people conduct themselves as they would normally? The issues raised by research conducted in discussion groups are similar to those raised with regard to chat rooms.
Moreover, researchers can use the Internet directly as a research tool by placing on a site a questionnaire with open or closed questions, using email as a means of dissemination or for conducting interviews, or conducting interviews on a discussion site—individually or in groups.
According to Porr and Ployhart (2004), obtaining informed consent constitutes a significant obstacle to transmitting information on a research topic. Without a face-to-face meeting with the participants, it is difficult to ensure that they understand instructions and are giving information voluntarily, particularly on a sensitive subject. That could call into question the validity of the informed consent. Secondly, the process of having the participant sign before a witness cannot be guaranteed, and it is difficult to prove that the subject thoroughly read the consent form. The issue of debriefing following research using deception also raises the question of how to ensure that the participant read this information. Some researchers therefore suggest using a multistage consent form, which requires the participant to check off something at each stage before moving on to the next stage of consent. Once the participant has gone through all the stages, he or she may access the research questionnaire. A similar process could be used for the debriefing. The subject should also be given the researcher’s coordinates (email and telephone) at both the consent and debriefing stages.
Use of an electronic signature could also partially resolve these constraints. Hill, King, Ecker-Denver, Gibson, Pankoff and Rice (2004) propose rules to consider in obtaining free and informed consent for Internet-based research. We highlight the following:
There is no unanimity regarding these complex procedures. As Kitchin points out (2003, p. 408), “Requiring informed consent for Internet research may be placing more stringent regulations on cyber researchers, vis-à-vis those who employ more traditional analyses.” Citing a critique from another author, she continues,
“Requiring all Internet researchers to acquire signed informed consent, or to be asked to demonstrate the real ages and competence level of subjects, especially in a minimal-to-zero–harm project, imposes regulations not required of alternative methodologies, and this … is unreasonable.”
In her most recent work (2007), however, she emphasizes the differences between non-intrusive research and research involving the participation of human subjects. For the former, she says, consent is less important; with the latter, researchers should follow the procedures in the TCPS.
The participation of minors in this type of research, including obtaining parental consent, also poses problems. As Bober (2004) mentioned, the researcher must ensure that the respondents are minors, and that the parental consent actually is that of the parents. Two positions emerge in the literature with regard to parental consent (see Bober, 2004). For some, contact with and access to minors in the context of Internet research should take place only through an intermediary, such as parents or the school. Others favour direct access to minors, with no parental consent, so that young people can respond freely to research questions and not be prevented from participating in the research by overprotective or concerned parents.
The fact remains, however, that it is difficult to verify the age and identity of the respondent, and that this may be an insurmountable obstacle in the current technological context. On this front, Binik, Kiesler and Mah (1999) propose the following strategies for research on sexuality, which may be extended to other fields:
Given the difficulty of authenticating participants’ identities and verifying their circumstances, a sensible strategy when research involves highly sensitive information or minors may be to verify the participant’s information with a telephone call and to cross-check information from the participant with information from other sources. In some institutions, local participants can be authenticated through their use of local accounts and passwords. Third-party authentication authorities provide a more general solution by serving as electronic notary publics and by requiring users to register with them. If the registration process is successful, then the authentication service will guarantee the authenticity of their client’s communications. Such procedures may become commonplace for business transactions, and, once they do, they can also be applied to research.
Other researchers believe that the problem of consent from minors should not be exaggerated. Walther (no date) stated:
Where questions of legal age may arise, two responses pertain. First, that there is nothing new about them. While the Report acknowledges that telephone surveys may rely on verbal consent, it does not consider that mail surveys and methods using other traditional media face the same issues with respect to the uncertain knowledge about whether people really are who they say they are. The question “Are you at least 18 years of age?” is a stock screener in many face-to-face research studies, as well as telephone surveys. Does the face-to-face encounter provide that much better a forum than any other media? One would think not….
Not much work has been done on the assessment of risks associated with Internet research, but some perspectives can be gleaned from reports and articles. In their report entitled “Ethical and legal aspects of human subjects research in cyberspace,” Frankel and Siang (1999, p. 3) of the American Association for the Advancement of Science say, “whether benefits and risks in online research are less or more than what occurs in the physical remains to be determined as research enters this new technological frontier.” For sensitive subjects, such as psycho-sexual themes, recourse to the Internet can pose problems to the extent that it would likely be difficult to help subjects deal with distress raised by the online interview or study.
On this front, Binik, Kiesler and Mah (1999) note, “The risk of distress is not unique to sex research on the Internet. We believe providing information about whom to contact with questions or concerns is just as easy on the Internet as in more traditional research settings. However, monitoring participants … is more difficult.”
As Whitty (2004, p. 216) points out, this problem can arise regarding access to adequate psychological services, particularly in the case of international research. She comments:
It is perhaps much more difficult to deal with psychological distress online and with individuals in other countries. It is difficult to provide names of psychologists in other countries available to counsel the participants. Nevertheless, it is imperative that we ensure that the participant does have counselling available to them, if the research has caused them distress.
The question of risk is also raised by Kitchin (2007, pp. 100 and following), who notes that the process for assessing minimal risk remains problematic in the TCPS, which she says does not define whether ethics boards are responsible for assessing risks and benefits or whether this is the prerogative of the researchers. As for the risks associated with Internet use, Kitchin explains that “when considering risk, for example, using publicly available materials non-intrusively would necessarily hold very different obligations vis-à-vis methods of research that directly involve speaker-writers, each of which carry innately different obligations of research ethics.”
This question therefore remains open, and the ethical responses will require further debate.
Quoting from archived material that meets the criteria of “public” does not pose any particular ethical problems. For material that does not meet these criteria, the ethical questions surrounding obtaining consent remain problematic and raise difficulties, such as identifying individual authors of texts or other materials. It could be suggested that anonymity and confidentiality remain in effect, unless the subject matter is inoffensive.
Use of material from discussions accessible through public chat rooms poses ethical questions involving anonymity and confidentiality. As Sieber suggested (2001), the ethical constraints differ depending on whether the chat room discussions are in the private or public domain, or are private discussions in a public space. Hence, if we consider information transmitted in chat rooms to be public, the ethical issues associated with confidentiality are not present. On the other hand, if we consider the information transmitted to be private, or private in a public space, the issue of confidentiality arises. The same issues arise with material collected through discussion groups.
The confidentiality and privacy of data gathered through online questionnaires constitute another problem (Porr and Ployhart, 2004). There are ways to determine the electronic origin of a message received (Internal Protocol address; internal cookies). Moreover, if the transmitted data are not carefully encrypted, they could be collected by unauthorized people, as Frankel and Siang note (1999, p. 7), “while survey research shares many characteristics of traditional survey research, it may increase the subjects’ risk of identity exposure, since subjects are transmitting the responses via the Internet and may not be aware of or sufficiently protected from the potential accessibility to their personal information by others.”
The problem of confidentiality arises in terms of the methods used to store data on a computer. Security measures can be thwarted if the computer is connected to the Internet. It is therefore necessary to propose countermeasures, such as closing cookies, inserting information regarding confidentiality measures on research sites, and using encryption when receiving and storing data. The TCPS should perhaps include recommendations regarding storage of data on computers.
As with other areas of research, ethical obligations can contradict the rule of law, as indicated in the SSHWC discussion paper “Reconsidering Privacy and Confidentiality in the TCPS.” Areas of the law that require clarification in terms of Internet research, as well as areas where the law diverges from or intersects with ethical obligations, need to be addressed. Sources of potential conflict between ethics and the law that the SSHWC paper identified can also be raised in the context of Internet research. To these we add the issues of copyright and intellectual property rights, as well as the use of photographic material.
The ethical issues associated with Internet-based research are many, and they should be taken into consideration in the new version of the TCPS. However, the importance accorded to them may vary along a continuum from minimalist to maximalist. Kitchin’s final remarks (2007, p. 102) with regard to the importance of Internet-related ethical issues reflect the minimalist position:
I see the current TCPS as already speaking—albeit implicitly—to various methods of research conducted with the aid of the Internet. Areas of the TCPS that do require specific attention to Internet research are those sections that address the meaning of “human subject,” the processes in risk and risk assessment, naturalistic observation, and the definition of and use of public materials. Insofar as we distinguish and subcategorize Web-based research and online research, future revisions need only be minor and should serve largely to clarify, rather than change, the spirit of the policy document.
The maximalist position, on the other hand, provides a more critical assessment, in the case of online research, of the issues associated with obtaining free and informed consent, reducing risk, and anonymity and confidentiality, thereby emphasizing the complexity of the criteria to apply in ensuring respect for respondents.
The ethical reflection and discussion should take place somewhere between these two poles.
American Association for the Advancement of Science. (1999). “Ethical and legal aspects of human subjects research in cyberspace.” www.aaas.org/spp/sfrl/projects/intres/main.htm.
Association of Internet Researchers (2002). Ethical decision-making and Internet research. www.aoir.org/reports/ethics.pdf
Binik, Y.M., Kiesler, S., and Mah, K. (1999). “Ethical issues in conducting sex research on the Internet.” The Journal of Sex Research, vol. 36: 82-90.
Birnbaum, M.H. (2004). “Human research and data collection via the Internet.” Annual Review of Psychology, vol. 55: 803-832.
Bober, M. (2004). “Virtual youth research: An exploration of methodologies and ethical dilemmas from a British perspective.” In Readings in virtual research ethics: Issues and controversies, E.A. Buchanan (ed.), pp. 288-315. London: Information Sciences.
Bruckman, A. (2002). “Ethical guidelines for research online.” http://www.cc.gatech.edu/~asb/ethics/
Buchanan, E.A. (2004). Readings in virtual research ethics: Issues and controversies. London: Information Sciences.
Ess, C., and Jones, S. (2004). “Ethical decision-making and Internet research: Recommendations from the AoIR ethics working committee.” In Readings in virtual research ethics: Issues and controversies, E.A. Buchanan (ed.), p. 27-44. London: Information Sciences.
Eysenbach, G., and Till, J.E. (2001). “Ethical issues in qualitative research on Internet communities.” British Medical Journal, vol. 323: 1103-1105.
Frankel, M.S., and Siang, S. (1999). “Ethical and legal aspects of human subjects research on the Internet.” Report of an AAAS workshop June 10-11, 1999. http://www.aaas.org/spp/dspp/sfrl/projects/intres/report.pdf.
Gosling S.D., Vazire S., Srivastava S., and John, O.P. (2004). “Should we trust web-based studies? A comparative analysis of six preconceptions about Internet questionnaires.” American Psychology, vol. 59(2): 93-104.
Hill, M.L. King, C.B. Ecker-Denver, C. Gibson, E., Pankoff, B. and Rice, T. (2004). "The Ethics of Online Research: Issues, Guidelines and Practical Solutions". Society for Prevention Research – Conference 2004, Poster #196. http://home.oise.utoronto.ca/~scottlab/colin.pdf
(2003). “The Tri-council Policy Statement and research in cyberspace: Research ethics, the Internet, and revising a living document.” Journal of Academic Ethics, vol. 1(4): 397-418.
Kitchin, H.A. (2007). Research ethics and the Internet: Negotiating Canada ’s Tri-Council Policy Statement. Halifax and Winnipeg: Fernwood.
Krishnamurthy, S. (2004). “The ethics of conducting e-mail surveys.” In Readings in virtual research ethics: Issues and controversies, E.A. Buchanan (ed.), p. 114-129. London: Information Sciences.
Krantz, J.H., and Dalal, R. (2000). “Validity of Web-based psychological research.” In Psychological Experiments on the Internet, M.H. Birnbaum (ed.), p. 35–60. San Diego: Academic Press.
National Committee for Research Ethics in the Social Sciences and the Humanities (NESH) (2003). “Research ethics guideline for Internet research.” www.etikkom.no/Engelsk/Publications/Publications/internet03_pdf/.
Porr, W.B., and Ployhart, R.E. (2004). “Organizational research over the Internet: Ethical challenges and opportunities.” In Readings in virtual research ethics: Issues and controversies, E.A. Buchanan (ed.), p. 130-155. London: Information Sciences.
Reips, U.-D. (2000). “The Web experiment method: Advantages, disadvantages, and solutions.” In Psychological Experiments on the Internet, M.H. Birnbaum (ed.), p. 89-117. San Diego: Academic Press.
Robson, K., and Robson, M. (2002). “Your place or mine? Ethics, the researcher and Internet.” In Ethical dilemmas in qualitative research, T. Welland and L. Pugsley (eds.), pp. 94-107. Burlington: Ashgate.
Sieber, J.E. (2001). “Privacy and Confidentiality: As Related to Human Research in Social and Behavioral Science.” In Ethical and Policy Issues in Research Involving Human Participants, National Bioethics Advisory Commission.Volume II, Commissioned Papers and Staff Analysis, Bethesda. http://bioethics.georgetown.edu/nbac/human/overvol2.html
Social Sciences and Humanities Research Ethics Special Working Committee (SSHWC)(2004).“Giving Voice to the Spectrum.”
SSHWC(2005). “Reconsidering privacy and confidentiality in the TCPS: A discussion paper.” Consultation paper. .
Sveningsson, M. (2001). “A case study for Ethics Working Group, Association of Internet Researchers.” www.cddc.vt.edu/aoir/ethics/private/Ethical.doc.htm.
Sveningsson, M. (2004). “Ethics in Internet ethnography.” In Readings in virtual research ethics: Issues and controversies, E.A. Buchanan (ed.), p. 45-61. London: Information Sciences.
Walther, J. (n.d.). “Research ethics in Internet-enabled research: Human subjects issues and methodological myopia.” www.nyu.edu/projects/nissenbaum/ethics_wal_full.html.
Whitty, M. (2004). “Peering into online bedroom windows: Considering the ethical implications of investigating Internet relationships and sexuality.” In Readings in virtual research ethics: Issues and controversies, E.A. Buchanan (ed.), p. 203-218. London: Information Sciences.
We felt it important to identify, for the purposes of Ethics Boards, the theoretical and methodological issues associated with Internet research.
As Norway’s National Committee for Research Ethics in the Social Sciences and the Humanities (2003, p. 1) has said, “Research on the Internet is valuable, not only because it can provide insight into a new and important communication channel, but also because the Net opens up the possibility to study known phenomena (for example, the formation of norms, diffusion of information, communication, and the formation of groups) in new ways.”
In an article dealing with the issue of ethics in Internet research, Kitchin (2003) related these issues to the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans by noting that two positions on Internet research can be identified among researchers. For some, the Internet does not differ significantly from traditional research methods. For others, the Internet offers “research opportunities not previously available. It is now possible to conduct interactive but virtual, ethnographies or participatory forms of research wherein cyber inhabitants or entire cyber/virtual communities become involved in the research project” (p. 401).
Among the scientific and methodological problems that Internet research raises, questions dealing with the validity of data and their representativeness have been the subject of several articles.
With regard to the use of online questionnaires, Reips (2000) singles out several advantages: access to large populations from diverse ethnic origins, increased statistical significance, reduced effect from the experimenter, and reduced costs. Porr and Ployhart (2004) report other advantages: less time spent distributing and collecting data, fewer errors in data collection because they don’t have to be transcribed, freedom of respondents to respond to the questionnaire when they wish, and financial savings through online administration of the questionnaire. Moreover, this strategy allows for quicker passage from data to results. The authors also suggest that respondents provided more honest answers through this method, though that honesty is not always ensured.
But there are also disadvantages. For example, we don’t always know who the participants are (the same individuals may respond to a questionnaire several times), the respondent dropout rate can be high, and interaction among participants is reduced. Porr and Ployhart (2004) raise methodological disadvantages, such as discrimination against people who do not have access to a computer or don’t know how to use one or who don’t feel comfortable using one.
According to the report of the American Association for the Advancement of Science (Frankel and Siang, 1999) and Birnbaum (2004), the methodological problems associated with using the Internet are many, involving the sampling techniques as well as the reliability and validity of the data collected. The authors also raise the question of bias involving the variables of type, ethnocultural origin and geographic distribution, as compared with samples collected through other procedures, and hence the possibility of collecting non-representative data. Another concern is the possibility that false respondents may answer the questionnaire, or that responses are false. It is also difficult to control the validity of strategic variables in online interviews.
In view of these limitations, Walther (no date) emphasizes that Internet users can be distinguished from non-users in terms of education and income, but he suggests that these differences tend to blur. As for the issue of generalizing the results to other populations, this is not unique to Internet research—it arises in other contexts. A variety of strategies are available to ensure good samples of people to respond to Internet questionnaires and thus obtain valid results. As for identity deception, Walther believes that this problem can also arise in non-Internet research. Furthermore, all indications suggest that this problem would not be very significant.
Krantz and Dalal’s (2000) analyses of psychological experiments on the Internet suggest that the data are just as valid as those from research conducted in a laboratory. Gosling, Vazire, Srivastava and John (2004) compare the myths and realities of Internet research and identify the following contrasts: Contrary to the myth about problems in the demographic representation of samples from the Internet, such samples are in fact diversified, although not completely representative of the population.
Contrary to the myth that samples obtained on the Internet are from maladjusted, socially isolated or depressed individuals, research shows that respondents recruited on the Internet do not present adaptation or depression profiles different from those of other populations. Nor does the myth seem to be true that the data collected vary based on the format of the questionnaires used.
Another myth suggests that participants in Internet research would not be motivated to take the study seriously and would not answer responsibly. In fact, it is possible to significantly motivate the participants. Furthermore, it is untrue that data would be compromised due to the anonymity of the respondents; on the contrary, anonymity would contribute to the reliability of the data, and it is possible to reduce the number of duplicates by reducing the motivation to respond more than once by targeting duplicates and by posing a direct question on the subject.
Lastly, it appears that data collected on the Internet do not differ from those obtained through other methods, but additional research is required to confirm that perspective.
In conclusion, it appears that research that uses the Internet to collect data through questionnaires does not present greater shortcomings in terms of validity and reliability than do other methods. From that conclusion, it is possible to assume that collecting other types of data would not lead to particular problems of representativeness and reliability. Thus, we can conclude that research projects using the Internet should be evaluated by means of the same criteria as other types of research.