Appendix G: Facebook Coding Process (Chapter 5)
Appendix G: Facebook Coding Process (Chapter 5)
Coding for the Facebook study of Chapter 5 had three elements: dutiful and actualizing coding, activities coding, and links coding. These processes, and corresponding reliabilities, are described on the two following pages; specific codebooks for the three are presented in Appendices G.1, G.2, and G.3, which follow.
Reliability for Coding of Dutiful and Actualizing Knowledge and Action in Status Updates
Two coders were trained to identify each type of communication, and a reliability test was conducted on a random 10% of each organization’s posts during March (181 status updates). This test revealed highly reliable measures of the communication types: for dutiful knowledge, the average pairwise percent agreement was 92.63%; for dutiful action, 87.85%; for actualizing knowledge, 95.95%; and for actualizing action, 99.26%.9 The complete dataset of status updates was then coded by the author and two assistants. One-third of each organization’s posts were coded by each of the three coders to ensure than any remaining unreliability was randomly distributed.
Reliability for Coding of Activities
As noted in Chapter 4, the cataloging of activities to be identified and coded was done in parallel for the two studies. This meant that essentially the same set of activities presented in Chapter 4 was assessed in the Facebook study.10 (Appendix H presents the details of how it was applied to Facebook.) A reliability check was conducted on those status updates that were coded as having dutiful action. Overall percent agreement for identification of presence/absence of each activity on each status update was 97.7%, between two coders.11
Reliability for Link Analysis Coding
The author and a graduate student assistant coded a subset of 184 (10%) of the 1,844 total status updates in the sample. In determining whether a link was internal or external, the reliability was 98.7% (Krippendorff’s alpha of .940). (p.208) The remaining sample was then coded by the author. In five cases (0.3% of sample) the link had expired and it was no longer possible to analyze it.
Appendix G.1: Codebook for Dutiful and Actualizing Coding
This codebook guides the analysis of Facebook status updates, including the text of notes and links visible with status updates on a Page’s wall.
What to Analyze
This codesheet will be applied to elements on an organization’s Facebook wall that meet certain criteria:
1. The text of status updates.
2. The text of notes and links visible on the wall, usually accompanying a status update.
Each piece of posted material should be analyzed as a package. Thus, if a status update contains a link, the whole package should be analyzed for the presence or absence of the four communication types.
Those four types are described in detail on the next page. They are:
• Organization-driven knowledge
• Organization-driven action
• Fan-driven knowledge
• Fan-driven action
For each post on a wall, each type of communication is either present or absent. The types are thus independent, and might occur in any combination (e.g., organization-driven knowledge and fan-driven action). It is possible but unlikely that all four will be present in a single post, and very unlikely that none of the four will be present in any post. Most posts probably will present one or two of the types.
For each post, record either a 1 or 0 for each type of communication (1=that form is present in this post; 0=that form is absent from this post).
Specific Coding Instructions and Guiding Notes
With organization-driven knowledge, the organization presents information that it wants fans to know. This may be general news relevant to the organization, or it may describe events or happenings within the context of the organization. The information does not need to be substantial or even true; it simply needs to include information that the organization wants users to know.
(p.209) • Because this is a very typical use of status updates and notes, most posts will include some organization-driven knowledge.
• NO: A message that describes an action that users are being asked to take, either implicitly or explicitly, should not be considered organization-driven knowledge, but organization-driven action (unless it is also accompanied by a separate element of the message that is itself knowledge). The key question is: Is there information here beyond the information that suggests action?
• NO: Similarly, information that explains how to do an action should not be considered knowledge unless there is information present beyond the context one would need to participate in the action.
With organization-driven action, the organization presents opportunities for action that it wants fans to take. These are any activities that fans are being asked to do.
• NO: Cases in which an organization asks fans to contribution information to their Page or website (e.g., “share your opinion with us!”) should not be considered organization-driven action—unless it involves structured processes, as below.
• NO: Cases in which an organization asks fans to learn more/read more/watch.
• YES: Cases in which an organization asks fans to contribute information to a web location that is not explicitly affiliated with the organization should be considered organization-driven action.
• YES: questions that ask fans if they have done or will do something should usually be coded as organization-driven action. (They’re asking you because they want you to do it.)
• YES: appeals to fans to participate in a structured organizational process, even if that process includes contributing information or opinion (for example, voting for something, nominating someone for a specific award, creating a video).
With fan-driven knowledge, the organization invites fans to contribute information or opinion, or invites them to participate in information exchange somewhere on its Page or website.
• YES: Cases in which an organization asks fans to contribute an opinion, or participate in a discussion that takes place on a Page or website affiliated with the organization should be considered fan-driven knowledge.
• NO: Rhetorical questions, such as when an organization opens with a question that clearly leads only to presentation of information, should not be considered fan-driven knowledge. (For example, from PugetSoundOff.org: “Do you take AP classes? Are they worth the pressure? This video features stories of students and teachers of AP classes and the pressures they face in our achievement-obsessed culture.”)
(p.210) • NO: appeals to fans to contribute information or opinion through a structured/institutionalized process (for example, voting for something, nominating someone for a specific award) are NOT fan-driven knowledge; rather, fan-driven knowledge occurs where an organization appeals to fans to share information/opinion in an open forum, such as the organization’s Page, or it’s website’s discussion board. It includes most appeals to share information/opinion in general (without a specific location or process).
With fan-driven action, the organization invites fans to suggest actions that others could take, or describe action that they themselves are taking.
Clarification of Difficult Cases
1. Rhetorical questions pose a challenge because sources sometimes present their information with a question that is not meant to be answered, but rather to stimulate interest in the topic. In coding status updates, it can be particularly unclear whether a question is meant to be answered when the question is accompanied by a link providing some kind of information related to the question.
2. In posts that have questions that are ambiguously rhetorical, err on the side of fan-driven knowledge. Sometimes, posts will include questions that are clearly rhetorical, with the organization clearly not looking for a response. But if it is ambiguous enough that you aren’t sure whether the organization is looking for an answer, assume that it is.
3. In posts that are ambiguous about the kind of information being solicited from fans—where it is hard to tell whether the organization is asking fans to submit knowledge or action—err on the side of fan-driven action. In such cases where it is unclear whether the thing to be contributed is knowledge or action, err on the side of coding it as action.
4. Some posts recommend an action to be taken in the form of a statement about the possibility of that action by supporters. These posts should be considered organization-driven action, and not organization-driven knowledge, unless there is also a separate piece of knowledge delivered with the action statement.
5. If an action could only be taken by a subset of group members (e.g., those over 18, women, Sam’s Club Members), it still counts as action.
Appendix G.2: Codebook for Activities Coding
Instructions for Assessing the Kind(s) of Activity Present in Status Updates
Read the text of the status update, and the text of any corresponding link that is visible with the status update. (Thus, in the Excel sheets, read the text in the “text” and “link/note text” columns.)
Assess the kinds of activities the organizations is encouraging supporters to do. Use the checklist of activities to decide what kinds of activities those are, (p.211) and enter the letter of those activities in the “Letter code” column on the right. Most status updates will only present one kind of activity; but where more than one kind of activity is presented, enter each letter, separated by commas.
Notes for clarification:
• If it is not clear whether the organization is asking supporters to take an action: When it is not clear that an action is present, or it is not clear that the organization is asking supporters to take an action that is described, assume that the action is being encouraged. Assume that when organizations are telling supporters about an action, they probably are asking/inviting them to take part in it.
• When NOT to follow a link: When the kind of action the organization wants supporters to take (that is, the action that is the purpose of the status update) is clear from the text of the update itself, it is NOT necessary to click on the link.
• Likewise, Links should also NOT be investigated where there is at least one clear action named in the status update—even if there are other, unspecified actions alluded to in the post. In that case, the specific action named in the post should be coded.
• When TO follow a link: Links should be investigated only where there is an action encouraged in a status update, but its nature is unclear, for example, when an organization suggests you urge an official to do an unspecified thing, or to help seals, but in an unspecified way.
• Example: For example, in one status update, Peta2 states, “Top five ways to help animals while poopin’!” In this example, there are clearly actions being encouraged by Peta2, but it is not clear what they are from the status update itself—supporters must click a link to find out what those five ways actually are. In this case, the link SHOULD be followed.
• However, if the post had read, “Don’t eat meat! And four other ways to help animals,” ONLY the “don’t eat meat” part of the action should be coded—and the link should NOT be followed. (Because there clearly are other ways to help animals, but it is the not eating meat that Peta2 is choosing to highlight.)
Appendix G.3: Codebook for Link Analysis Coding
For each status update, coders will examine up to one URL, which can be found in the link_dest field of the datasheet. If the organization has not used Facebook’s link feature (in which case the link_dest field will be N/A or blank), no link will be analyzed for that status update.
For each URL, two variables will be measured: first, the extent to which organizations link to themselves, or to information clearly about themselves, which will be referred to as internal versus external links; and second, the types of sites that are linked to (such as different kinds of websites, and social media applications).
(p.212) Instructions: Internal/External Link Coding
This coding explores the extent to which organizations use links in Facebook to direct supporters to more information about themselves (using internal links) or to a broad range of information from diverse sources and about diverse topics (using external links). In essence, the question is whether the page is being linked to because it is affiliated with the organization—its own page, or a page that talks about the organization, or, for example, an op-ed written by a director of the organization (internal)—or because it is new information the organization wants to notify supporters about (external).
1. Be aware of which organization posted each link. The organization’s name is available in column A, “Page_name.” Of course, it is necessary to know which organization posted the link to assess the relationship between the organization and the content of the link destination.
2. Look at URL itself. If the URL is clearly affiliated with the organization (for instance, by containing the organization’s domain, or including the organization’s name in its extension), it is an internal link. No further internal/external examination needs to take place.
3. If the URL is NOT clearly affiliated with the organization, it may still be a link associated with the organization; click the link to see the page. Look over the whole page to which the link directs, but do not click further into the website. (The one exception are Facebook links to external content, in which Facebook warns the user that they are leaving Facebook. Coders should click through that page to arrive at the true destination of the link.)
a. Scan the page for the organization’s name. If it appears, it is an internal link. If there is NO mention of the organization anywhere on the page, it is an external link.
b. Look particularly at the top and bottom of the page, where organizations sponsoring a page are often presented. If the organization’s name is present, it is an internal link.
c. If the page is a news article, scan the article for the organization’s name. If the organization’s name appears, it is an internal link.
d. If the page is a Facebook Page, look for the organization’s name. For Pages, determine whether the Page is the organization’s own Page, or one sponsored by the organization. For photos, determine whether the photos were posted by the organization.
4. Use the “Internal/External” column in the datasheet to record your determination. Use 1 to indicate that the link is an internal one; use 2 to indicate the link is an external one. For status updates that had not link, leave this field blank.
(p.213) 5. If the link directs to a page that is no longer available, or content within a site that has been removed (AND the URL itself does not make evident what sort of link it is), mark “Unavailable” in both coding columns.
Instructions: Type of Site Coding
This coding records the kinds of sites directed to by the links. It asks coders to catalog the social media affordances directed to, and to roughly characterize the kinds of websites directed to.
6. Whether the link destination is internal or external, use the appropriate column to record what kind of page is linked to (e.g., webpage, Facebook page). The other column should be left blank.
1. Facebook (or other social networking site; if other, indicate—e.g., “MySpace”)
2. YouTube (or other video sharing site; if other, indicate—e.g., “Vimeo”)
3. Flickr (or other photo sharing site; if other, indicate)
(a) Own website, affiliate website or website sponsored by organization
(b) Civic organization (use only if NOT own website)
Nonprofit civic, charity, or political organizations, including other organizations from the sample
(c) News organization (use only if NOT own website)
Recognizable news organizations, including well-known online sources (e.g., Washington Post, New York Times, Huffington Post, Talking Points Memo)
(d) Government agency (use only if NOT own website) (e.g., EPA, White House)
(e) Gorporate entity (use only if NOT own website) (e.g., Coca-Cola, Walmart)
(10) There was one addition that needed to be made: following from the discussion of organization-driven action and fan-driven knowledge above, one type of activity that occurred in status updates but was not observed in the website context was an invitation to participate in a contest or organization-led process that involved contributing some kind of idea or content. This type of content was specifically coded for in activities coding in the following way: in the initial coding process an identical codesheet to that used in the website study was employed, and cases of entering contests or otherwise participating in institutional activities were grouped with the “join an online group” activity from the website study. That category was then separately analyzed to separate cases of merely joining from those in which the joining in some way involved a supporter creating some kind of novel content, or otherwise contributing some kind of idea (for instance, contributing a name to a nomination process).
(11) Reliabilities when collapsing the individual actions into the three groups, as in Chapter 4, were similarly high: 89.7% for offline activities, 95.3% for dutiful activities, and 87.5% for networked/expressive activities. The process of bifurcating the one category that included some instances of supporter-produced content received 91.7% agreement.