Grant: Discovery Grants Program - Individual

Risk can be quantified for operational assessment as the product of probability and consequence. In some settings, this is considered a triplet that defines risk, and the third element is an index by scenario. Social media is characterized by a constant flow of constantly updated scenarios that all have the possibility of being toxic. This presents a challenge to social media platforms as they must flag and limit the spread of problematic content, including toxic speech, or risk alienating users and thereby lose out on user contact time and thereby advertising revenue or revenue from other services. This high volume of interactions and large variability among posts presents a complex decision context for the firm, as moderation processes must balance conflicting trade-offs that impact technical aspects of operation as well as its profitability in the short and long-term. Moderation can take the form of humans (either as employees or users, pre- or post-posting) as well as algorithms that detect specific types of language and interactions. These machine learning processes again rely on human annotators to allow the code to flag negative events. The specific levels of investment required in algorithmic tools, human resources and processing will depend on the toxicity context in which the platform user base operates, along with the level of coordination among users. Finally, the bias and variability of the algorithmic tools deployed can impact time until approval, and could cause misidentification of toxicity (as false positives, or type 1 error, and false negatives, or type 2 error). Platforms need decision support to be able to navigate these questions. The proposed research seeks to support his need by helping to quantifying the costs and benefits of toxic speech detection and management for the platform. Using an agent-based model to simulate platform response and estimate processing time, computing cost, human intervention in terms of data annotation and flagging, and the impact on user base as well as advertising revenue sources of different toxicity detection approaches. This simulation will allow us to identify the combination of tools and moderation policies that can best support the platform in their particular social media environment. Thus, the proposed research will test existing algorithmic approaches to identifying optimal tools for detection of toxic speech. Our results will allow us to iteratively improve and develop new tools. Algorithmic approaches must also exist with policies and processes within the firm. We will use these to quantify costs and revenue as well as benefits. Exploring these costs and benefits must also include quantifying the social costs, including the cost of annotation on humans (mental health impacts), as well as the reliance on underpaid gig work. These human costs must be considered alongside technical considerations like processing time and the advertising revenue raised from users.