Utilizing Machine Learning to Catch Social Media Scammers

September 05, 2024

Social media has become the most common place for people to fall victim to scams. That’s why this year, Cash App rolled out a machine learning-powered tool designed to help customers avoid interactions with potential scammers. We achieve this by identifying and suppressing scam posts that interact with Cash App-related conversations related to our branded posts or social media accounts.

Cash App employs a suite of tools designed to protect our customers from fraud before they ever attempt to send a transaction (for example, our Scam Warnings tool we recently shared). Beyond our own platforms, we also take proactive action to limit scammers’ interactions with customers on our social media profiles. We call this our “Social Media Guardian.” Here’s how it works:

• We employ a third-party social media monitoring service to identify and assess Cash App related conversations.
• These conversations are ingested into an internal machine learning system that we purpose-built to evaluate whether a post is a scam. If the model is unsure, the post is reviewed manually by a member of our social experience optimization team.
• For our modeling approach, we used BERTweet, which is a public language model pre-trained on X/Twitter data, since our first use case was to detect scams on Twitter. We further tuned BERTweet on our own labeled X dataset to improve the performance.
• When posts replying to our branded posts are identified as scams, they are hidden and the poster is permanently blocked from interacting with our social media accounts.

Using this model, we’ve been able to identify and suppress with a high degree of accuracy more than 2,000 posts weekly across X and Instagram. That's in addition to the nearly 1,000 posts, on average, we report to X and Instagram each month for terms of service violations and brand impersonation, identified and submitted at scale using an AI-based detection platform. Financial platforms are often the last stop in the lifecycle of a scam, which is why we know it’s crucial that we remain diligent and invest in tools that protect our customers within Cash App and consumers at large.