Handling Bad User Input

When you let people be responsible for their own inputs, you inevitably get some... interesting inputs.

NSFW content and "bad" words are probably the most common examples, so let's go over how this site handles them!

The problem

Text inputs risk containing profanity, and drawings risk containing, well, absolutely anything. Giving users so much freedom also requires a lot of behind-the-scenes logic to make sure that freedom is used responsibly.

And obviously, for a site built by a SWE for an audience of other SWEs, we also have to handle malicious inputs :3

Text

For name inputs, I simply check the name against the bad-words package, a JavaScript package that filters out bad words for you!

Drawing

Empty case: Disable submission when the canvas is empty.

NSFW case: During uploading, the backend runs NSFWJS to classify the drawing. This open-source model predicts the likelihood of explicit content across five different categories. The content warning triggers if any one of the categories exceeds 30%.

I originally had this check happen client side, which looks like loading in TensorFlow and the model in after uploading. This posed some major issues though:

• Bad UX from the browser freeze up during load in, there's honestly no way around this client side
• Users can bypass this check by calling the upload endpoint directly

Malicious case: Luckily, displaying images (as long as their content isn't inappropriate) is quite safe! Both the frontend and backend accept only image file types :)