Creating a solution for paywalled HN submissions

November 29, 2022 • Side Projects • User Scripts

Caption: A detailed image of a hacker intently working on their laptop in front of a border wall during twilight, taken with a DSLR camera, using a wide-angle lens with dramatic cinematic lighting, the walls have political slogans and art spray painted on them —v 5 —ar 16:9

This userscript adds archive URLs to the metadata section of Hacker News(HN) (https://news.ycombinator.com) submissions without breaking the immersion. Here are 2 screenshots: https://imgur.com/a/PdUu6oG

GreasyFork: https://greasyfork.org/en/scripts/452024-hacker-news-anti-paywall
The actual code: https://github.com/MostlyEmre/hn-anti-paywall

Now let me overexplain.

• Why this?
• First try
• Hacker News Paywall Archiver Userscript

Why this?

I never liked paywalled articles. I understand where they come from, but I don’t like where we cross our paths.

This is why I don’t use major news aggregators anymore. Instead, I spend my “catching-up-with-the-world-time” on Hacker News. However, Hacker News (HN) also has its fair-share of paywalled articles. (Around 11.6% according to my short-lived, half-assed attempt at measuring it. See my super old data https://hpa.emre.ca/ I tell the story below.)

I even saw an upvoted comment suggesting a feature so that the users can read the article without paywall. These paywalled link submissions are sometimes coupled with an https://archive.is(archive.today) URL in their comment sections. The reason for this is to “archive” but remove the paywall for the shared URL and let the users read the article without any barriers as archiving services like Archive.org and Archive.today remove the paywalls while archiving to preserve the information for future generations.

First try

Around a year ago, when I ran the above experiment, my goal wasn’t to run that experiment. It was during my self-teaching & career-changing process, I decided to build a HN clone. To make it stand out from the bunch, I added a paywall feature. It would detect paywalled articles and would add an archive URL into the metadata.

The issue with archiving is unless someone archived the link before on the {archiving-project} then the link is most likely not archived. So me sending people to those projects meant nothing. It actually meant something for me from an ideological stand-point but you are not me.

This rubbed me in the wrong way. I decided to build a backend (See https://github.com/MostlyEmre/HN-Paywall-Archiver) that would scan the links and automatically submit paywalled ones to archive.is for archival. I used Nodejs, Firebase, and React. I was -still am- really proud because I believed it was doing public good in terms of digital preservation. Only 1 person needed to run this script to benefit everyone. I was curious on how many paywalled articles were being shared, by whom, at what time. So I also created some analytics functionality to gather the data. And then created a UI to present the data. Now heads up, the above repo is from ancient times when I started teaching myself how to code.

HN-Paywall-Archiver was great but I stopped running the backend at some point. Because at that point couldn’t find a way to continuously run my backend code on some platform for cheap or didn’t try hard enough.

P.S. Recently I’ve been thinking of remaking this version with Cloudflare Workers as I’m in love with the Workers concept and want to improve my know-how.

Hacker News Paywall Archiver Userscript

After almost a year, I got into userscripts. Super great super awesome concept. People seem to hate javascript unless it is presented as a userscript. So I decided to get my hands dirty to create a simple solution that solves the paywall issue on HN without breaking any hearts.

My solution is not perfect as it had to be simple. But here’s the rundown.

• Pros:

  • Not intrusive.
  • Simple code, simple concept.
  • Unintentionally indicates which submissions are paywalled.
  • Not-yet-archived archive links can make you feel like you are contributing to the society after you click on the “archive this URL” button on project page.
  • Uses the HN html defaults, so I hope it plays well with the HN skins/plugins/userscripts you use.

• Cons:

  • It doesn’t automatically archive the links.
  • It uses a static list of paywalled websites sourced from a popular Chrome extension. So changing the paywall list is slow and manual.
  • No guarantees of archived links actually having the archive readily available for reading. Though there are currently 3 projects added, so it should be enough for most links.

So, there you go. I hope you enjoy it. It can break occasionally due to changes in news.ycombinator code, if you let me know on Twitter, I can fix it ASAP. Otherwise you have to wait until I notice that the script is broken, which can take quite a while as I browse HN on mobile.