UPDATE (February 4, 2024): This is the discussion about this project on HN: here. Please specifically read @dang's comment regarding the core assumption of this project: here. On a personal note, the number of Stories removed yesterday (Saturday, February 3, 2024) was the lowest ever recorded by the service. This includes 2 duplicate Stories. As a side note, in the list always check whether a Story is a duplicate or not: this is a very reasonable reason for removal and unfortunately I have no way of automatically determining it in the service!
The purpose of this project is to try to understand the type and scale of the moderation of the Hacker News Front Page.
NOTE: I love Hacker News. I try to read it every day. In the case of OnnxStream (here for example), 95% of the comments were helpful and intelligent. I also understand that moderating a site with huge traffic and where users are basically anonymous must be a very difficult task.
Returning to the purpose of this project, from what I have been able to see, the "public" (i.e. observable from the outside) moderation of the Front Page consists of two main tools: modification of the title of a Story (voluntarily or involuntarily influencing its growth in terms of rank) or directly its removal.
Regarding the first type of moderation, an excellent site is already available that tracks changes to Story titles. Here instead I will focus on the second type.
For the reasons explained in the "Why?" section below, I have developed a small application that logs all the Stories that are removed from the Front Page, for personal use. I later discovered that there is no tool/website that provides this type of information and I decided to make it public here. It was a difficult decision but my rationale is: is it better to have more transparency or less transparency?
If you know of a tool/website similar to this, please let me know: I will archive this repo or set it to private.
A possible very positive outcome for this project could be to have a list similar to this, but available directly among the HN lists. Or even to notify a user when a Story is penalized on the Front Page, perhaps indicating the number of flags and/or the reason, for example.
Feel free to skip this part or click to expand
A friend of mine posted two Stories on Hacker News related to OnnxStream (31 days apart), the first related to SDXL Turbo support and the second related to TinyLlama and Mistral 7B support.
In the case of the first, the Story was among the first on the Front Page, until its title was changed from "Stable Diffusion Turbo on a Raspberry Pi Zero 2 generates an image in 29 minutes" to "OnnxStream: Stable Diffusion XL 1.0 Base on a Raspberry Pi Zero 2". This effectively "killed" the Story. One user pointed out that the new title didn't reflect the spirit of the Story (thanks @practice9).
In the case of the second, the Story was in third place on the Front Page, less than an hour after the submission. In this case it was simply removed from the Front Page.
Having discovered this, perplexed, I sent an email to the moderator. @dang, who was very kind and quick in his response, explained to me that the Story had been flagged by users even without being explicitly [flagged], and that he could therefore only hypothesize the causes of the flag. His hypothesis was that (some?) users might be fed up with news related to LLMs.
While I have no reason to doubt Daniel's good faith, it's hard to believe that HN users would be tired of LLM-related news.
So I decided to develop a small console application to determine the frequency of this phenomenon (actually I was also motivated by the prospect of writing some C# code, after more than 2 years of complete abstinence). I subsequently discovered that there were no tools/websites that monitored this specific phenomenon and I therefore decided to make it public here.
Using the official HN API, the service fetches 90 Top Stories every minute and makes a comparison with the first 30 Top Stories (i.e. the Front Page) fetched the previous minute. It logs all missing Stories here. The assumption is that a Story cannot go from the top 30 to a position greater than 90 in a single minute, without having been explicitly removed. If a Story reappears on the Front Page, it is removed from this log. All Stories present in the second-chance pool are excluded from the log. Title and URL are those from when the Story first appeared in the top 30. The number of points and comments and the rank are those from when the Story was removed from the Front Page. The ID points to the news.social-protocols.org page for that Story, which provides a graph of the Story's position on the Front Page over time.
NOTE: always check whether a Story is a duplicate or not: this is a very reasonable reason for removal and unfortunately I have no way of automatically determining it in the service!
- 42519202 #27 8 points 6 comments -> OpenAI is Lehman Brothers: A crash is coming
- 42519913 #5 5 points 0 comments -> Blazed – Rust-based 3D renderer
- 42521914 #2 61 points 18 comments -> Denmark Offers to Buy USA (2019)
- 42521782 #18 7 points 2 comments -> From Pen and Paper to an AI Factory: McLaren's F1 Reinvention
- 42521545 #9 -> Supernovae evidence for foundational change to cosmological models
- 42523491 #4 -> Show HN: DeepSeek v3 – A 671B parameter AI Language Model
- 42521735 #26 23 points 7 comments -> An Evidence-Based Approach to Goal Setting and Behavior Change
- 42523590 #3 8 points 1 comments -> Induced Demand Debunked
- 42523934 #25 49 points 34 comments -> Matt Mullenweg Asks What Drama to Create in 2025, Community Reacts
- 42526150 #21 12 points 10 comments -> Show HN: NeatShift – Organize Windows Files with Symbolic Links
- 42507395 #14 5 points 2 comments -> Clap to Find My Phone App
- 42529207 #12 50 points 23 comments -> LLMs are everything that it wrong in computing
- 42530293 #3 9 points 7 comments -> Casual Viewing – Why Netflix looks like that
- 42529396 #17 29 points 3 comments -> Ten things in tech I found joy in during 2024
- 42506623 #16 90 points 26 comments -> PixLens: A lens that turns the scenery you see into pixel art
- 42532156 #17 13 points 8 comments -> Costco Denies Shareholder Demand to Back Down from DEI Hiring
- 42532429 #4 40 points 19 comments -> Missile Strike Confirmed as Cause of Azerbaijan E190 Crash – Aeroxplorer.com
- 42532353 #30 11 points 6 comments -> Per Seat Pricing Sucks
- 42536342 #23 10 points 3 comments -> Dr. Paul Releases 2024 'Festivus' Report on Government Waste
- 42537790 #4 4 points 6 comments -> Show HN: I built a website that explains any difficult concept in seconds
- 42537829 #12 30 points 16 comments -> America in Two Headlines
- 42538458 #26 9 points 5 comments -> SQLite3: No Room for Drama
- 42539398 #29 23 points 7 comments -> McKinsey Destroyed the Middle Class
- 42539673 #7 5 points 0 comments -> Year 2024 ends on bad note for Boeing with twin 737-800 disasters on same day
- 42540870 #14 8 points 0 comments -> Pirate Party – a political party in Iceland (and in many other countries)
- 42540850 #4 4 points 0 comments -> The BBC's Civil War over Gaza
- 42541536 #25 24 points 7 comments -> Enslaved on OnlyFans: Women describe lives of torment and sexual servitude
- 42540427 #30 204 points 161 comments -> China turns members of its diaspora into spies
- 42542215 #4 5 points 1 comments -> Epoll Is Fundamentally Broken
- 42544923 #4 11 points 3 comments -> What ails America and how to fix it? (Jeffrey Sachs)
- 42545772 #15 46 points 10 comments -> Israel built an 'AI factory' for war. It unleashed it in Gaza
- 42550367 #24 3 points 0 comments -> Show HN: Learn Japanese Kanji, no silly mnemonics
- 42552741 #9 3 points 0 comments -> Enjoy Apple TV+ for free the first weekend of 2025
- 42554088 #8 4 points 1 comments -> Is a Dosa a Crepe?
- 42519168 #18 100 points 186 comments -> Where is James Bond? Trapped in an ugly stalemate with Amazon
- 42555040 #6 5 points 0 comments -> Former Meta Recruiter Reveals Why You're Struggling to Land a Big Tech Offer
- 42555590 #17 3 points 2 comments -> Show HN: I Made a Dumb Game
- 42530620 #26 21 points 28 comments -> Monokai Pro for Jetbrains
- 42547367 #15 11 points 4 comments -> Letting Language Models Write My Website
- 42534100 #24 -> The most stunning portrait that I have ever seen
- 42559048 #2 20 points 40 comments -> Show HN: Obsess Jobs – Apply to jobs in your sleep
- 42559598 #2 10 points 9 comments -> Apple TV+ free first weekend of 2025
- 42558142 #27 19 points 7 comments -> The British Empire Was Much Worse Than You Realize
- 42560933 #5 17 points 9 comments -> Show HN: I made a tiny game that helps increase IQ
- 42565524 #19 9 points 3 comments -> At least 10 killed after vehicle hits crowd of people in New Orleans
- 42566387 #21 32 points 40 comments -> A Legendary Exposé of the Brutality of the Soviet Union
- 42565442 #23 15 points 4 comments -> Should you do a newspaper workout?
- 42571045 #12 23 points 40 comments -> Could the Soviet Union have survived?
- 42571569 #19 29 points 15 comments -> Spirituality Is Secure Attachment with Reality
- 42573214 #8 6 points 2 comments -> Show HN: A tool to generate dragon names
- 42569236 #25 320 points 184 comments -> Rails for everything
- 42573936 #28 6 points 1 comments -> Pink Floyd's Young Lust telephone signalling explained
- 42542576 #22 36 points 54 comments -> Where are Mr. Beast's early sponsors now?
- 42540005 #17 95 points 21 comments -> DuPont and "DuPont" connectors, and how to crimp them properly
- 42574420 #26 3 points 0 comments -> Things we think will happen in 2025
- 42570138 #28 106 points 80 comments -> Are there more trees on Earth than stars in the Milky Way? (2016)
- 42574496 #25 8 points 0 comments -> What It's Like to Work for Elon Musk - Genius, Chaos, and Burnout
- 42574238 #26 9 points 3 comments -> US culture breeds 'laziness' and 'mediocrity
- 42553657 #13 5 points 1 comments -> The Upside of the Pentium Bug (1995) [pdf]
- 42569728 #23 73 points 2 comments -> (Amiga) AROS Research Operating System: Summary of 2024
- 42574545 #26 5 points 0 comments -> Korean Assembly Speaker's adherence to procedure key to making troops stand down
- 42571995 #29 117 points 64 comments -> Autodesk deletes old forum posts suddenly
- 42571963 #28 25 points 9 comments -> How I Use Claude
- 42574352 #30 6 points 4 comments -> Nearly Half of US Unicorns Have Immigrants Founders
- 42539700 #26 132 points 49 comments -> We fine-tuned Llama and got 4.2x Sonnet 3.5 accuracy for code generation
- 42540792 #28 72 points 28 comments -> What is a second?
- 42576141 #16 6 points 1 comments -> Employees Speak Out: Google, Apple, MS, & Intel Discriminate Against White Men [video]
- 42576001 #23 6 points 1 comments -> Palestinian Authority Suspends Al Jazeera TV in West Bank, Citing Incitement
- 42576090 #27 8 points 0 comments -> The Dogs of (Urban) War: Lessons from the IDFs' Specialized Canine Unit