@ExAniron

Aug 14, 2022

Taking advantage of Neocities’ backend for fun and profit

Let’s say that you want to add a “preview” of your webpage for social media. There are standards like OpenGraph and Twitter Cards where you can specify a preview of your page in hidden sections of your code. All nice and easy, but now you need to actually make the preview image, which can be annoying or time consuming. The lazy solution would be to make a screenshot of the webpage and use it as a preview of the webpage, but the process is still too long (i.e. how do you make a screenshot if you’re on mobile?). Maybe we can aoutomate it in our building pipeline using some external API, but they are limited in number of requests and you have to expose your private key while hoping of not hitting the rate limiting.

Neocities arrives on our rescue: if you check your site dashboard you will notice that it displays a nice preview of all your updated pages.

For example this is the screenshot of this webpage for this page:

Pretty neat, huh?

So, can we use this reliably to display a preview of our website?

Let’s see how the dashboard calls to this images:

https://neocities.org/site_screenshots/26/53/exaniron/posts/puppeteer/index.html.540x405.webp

Ok, we have our endpoint, followed by two random (?) numbers, our website subdomain, the relative path of our webpage and the size.

Can we use this directly? Let’s try with a different page:

https://neocities.org/site_screenshots/26/53/exaniron/about/index.html.540x405.webp

Ok, maybe we were lucky with the two random numbers. We have to be sure that we can use the URL without fear of pointing to a broken address. Let’s see how the Neocitie’s backend works. Luckily the infrastructure is open source so we can easily search trough its source code about the /site_screenshots endpoint.

In the end we can find the files that make it work: /workers/creenshot_worker.rb and models/site.rb.

The first one manages the scheduled workers that make the screenshots and manages them. The second one handles the display on the dashboard, including providing the URL. Our functions of interest are:

def self.sharding_dir(name)
    chksum = Zlib::crc32(name).to_s
    File.join(chksum[0..1], chksum[2..3])
end

def sharding_dir
    self.class.sharding_dir values[:username]
end

def base_screenshots_url(name=username)
    raise 'screenshots name missing' if name.nil? || name.empty?
    File.join SCREENSHOTS_URL_ROOT, self.class.sharding_dir(name), name
end

def screenshot_url(path, resolution)
    path[0] = '' if path[0] == '/'
    out = ''
    out = 'https://neocities.org' if ENV['RACK_ENV'] == 'development'
    out+"#{base_screenshots_url}/#{path}.#{resolution}.jpg"
end

As we can see the two “random” numbers are calculated from the CRC32 of the username so we can be sure that they won’t change during normal use.

Calculating the CRC32 of my username we have 2653996457 and that’s where our 26 and 53 come from.

Now we have everything we need to include this functionality in our codebase.

<meta property="og:image" content="{{ if .Resources.GetMatch "socialbanner.*" }}{{ (.Resources.GetMatch "socialbanner.*" ).Permalink }}{{else}}{{ printf "https://neocities.org/site_screenshots/26/53/exaniron%s/index.540x405.webp" ( .Permalink | relURL) }}{{ end }}" />
<meta name="twitter:image" content={{ if .Resources.GetMatch "socialbanner.*" }}{{ (.Resources.GetMatch "socialbanner.*" ).Permalink }}{{else}}{{ printf "https://neocities.org/site_screenshots/26/53/exaniron%s/index.540x405.webp" ( .Permalink | relURL) }}{{ end }} />

update January 2023

Neocities now uses webp instead of jpg: commit