Blog

In this week's experiment I'll be diving into the Hilbert curve algorithm following along this video from The Coding Train.

What is the Hilbert Curve #

According to Wikipedia:

[...] a continuous fractal space-filling curve.

In simpler terms, it's a mathematical curve that can theoretically fill an entire two-dimensional area by following a precise recursive pattern.

Each iteration (called an order) increases the resolution, bending the line tighter and tighter until it nearly becomes solid.

Hilbert curves show up in computer science, image compression, and even dithering algorithms for grayscale rendering.

How It Works #

The sketch builds the curve point by point.

Each position in the sequence is converted into an (x, y) coordinate using a Hilbert index function: a mapping from a single integer to a two-dimensional location.

For every frame:

• The curve progresses one segment forward.
• Each segment's color is smoothly interpolated along a palette, creating a gradient through the path.
• Once the final point is reached, the animation halts, revealing the entire pattern.

The algorithm scales with order so higher orders produce increasingly dense, woven paths that almost resemble clothing fibers under a microscope.

Final result # References #

• Coding the Hilbert Curve by The Coding Train.
• One file JS program that draws Hilbert curve iteratively by Marcin Chwedczuk.
• Hilbert's Curve: Is infinite math useful? by 3Blue1Brown.

Outro #

This one is less about randomness and more about rhythm.

After experimenting with procedural generation for the past few weeks, it's a breath of fresh air to work with something deterministic and geometric.

After a short break, this week I’m exploring the Wave Function Collapse algorithm, following this video by The Coding Train.

What Is Wave Function Collapse #

According to Wikipedia:

[...] a family of constraint-solving algorithms commonly used in procedural generation, especially in the video game industry.

In simpler terms, it’s a system that generates patterns or terrains based on rules: it decides what can go where, making sure every new tile fits the constraints of its neighbors.

It’s often used in procedural world generation for games, where randomness needs to feel intentional.

How It Works #

The sketch creates a grid of cells across the canvas and assigns a tile to each one.

Every time a new tile is placed, it checks the constraints of its neighbors and narrows down the possible options. If more than one tile fits, it picks one at random while balancing order and chance.

What emerges is a patchwork of tiles that looks structured yet organic, as if the design grew naturally from a set of invisible rules.

Final Result #

Here’s the sketch in motion. Watch how the grid resolves itself into a coherent pattern.

References #

• Coding Challenge 171: Wave Function Collapse by The Coding Train.
• Wave Function Collapse algorithm by Maxim Gumin.
• Model Synthesis by Paul Merrell.
• “Wave Function Collapse” in Processing at The Processing Foundation.

Outro #

Another week down, and another step deeper into procedural generation following last time’s Marching Squares.

This one feels especially rewarding: seeing a blank grid gradually resolve into structure feels almost surreal. It’s easy to understand why so many of my favorite games use this kind of system: it’s order emerging from possibility.

This week's experiment dives into the marching squares algorithm, following this video from The Coding Train.

What are Marching Squares #

According to Wikipedia:

[...] an algorithm that generates contours for a two-dimensional scalar field (rectangular array of individual numerical values)

In simpler terms, it’s a way to turn a field of numbers into shapes.
Its 3D cousin, marching cubes, is often used in game development to build procedural worlds — think Minecraft or Terraria.

If you're curious about how that looks in practice, Sebastian Lague’s video is a great deep dive into generating landscapes on the fly.

Notes #

The script builds a grid of cells across the canvas like a chessboard, and assigns each corner a value based on a noise function.

Each square is then evaluated: depending on which corners are above or below a threshold, a contour line is drawn through it. The result is a fluid, organic network of lines that seem to outline invisible terrain.

It's a surprisingly compact algorithm for what it does. With just a few loops and a bit of logic, a world starts to take shape from numbers alone.

Final result # References #

• Coding Marching Squares by The Coding Train.
• open-simplex-noise-js by Josh Forisha.

How It Works #

Each frame begins with a grid of noise values, generated using OpenSimplex noise. These values act like a topographic map: higher numbers mean "solid ground", lower ones mean "empty space".

The marching squares algorithm checks every cell in that grid and compares them to a threshold. Depending on which corners are above or below that threshold, the cell is assigned one of sixteen possible configurations. Each configuration defines how the contour line should pass through it.

The program then connects these line segments across the grid, forming smooth boundaries that outline the noise field.
The result is a shifting pattern that feels like terrain depending on how the parameters are tuned.

Because the grid regenerates every frame, the shapes flow and evolve in real time, making the sketch feel less like static geometry and more like a moving landscape drawn by the code itself.

Outro #

Another week down and this one feels like a small step toward procedural generation.

There's something satisfying about watching a blank canvas slowly fill with shape and structure, as if the algorithm itself is discovering the terrain one line at a time.

This week starts my first experiment with p5.js. The subject for today is the paper marbling algorithm, based in the implementation explained in this video by The Coding Train.

What is the Paper Marbling Algorithm #

Paper marbling, as defined by Wikipedia, is:

[...] a method of aqueous surface design, which can produce patterns similar to smooth marble or other kinds of stone.

In this context, the algorithm is a way to recreate that effect digitally: a simulation of ink swirling across a virtual surface.

Notes #

To bring that idea into code, the script simulates the process of dropping colored ink onto a virtual surface and shaping it into marbled patterns. Each ink drop is represented by a set of vertices forming a circle. When new drops interact, they distort each other using a geometric transformation that mimics how fluids displace ink on water.

Dragging the mouse adds "tines", brush-like waves that pull and stretch the surface in a given direction. The deformation strength fades with distance, creating the rippled lines typical of marbled paper.

All the parameters (color palette, drop sizes, and deformation strength) can be tweaked through simple UI elements. It's a surprisingly compact sketch for how organic the results look.

Final result # References #

• Coding Challenge 183: Paper Marbling Algorithm by The Coding Train.
• Digital Marbling by Amanda Ghassaei.
• Mathematical Marbling by Audrey Jaffer.

Outro #

Week one down, and it already feels like a small glimpse into how art and code can overlap. There's something calming about turning a bit of geometry and color into motion that feels almost alive.

A few years back I stumbled onto The Coding Train by Daniel Shiffman, and ended up watching a handful of his videos on YouTube. That's where I first came across p5.js, an open-source JavaScript library built on top of Processing.

Processing describes itself as:

[...] a flexible software sketchbook and a language for learning how to code.

I've been developing software for a while, but I've never used JavaScript in the context of visual and interactive art. p5.js looked like the perfect excuse to try.

So this is a little learning log. I'll be posting sketches and notes about the journey here on the blog. You can follow along by checking out the learning-p5js tag.

References and tools #

Here are the main references I'll be leaning on:

• The Coding Train website and YouTube channel.
• The p5.js documentation.
• The p5.js web editor.
• The p5 npm package.

And if you're curious about the full setup I'm using, you can check out my /uses page.

Thanks for following along on this sidequest. Let's see where it goes.

It's pretty common in the indie web to have a /guestbook, a little corner where visitors can leave a message for others to read. I'd been meaning to add one for a while, but I wanted something that felt native to the site: no external service pages, and styling that wouldn't clash with the rest of the site.

Integrating Chattable #

Chattable turned out to be a nice fit. Once you make an account, it gives you an iframe snippet that drops straight into your page. I just created a /guestbook route, tossed the snippet into it, and initialized Chattable with my own stylesheet:

<script>
  chattable.initialize({ stylesheet: "/css/chattable.css" });
</script>

That was basically it, the guestbook showed up right away.

Theming the iframe #

Styling iframes is usually a headache, but Chattable has a nice escape hatch: you can pass in a single CSS file during initialization.

The trickier part was theming. My site uses a data attribute on the element to store user preferences: theme, font stack, font size... But the iframe doesn't inherit those. I couldn't just poke into its DOM and patch things up.

My solution: pre-generate a CSS file for every possible combination of preferences and point Chattable to the right one.

In practice that means: theme × font-stack × font-size which generates 252 CSS files.

Not something I'd ever do by hand. Instead, I added a beforeBuild function in Eleventy that spits them out for me. Whenever I change a theme file, the CSS set rebuilds and the guestbook stays in sync with the rest of the site.

You can check out the full implementation here.

Then I just needed to point Chattable to the correct CSS file:

<script>
  const theme =
    document.documentElement.dataset.theme ||
    (window.matchMedia && window.matchMedia("(prefers-color-scheme: dark)").matches ? "dark-blue" : "grayscale");
  const stack = document.documentElement.dataset.fontStack || "system-ui";
  const size = document.documentElement.dataset.fontSize || "standard";

  chattable.initialize({ stylesheet: `/css/chattable/${theme}-${stack}-${size}.css` });
</script>

Handling required Javascript #

The guestbook won't run without JavaScript. That's fine, but I still wanted to handle it gracefully.

By default, I hide the guestbook:

#chattable {
  display: none;
}

Then I only show it if JS is available:

document.querySelector("#chattable").style.display = "block";

And for folks with JS disabled, I added a fallback message with <noscript>:

<noscript>
  This guestbook is provided by <a href="https://iframe.chat/">Chattable</a>, and requires JavaScript to run. If you
  want to leave a comment enable Javascript, or <a href="/contact">contact me through other channels</a>!
</noscript>

This way, the page still makes sense even if the interactive part can't load.

Wrapping up #

Getting the guestbook up and running wasn't hard. The bigger challenge was making sure it blended in with the rest of the site. Chattable handled most of the heavy lifting, and a little build-script trickery took care of the theming edge cases.

For now, it feels like a natural extension of the site, not an add-on. Which is exactly what I wanted.

So the guestbook's open. Leave a note, a doodle, a hello. It's all part of the fun.

Every app with authentication eventually needs to tackle authorization. In our case, once our product grew from a small experiment into a mid-sized application, it became clear we needed a proper model to keep users from doing things they shouldn't.

To make the idea more concrete, let's imagine a simple blog with multiple authors and readers. Here's how I explored different approaches.

The naïve approach #

When we built our MVP, we started with the obvious: inline role checks.

function submitUserPostEdit(user) {
  if (user.roles.includes("admin") || user.roles.includes("editor")) {
    // submit the code to the back-end
  }
}

Or inside a component:

<form>
	<textarea>
	{user.roles.includes("admin") || user.roles.includes("editor") && (
		<button id="edit-post-button" type="submit">
	)}
</form>

It's fast, easy, and it works. But the problem shows up later: these conditions are scattered through business logic and UI. When the app grows and a permission needs to change, you have to hunt down every conditional buried in the code.

So, the first improvement is evident: centralize the rules.

Role-based access control #

In the blog example, we can define a few concepts: subjects (users), roles, permissions, and actions. Role-based access control (RBAC) links them together: each user gets one or more roles, and each role comes with a set of permissions that decide which actions are allowed.

Here's a minimal setup:

const PERMISSIONS = {
  admin: ["create:post", "update:post", "view:post", "delete:post"],
  editor: ["update:post", "view:post"],
  reader: ["view:post"],
};

And a simple checker:

export const hasPermission = (user, permission) => {
  return user.roles.some((role) => {
    return PERMISSIONS[role].includes(permission);
  });
};

Which we can use like this:

// User should be like { id: "a", roles: ["editor"] }

hasPermission(user, "update:post");

This already improves things: no more hardcoded checks sprinkled across the codebase. But RBAC still has limits. It doesn't handle nuanced cases where context matters.

Attribute-based access control #

What if editors can only update their own posts? Or if readers shouldn't be able to see posts tagged as drafts? Roles alone can't answer those questions.

Attribute-based access control (ABAC) expands the model by considering more than just roles. It factors in attributes of the subject (user), the object (post), the environment, and the policy itself.

Here's a sketch:

const PERMISSIONS = {
  posts: {
    editor: {
      view: true,
      update: (user, post) => user.id === post.authorId,
    },
  },
};

Now permissions can be a boolean or a callback. The hasPermission function just needs to handle both cases:

const hasPermission(user, resource, action, data) {
	return user.roles.some(role => {
		const permission = PERMISSIONS[role][resource]?.data;
		if (permission === null) return false;
		if (typeof permission === "boolean") return permission;

		return data !== null && permission(user,data)
	})
}

And usage looks like this:

hasPermission(user, "posts", "update", post);

With ABAC, we get flexibility. Whether the condition is based on user IDs, tags, dates, or environment variables, the rules live in one place and can scale with the application.

Next steps #

What we've built so far is already maintainable and expressive, but there's room to grow:

• Centralize permissions across the stack. Right now, these checks only exist in the front-end. Moving them server-side (or sharing them between client and server) would strengthen security.

• Outsource permission management. Instead of living in a JSON file, permissions could be stored in a dashboard or policy service so non-developers can adjust them.

• Use a library. Rolling your own system is a great learning exercise, but in production a library like CASL would decrease maintenance time and delegate new features.

Final words #

That's where I landed for now. It's not the final word on permissions, but it's a pattern that feels maintainable without being over-engineered. Maybe in a year I'll swap it all out for something else, but that's half the fun of keeping a log like this.

At our startup, UI testing was minimal. This was fine during MVP, but as the platform grew in size and userbase, so did the need for increased maintainability. A few weeks ago I remembered this talk by Kevin Yank where he describes how Culture Amp uses Storybook for nearly every test case.

During this last holiday season, my project manager scheduled a couple of weeks to tackle technical debt, and one of the things I had to do was implement Storybook into our existing codebase.

What is Storybook #

You can think of Storybook as a sandbox where each component lives independently from each other. Instead of lauching a whole Next.js app to test some UI components, you can preview them and tweak them in isolation.

Adding Storybook to a Turborepo #

We chose Turborepo to manage our apps since we needed a shared ecosystem of components, hooks, API calls and providers. Turborepo gave us scalability with a packages/ui folder for shared UI elements and packages/lib for API configs, integrations and utils.

We set up Storybook as a new Turborepo app (apps/storybook) alongside our two main Next.js ones. This kept configurations isolated while letting us import from packages to build our Stories.

Our first Story was for the Button primitive component:

import type { Meta, StoryObj } from "@storybook/nextjs";
import { Button } from "@platform/ui/primitives/button";

const meta = {
  title: "Primitives/Button",
  component: Button,
  parameters: { layout: "centered" },
  tags: ["autodocs"],
  argTypes: {
    variant: {
      control: { type: "select" },
      options: Button.variantOptions,
      defaultValue: "primary",
    },
    size: {
      control: { type: "select" },
      options: Button.sizeOptions,
      defaultValue: "default",
    },
  },
  args: {
    variant: "primary",
    size: "default",
    children: "Button",
  },
} satisfies Meta<typeof Button>;

export default meta;
type Story = StoryObj<typeof meta>;

export const Playground: Story = {
  args: { variant: "primary", children: "Button" },
};

Adding NextAuth support #

Many of our components depend on user state, and without NextAuth they would break or show meaningless output. We solved this by setting up a custom session privder for Storybook:

import { unauthenticatedSession } from "@storybook/mocks/session";
import { SessionContext, Session } from "next-auth/react";

export default function NextAuthDecorator({
  children,
  session,
}: PropsWithChildren<{ session?: Session }>) {
  const sessionData = session || unauthenticatedSession;

  return (
    <SessionContext.Provider
      value={‎{
        update: () => Promise.resolve(sessionData) as any,
        data: sessionData as any,
        status: "authenticated",
      }}
    >
      {children}
    </SessionContext.Provider>
  );
}

With this decorator, we can preview components from different user POVs. For example, our PlanCard shows different available actions for admins vs. users, so we can swap sessions instantly without creating a new test account for each user type.

import NextAuthDecorator from "@storybook/decorators/NextAuthDecorator";
import { sessionOptions } from "@storybook/mocks/session";
import AddServiceButton from "@platform/ui/components/projects/business-add-service/AddServiceButton";

export default {
  title: "Components/Project/Creation/Services/AddServiceButton",
  component: AddServiceButton,
  argTypes: {
    session: {
      control: { type: "select" },
      options: Object.keys(sessionOptions),
      mapping: sessionOptions,
    },
  },
};

export const Default = (args: { session?: keyof typeof sessionOptions }) => {
  return (
    <NextAuthDecorator session={args.session}>
      <AddServiceButton />
    </NextAuthDecorator>
  );
};

Adding add-ons to Storybook #

Once the technical setup was done, I focused back on what I wanted to achieve in the first place: easier design collacoration and testing business logic in isolation.

Design collaboration #

Using the @storybook/addon-designs, we can link and display Figma files directly inside Storybook. Our designers can now review UI compliance from a single interface:

export const Playground: Story = {
  args: { variant: "primary", children: "Button" },
  parameters: {
    design: {
      type: "figma",
      url: "https://www.figma.com/design/...",
    },
  },
};

Data & API simulation #

Many of our components depend on backend data. In Storybook, pointing to live APIs is brittle and slow and takes away control over the data we want to perform our tests with. Instead, we:

• Provide React Query context with a global QueryClientProvider.

  import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
  
  const queryClient = new QueryClient({
    defaultOptions: {
      queries: { retry: false },
    },
  });
  
  export const decorators = [
    (Story) => (
      <QueryClientProvider client={queryClient}>
        <Story />
      </QueryClientProvider>
    ),
  ];

• Mock backend calls using Mock Service Worker (MSW).

  import { HttpResponse, http } from "msw";
  import { unauthenticatedSession } from "@storybook/mocks/session";
  
  export const authHandlers = [
    http.get("/api/auth/session", () => {
      return HttpResponse.json(unauthenticatedSession, { status: 200 });
    }),
  ];

  Then, enable it globally in preview.tsx:

  import { initialize, mswLoader } from "msw-storybook-addon";
  import { authHandlers } from "../src/mocks/handlers/authHandlers";
  
  initialize();
  
  export const parameters = {
    msw: {
      handlers: [...authHandlers],
    },
  };
  
  export const loaders = [mswLoader];

This let us simulate responses (e.g., logged-in user vs. error state) and test components without touching real APIs.

Adding internationalization #

Our app relies heavily on next-intl. Without i18n, Storybook would render keys like auth.login.button instead of translated text. The fix was to reuse the same 18n provider we use on our apps inside Storybook:

import LocaleProvider from "@platform/lib/client-only/providers/locale-provider.storybook";

const preview: Preview = {
  decorators: [
    (Story) => (
      <LocaleProvider locale="es">
        <QueryClientProvider client={queryClient}>
          <Story />
        </QueryClientProvider>
      </LocaleProvider>
    ),
  ],
};

Now components display actual strings ("Iniciar sesión" / "Login"), making design reviews realistic.

Next steps #

Storybook began as a way to preview components in isolation, but it's quickly becoming our UI hub. Next up:

1. Interaction testing – script user flows (e.g., form validation) with Storybook's testing utilities.
2. Accessibility checks – add @storybook/addon-a11y to catch accessibility issues early.
3. Living documentation – use @storybook/addon-docs to make stories double as dev & design docs.

Even though the team has shifted to other features, Storybook is now part of my daily workflow. I prototype in Storybook first, which makes development smoother, reviews clearer, and collaboration more inclusive.

---

If you're working in a startup or small team, Storybook can be a game-changer. Instead of juggling staging builds, backend mocks, and screenshots, you get a single space where components come to life.

Try setting up a single component in Storybook this week and you'll see the benefits immediately.

Something that is usually seen in any kind of blog is a way to interact with each post. Usually you can leave comments, but there are also ways to track likes and shares in social media. Some website authors use a ready-to-use service like Disqus, an automated third-party system Netlify functions, or something handmade with a custom database.

This website is SSG'd (static-site generated), by which I mean that all the content is pre-built and my hosting service serves raw HTML files. In order to fit the what I want from this website, whatever solution I came up with had to match certain requirements.

• It had to give me full control of how to display the interactions,
• It should be a free and/or open-source solution
• It should not require the user to login into this site or provide any information to me
• It should be simple, unobtrusive and optional

To fulfill all these requirements, I started to look around the web. Enter Webmentions.

Webmentions #

Paraphrasing W3C, webmentions are a way for a website to be notified whenever another site links to it –hence, web mentions. In this way, instead of having a centralized service to keep track of interactions, it allows a federated approach in which instead of forcing the user to interact with my platform, I simply get notified whenever they do it in any external site.

Webmention io #

There are a lot of ways to implement webmentions, but as far as I've seen a top contender is webmention.io; a simple hosted service to receive webmentions from anywhere, with an API to retrieve these webmentions and showcase them in your site however you like.

Since webmention.io is a webmention receiver, this should work fine if my site is shared by someone who has webmentions integrated in their site. The only problem is that in this day and age, interaction mostly happens through social media and most of them don't use the webmentions protocol.

For that, there's a service called brid.gy.

Bridgy #

brid.gy monitors any linked social media to see who interacts with your site. If I share a link to this article in Bluesky and someone replies, likes or reposts, brid.gy would send a webmention to webmention.io, which I could later fetch with their API and showcase it in my site.

Fetching my webmentions #

Once brid.gy and webmention.io are sending and receiving webmentions respectively, I then can bring them into my site and handle them as I see fit. Since this site is made with 11ty, I can retrieve my webmentions in a global data file and have access to them in any of my templates.

To do this I just need to do a simple authorized HTTP Request.

const response = await fetch(
  `https://webmention.io/api/mentions.jf2?token=${process.env.WEBMENTIONS_TOKEN}&per-page=1000`
);

const body = await response.json();
const webmentions = body.children;

return webmentions;

With my webmentions available as JSON files, I can hanle them and display them however I want. If you wanna see how they are working, you can check them in this article's footer.

{% set reposts = mentions | webmentionsByType('repost-of') | filterOwnWebmentions %}
{% set likes = mentions | webmentionsByType('like-of') | filterOwnWebmentions %}
{% set replies = mentions | webmentionsByType('in-reply-to')  %}

{% set repostsSize = reposts.length %}
{% set likesSize = likes.length %}
{% set repliesSize = replies.length %}

{% set interactions = likes | mergeArrays(reposts) %}

{# ... #}

<div class="social-media">
	{% if likesSize > 0 %}
		<svg {# ... #}></svg>{{ likesSize }}
	{% endif %}

	{% if repostsSize > 0 %}
		<svg {# ... #}></svg>
		{{repostsSize}}
	{% endif %}

	{% if repliesSize > 0 %}
		<svg {# ... #}></svg>
		{{repliesSize}}
	{% endif %}

	<div class="share-buttons">
		{% include "components/share-buttons.html" %}
	</div>
</div>

Next steps #

The main problem remaining with my approach to webmentions is the fact that they will only be obtained whenever I choose to rebuild and deploy a new version of the site. The easiest way to fix this may be to configure some sort of cron job to periodically rebuild my site and deploy it, which would also refetch any webmention I may have available.

Further reading #

There were a few articles I used to help me implement these webmentions. Here they are:

• Adding Webmentions to Your Site by Robb Knight.
• Getting web mentions and brid.gy working on the blog by Jay Caines-Gooby
• Adding webmentions to my site by Bob Monsour
• Add the ability to send Webmentions to other IndieWeb sites by IndieWebify.Me
• Adding webmentions to your static blog by Jan Monschke
• An In-Depth Tutorial of Webmentions + Eleventy by Sia Karamalegos

In the first few versions of my personal site all my Global data was from hand-crafted JSON files I'd update myself from time to time. This was fine for its purpose: I would add new info very rarely and I had complete control over how I wanted the data to be structured.

A few weeks back I came across PhotoGabble's bookshelf and Cory Dransfeldt's site, and liked how they showcased their favourite media. I think your artistic preferences can say a lot about who you are as a person, and sharing it with the world can help other like-minded people to discover new things they may have not even have heard about.

Needless to say, I wanted to do something similar to those cool websites and got carried away with it. You can see the full result at my media page and each individual page such as manga or games.

Obtaining from the source #

First I thought about what media I enjoy and interact with the most, and would like to share with people. If I had to put it in order it'd be something like Youtube videos, comics, games, shows and music. Knowing this, I wanted a way to automate retrieving some kind of lists that I can parse and display in this site.

Aside from the obvious –like using Youtube to track videos –, I had to see where I did or could track these lists. For games I chose the following:

• How Long To Beat for games,
• Trakt.tv for shows and movies,
• MangaDex for manga,
• YouTube for videos,
• and Spotify for music.

I also later added webcomics, but I made that a little differently and I may write a follow-up article about it.

The problem with some of these services –specifically HLTB and Trakt.tv –is that no public API is available. HLTB has an an open discussion about it, but there's nothing planned in the roadmap yet.

For these specific cases, I resorted to webscaping.

Webscraping #

Webscraping is a practice I'm skilled at but is never my first choice to obtain any kind of data from a provider. Aside from any ethical implications it may have, its unreliability and dependance on the source's front end makes it a not very robust way of retrieving data.

However that may be, I created some data files in my site and implemented the webscraping algorithm with Puppeteer.

Webscraping my game lists #

HLTB allows you to have a few "lists" depending based on the status of the game. To organize my data, I created an object based on these status keys and which public URL matches it.

const PAGES = {
  playing: `https://howlongtobeat.com/user/${HLTB_USER}/games/playing/1`,
  backlog: `https://howlongtobeat.com/user/${HLTB_USER}/games/backlog/1`,
  favourites: `https://howlongtobeat.com/user/${HLTB_USER}/games/custom/1`,
  played: `https://howlongtobeat.com/user/${HLTB_USER}/games/custom2/1`,
  completed: `https://howlongtobeat.com/user/${HLTB_USER}/games/completed/1`,
  retired: `https://howlongtobeat.com/user/${HLTB_USER}/games/retired/1`,
};

Thanks to this, I can iterate through each page in order to access it with Puppeteer.

for (const [status, url] of Object.entries(PAGES)) {
  collection[status] = await scrapeGamesFromPage(page, url, status);
}

In each of these pages, there's a list from where we can obtain a div element with some game information.

await page.waitForSelector("#user_games");

Since the information obtained from can be a little scarce, I created two functions: one scrapes the list itself, and the other navigates to the game's profile where it obtains the rest of the data needed. With this, I had all the data I needed to structure my objects.

{
  id,
  type: "games",
  title,
  description,
  genres,
  platform,
  link,
  thumbnail: image,
  updatedAt: extendedInfo?.date_updated,
  addedAt: extendedInfo?.date_added,
  startedAt: extendedInfo?.date_start,
  completedAt: extendedInfo?.date_complete,
  playtime,
  rate,
  author: { name: developer },
}

See the full implementation here.

Webscraping my movies and shows #

Trakt.tv works extremely similar. I also had a object with [key: Status]: Url where I iterate to visit and scrape data from a grid of elements. As a small benefit, Trakt.tv makes use of a lot of data-attributes in their DOM elements, so obtaining the data was easier in some instances.

const [title, link, originalTitle, id, date_created, date_added] = await Promise.all([
  titleEl.evaluate((el) => el.innerText),
  linkEl.evaluate((el) => el.href),
  element.evaluate((el) => el.getAttribute("data-title")),
  element.evaluate((el) => el.getAttribute("data-list-item-id")),
  element.evaluate((el) => el.getAttribute("data-released")),
  element.evaluate((el) => el.getAttribute("data-added")),
]);

See the full implementation here.

Using the Youtube API #

In order to retrieve the videos I wanted to share in my site, I made a public playlist with my favourite ones. My first instinct to retrieve them was through its feed URL https://www.youtube.com/feeds/videos.xml?playlist_id=; however the data available through this method was lacking so I started looking into Youtube's public API.

After setting up the API keys, I could implement an HTTP Request to obtain the playlistItems.

const url = new URL("https://www.googleapis.com/youtube/v3/playlistItems");

url.searchParams.append("part", "snippet");
url.searchParams.append("part", "contentDetails");
url.searchParams.append("maxResults", "50");
url.searchParams.append("playlistId", playlistId);
url.searchParams.append("key", process.env.YOUTUBE_API_KEY);

const response = await fetch(url);

The problem again was that it lacked some of the details I wanted to share, like the number of views or the tags. For that, I had to make another Request.

const url = new URL("https://www.googleapis.com/youtube/v3/videos");
url.searchParams.append("part", "contentDetails");
url.searchParams.append("part", "snippet");
url.searchParams.append("part", "statistics");
url.searchParams.append("id", videoId);
url.searchParams.append("key", process.env.YOUTUBE_API_KEY);

const response = await fetch(url);

With this I had all the info I could see myself needing, including some additional properties I may use in the future.

{
  id: video.contentDetails.videoId,
  type: "videos",
  title: video.snippet.title,
  description: video.snippet.description,
  link: `https://youtube.com/watch?v=${video.contentDetails.videoId}`,
  thumbnail: video.snippet.thumbnails.standard?.url,
  createdAt: video.contentDetails.videoPublishedAt,
  updatedAt: video.snippet.publishedAt,
  author: {
    name: video.snippet.videoOwnerChannelTitle,
    link: `https://youtube.com/channel/${video.snippet.videoOwnerChannelId}`,
  },
  rate,
  views,
  duration,
  tags,
}

See the full implementation here.

Using the MangaDex API #

MangaDex also has a public API, documentation and Swagger so it wasn't particularly difficult to retrieve my data. In that case I had the following object to do a lookup for the URLs.

{
  MANGA_LIST: `https://api.mangadex.org/list/${MANGA_LIST_ID}`,
  MANGA_BASE:
    "https://api.mangadex.org/manga?includes[]=cover_art&includes[]=artist&includes[]=author&contentRating[]=safe&contentRating[]=suggestive&contentRating[]=erotica",
  CHAPTER_BASE: "https://api.mangadex.org/chapter",
  FOLLOWS:
    "https://api.mangadex.org/user/follows/manga?&includes[]=cover_art&includes[]=artist&includes[]=author&includes[]=manga",
  STATUS: "https://api.mangadex.org/manga/status",
};

The first difference I needed to implement was an authorization function, where I would retrieve a JSON web token in order to send in further request as a Bearer token.

async function authenticate() {
  const credentials = {
    grant_type: "password",
    username: process.env.MANGADEX_USERNAME,
    password: process.env.MANGADEX_PASSWORD,
    client_id: process.env.MANGADEX_CLIENT_ID,
    client_secret: process.env.MANGADEX_CLIENT_SECRET,
  };

  const formData = qs.stringify(credentials);

  const response = await fetch(AUTH_URL, {
    method: "POST",
    headers: { "Content-Type": "application/x-www-form-urlencoded" },
    body: formData,
  }).then((res) => res.json());

  headers.Authorization = `Bearer ${response.access_token}`;
}

With this, each and every subsequent request would be properly authenticated and authorized.

const [followList, statusMap, favouriteList] = await Promise.all([
  fetchAllPaginated(ENDPOINTS.FOLLOWS, headers),
  fetchJSON(ENDPOINTS.STATUS, headers),
  fetchJSON(ENDPOINTS.MANGA_LIST),
]);

The /user/follows endpoint has a maximum limit of 100, so in order to retrieve all my following manga –with all its different potential status –, I had to implement a paginated request. This was fairly easy, since the API itself returns everything needed.

async function fetchAllPaginated(url, headers = {}) {
  const limit = 100;
  let offset = 0;
  let allData = [];
  let hasMore = true;

  while (hasMore) {
    const paginatedUrl = `${url}&limit=${limit}&offset=${offset}`;
    const response = await fetchJSON(paginatedUrl, headers);
    allData = allData.concat(response.data);
    offset += limit;
    hasMore = response.total ? offset < response.total : response.data.length === limit;
  }

  return allData;
}

The other small pitfall is that MangaDex includes a CORS firewall to disallow linking to their images –for me, I wanted the covers –, so instead I had to download them into memory.

async function downloadCover(url, fileName) {
  const buffer = await fetch(url)
    .then((res) => res.buffer())
    .catch((err) => OPTIONS.logError && console.error(err));
  const filePath = `${coverPath}/manga/${fileName}`;

  if (!fs.existsSync(filePath)) fs.writeFileSync(filePath, buffer);
}

And with this I had all the data needed.

{
  id,
  type: "manga",
  title,
  description,
  genres,
  author: { name: author },
  rating,
  link: manga.attributes.links?.raw,
  thumbnail: `/assets/images/covers/manga/${coverFileName}`,
  updatedAt: latestChapter.data?.attributes?.publishAt ?? manga.attributes.updatedAt,
}

See the full implementation here.

Using the Spotify API #

I have to say I'm not a big Spotify user, but in order to keep track and order my music for this site, I created a playlist which then I wanted to recover its information through their API.

const PLAYLISTS = {
  favourites: "79jHGYxWHmhXthpE0o8DIK",
};

After setting up my API keys I needed to use them in the authentication step to generate the headers needed for every other request.

async function authenticateSpotify() {
  const authRes = await fetch("https://accounts.spotify.com/api/token", {
    method: "POST",
    body: new URLSearchParams({
      grant_type: "client_credentials",
      client_id: process.env.SPOTIFY_CLIENT_ID,
      client_secret: process.env.SPOTIFY_SECRET_KEY,
    }),
  });

  const { access_token } = await authRes.json();
  headers["Authorization"] = `Bearer ${access_token}`;
}

After that I could iterate through all the playlists I wanted to recover, and fetch the playlist tracks.

for (const [key, playlistId] of Object.entries(PLAYLISTS)) {
  log("[Spotify]", `📥 Fetching playlist: ${key}`);
  const playlistData = await fetchPlaylistTracks(playlistId);
  collection[key] = await transformPlaylistItems(playlistData.items);
}

Once fetched I just had to map the data into the object structure I wanted to use and I was done.

{
  id: track.id,
  type: "music",
  title: track.name,
  thumbnail: track.album.images[0]?.url || null,
  author: {
    name: track.artists.map((a) => a.name).join(", "),
  },
  genres,
  addedAt: item.added_at,
}

See the full implementation here.

Applying the data to my HTML templates #

For this website I'm using 11ty with Nunjucks, so once I have my global data accessible in every template, I just had to display it.

I created a few reusable components to show my media as either a description or a grid of thumbnails, with toggle buttons inside each header to show it as the users preference. In order to do that I only had to choose which data I wanted to pass to the component.

{% set media = shows.favourites | limit(5) %}
{% set href = "/shows/status/favourites" %}
{% set sectionTitle = "Favourite shows" %}
{% include "components/media-list-section.html" %}

This rendered the html as I wanted it. You can see how this looks in my media page, where each section is defined with this structure. The choise of grid or list is saved into the users LocalStorage, as I wanted it to persist between navigation and visits to my page.

Final notes #

The full implementation is my site's repository. You can check out all my data files, my template implementations and how I generate different pages for each of them.

Thanks for reading this far and if you have any questions