https://zumi.neocities.org/bloge/feed.rss

Literate programming—or LP—is one of those things that represent some utopian way of Building Shit™, promising that you will be elevated to some higher level of being. Or something. Just like purely-functional-programming or the borrow checker. It fixes bugs! Makes you a better programmer! And yet, even without such qualifications, this concept has always fascinated me. Essentially, it's the formalization of every single "code tutorial". As I say, it's literally "code, but pretend you are writing a tutorial on how to do it". Hell, people point to Physically Based Rendering as an example of "LP done right".

Most modern day conceptions of LP involve some sort of notebook, like Jupyter Notebook. Don't get me wrong, those are great. The fact that you can see output of some piece of code from the get-go is very helpful for understanding in a way not so easily achieved by traditional LP. But "trad LP" is the slightly more interesting formulation of Donald Knuth (you might know him from TeX). The Knuthian definition of "literate programming" emphasizes code reordering, something that you couldn't get in notebooks. It is essentially "folding code" 40 years before it was cool. (Then again, he was always like 40 years ahead of his time—if you've heard of variable fonts, why not check out METAFONT?). It involves two separate, yet parallel, processes:

1. Tangling: generating source code files to feed to a programming language compiler.
2. Weaving: creating documents to print and read, e.g. HTML, PDF, PS, etc.

And they both are generated from the exact same source or set of sources. You may look around for examples, but they should involve pieces of code that are hidden underneath names, and then instantiated sort of like a "macro version" of a macro. And it really is, in a way, another view of programming: instead of writing-as-you-go or writing pseudo-code first, you literally just write pseudo-code and then translate it in the same document into the language of your choice.

But trying this out, of course, has been a bit tricky and definitely experimental. A lot of the documentation pertaining to LP as understood as such, dates back to the 1980s through the early 2000s. Which makes sense, as some of the "exciting" benefits of LP has since been evaporated by advances in tooling:

1. We don't need pretty-printed source anymore; as syntax highlighting has become fast and practical for most programs out there. Besides, seeing ≠ instead of != seems rather disorienting (although some people don't think so—evidenced by the myriad of coding fonts out there that have that as an actual ligature!)
2. It isn't as necessary anymore to create indices on our own; since with IDEs (especially with the invention of the LSP) you can simply jump to where something is defined and have the symbol index right there on the sidebar.
3. Chunks are an early form of "code folding". Well today, as mentioned, we have #region and actual code folding. Hell, there's some pushback against #region—some of which seems also applicable to LP as is applied.

Regardless, I think my entry point to all this seems to be a few articles—probably this one—that, despite the advances above, still believe in Knuth's vision. Maybe people really like explaining. Maybe capturing the context with the code, especially in that way, is still seen as a nice thing. Or that reordering code is still pretty cool. Even in the age where you can define functions anywhere and not have to define one after another, as in the days when LP was first invented. I'm a little bit of all of those, and so I've experimented with a few different tools for Knuthian LP.

These two are mentioned in the article linked to in the previous paragraph. srcweave in particular was written by the article's author. As I said before, LP is like a formalization of "code tutorials", well, why not write an entire tutorial using this very method? Initially in literate, I rewrote it in srcweave after a bit. I did try to make the most of it—putting in little graphics of my own that explain some hardware-related concept while having the actual code to operate it beneath. If I had something to say about these two:

• srcweave: More versatile, harder to setup. srcweave does not assume text processing will be done using it and leaves things like markdown conversion to other tools. Moreover, it's using SBCL, which I don't find its package story to be all that smooth-sailing for some reason.
• literate: All-in-one, less versatile, easier to setup. It uses D's package system, which works well. Unlike srcweave however it assumes a certain structure.

Both literate and srcweave use Markdown. Which is nice, but they still have their own format which makes it not that presentable on GitHub. Or any other git forge with a Markdown viewer. So I thought, why not make a tool that… just used Markdown? It'd be great to have one as a README.md. That's why I wrote NailIt. I don't think I've ever used this in any serious sense other than its own program description, maybe that time will come yet. I did say, I want less "refer to section numbers a code block is in" and more "refer to code blocks directly" in my output—indeed literate and srcweave still refer to blocks/chunks by their section numbers. Which I think is probably a carry-over from Knuth's original tooling which is built for books (WEB, CWEB, and then noweb follow this model). But we live in an age of hyperlinks and so maaaaybe we ought to say the exact context where X is used.

This one is essentially the LP IDE. You create or load in a single project file that contains all your code, and you navigate through an outline of nodes (chunks), each of which can be edited interactively. Nodes have hierarchy—sub-nodes can either be the traditional <<program part>> or another documentation node. A single node can even have multiple references (called "clones")—for instance, you have a node containing a FooBar class. You can have a reference to the node underneath a node called How to do FooBar and a node called Class index at the same time—it's exactly the same node and a change to one will change the other—but it allows you to view the thing in multiple different contexts.

I thought this was the closest LP has been to being actually viable, especially for languages whose LSPs are iffy. It looked so clean, and so useful. Especially its killer feature: Leo can "untangle" changes made to tangled files back to the source! Especially @clean, which outputs… basically regular source code, so you get the best of both worlds! …In theory, of course. While it does a pretty good job, it's not perfect and it can trip up sometimes, so care must still be taken not to change too much in the tangled program, lest it untangles it all wrong. As always, too good to be true.

Since this is a tool with its own format, that means there's a certain level of "lock-in". This is mitigated somewhat due to Leo having multiple implementations: There's the main version that runs on PyQt (which in my experience isn't quite smooth-sailing on some setups I have), and then there's LeoJS (a VSCode extension). No matter which one I use, there's unfortunately some level of friction, probably on my end rather than theirs. The Qt version does run on Termux while I'm "on-the-go", so to speak, but opening the file browser is slow for some reason. Again, I'd chalk it up to Qt. Even without the technical problems, I'm still not sure if I want to go with this for my day to day work on account of "plain code" being easier to work with in general.

This is the solution built by Graham Nelson, of which I am told is practically the king of interactive fiction—which in turn the only idea I have is... uh... going North, South, or Dennis. But it is also a fascinating tool in its own right, its output very much modern. It imposes a book-like structure right down to the file names: code and documentation live in sections which have Names like This.w (and yes, it wants spaces), and those sections live in folders like Chapter 1 and Chapter 13—their titles given in a file called Contents.w given at the root of the project. You can string multiple project together in "webs" although I haven't found a use for this feature yet. I think I might like the idea of forcing me to structure my program like an actual book, because of this I can seek out at a glance the context of where a piece of code might be—if I want to know where to Configure the Reflurberator I could go to the section called "Configure the Reflurberator" in the table of contents.

But while—like other LP tools—it claims support for "any language", there is a notable red-carpet treatment towards C-likes—but in practice, it's really just C. And its own little C dialect "InC" which among other things adds a sort of Namespace::capability.
Attempting to use it with C++ has made my C++ code end up just being C. As Inweb recognizes function definitions like void MyClass::something(, it helpfully tries to define it first as a C declaration. Which causes all sorts of compiler errors as it tries to reconcile the resulting declaration with the class that follows, and I need to use = (very early code)hacks to resolve it. I ended up refactoring my code to look like void MyClass__something(MyClass &self, ..). You can imagine from here how it'll work with templates and use all complicating matters. Yeah again I might as well just use C or InC. What about indented languages like Python or Nim? Suffice to say, it's not a good fit. Its output doesn't really care about spacing as much as it should for those kinds of languages, which is sad because I really like its output.

Unlike the tools mentioned so far, this tool does not allow multiple tangled files to be generated. I get why: like, you have a complete program here, why would you want to split that into files again? You already organize the program in a structure that fits it. But that's not the problem here—rather, the opportunity for incremental compilation has been utterly decimated. Tangling this thing always results in one big program, and that program has to be compiled anew every single time. You couldn't recompile, say, only dingo.c of myprogram.c, because it always creates myprogram.c as one big amalgamation (as SQLite would say). There is support however for creating files that just so happen to be attached to the program, maybe as in configuration or Makefile, and those are quite intentionally limited.

Initially I brushed this off after thinking it was seemingly tied-to-the-hip with TeX. Debian attaching TeXLive as this thing's dependency did not help at all and has, yes, in fact scared me off it. But after looking into this again, it's absolutely none of those things. It's even better.

1. TeX isn't actually parsed or even actually required by the tool. It only parses: <<code usages>>, <<code definitions>>=, and [[quoted code]]. That's literally it. The documentation parts aren't even touched, and will pass right through.
2. Building the thing is also equally simple—it only needs C and some form of AWK. There are mentions of a language called Icon (think of: alternative to Perl and Python) but it isn't needed either.
3. Finally, it's actually a whole suite of tools. noweb, notangle, and noweave are actually shell scripts that drive more granular tools: markup, noidx, and tohtml/totex.

It's Unix as all hell, and that is actually its killer feature. Instead of being an all-in-one parser and converter like the other tools so far, they are each their own little programs. Right down to the fact that stdin and stdout is literally all you need to operate these things. The parser, markup, actually converts the source into its own intermediate format. The idea here being that anyone can ingest this intermediate format and produce documents in their own way. Everything else, even code indexing (provided by noidx), is optional.

Anyone can slot in their own tooling—no wonder it has a 30-year (and counting) lifespan. So, made my own I did. It definitely isn't quite what I want just yet, but to me it's already preferable than what noweb's default HTML output looks like. Notably at present, references still refer to chunk numbers despite what I set forth in NailIt, but I'm sure even that can be worked around with. But the notable thing is that since I'm working with an intermediate format that is easily parsed, I can add restrictions of my own—in my tool, I forced file names to follow a number-chapter format, the idea being that a Makefile or a glob on a source directory should be all I need instead of a Contents.w. If I wanted to, I could also restrict adding to code chunks because that would hinder being able to analyze the code quickly. Or making the chunk name do special stuff like function names. Whatever I want, I can do it here.

In trying to use LP as a practical tool for assisting coding in some instances, here's my attempt at formulating some light guides:

Don't hide your variables. Code chunks are, in a way, its own function. In a chunk, I want to, know what output is and does. So I name my chunks something like add something to "output".

Ensure that any variables mentioned in a code chunk is either:

1. defined in whatever chunk is using it—so you can "follow the yellow brick road" of "Used in" links), or
2. the variable is local to that chunk.

How Inweb handles this in C code is that every chunk is surrounded by { and }, which is meant to make if (something) @<do other thing@> possible, but the local-scoped thing is a good side effect.

Not every chunk needs a whole novel. "Old-school" LP says that the explanation is the enlightening part, but it worked in a time where it was still humanly possible to slow down and bask in every little detail. Now? Well, you just want things to work. Over-explanation was one of the things that led ifupdown's current maintainer to cut out noweb entirely. That leaves us with: when to split off chunks? The following circumstances should apply:

1. The chunk is more than a page's worth, but you feel it shouldn't be its own function.
2. It's not completely obvious what it does and it genuinely needs a paragraph instead of a one-line comment up top.
3. The chunk is related to some other aspect of the program that would not make sense to be explained in the current section.

My trick is to make each function its own chunk. Inweb handles this automatically with C, because C function declarations follow a quite regular pattern. Not so with noweb, since it's a close to "universal" tool. My trick then is to just use <<function somethingHappens(shit)>>, the declaration either "above" the chunk (in the thing that uses it) or "under" the chunk (as part of the chunk)

Reduce the number of chunks overall. More chunks mean another possiblity to break the control flow as the user reads it. And this especially includes the kind of "appending to chunks" common to LP tools. While useful to "add functions" as they are defined, when overused it can hinder understanding a function quickly (remember, world moves fast these days…)

So what's your New Years' resolution? I don't have anything serious for mine, but I want to try out a few "principles" (if you can call it that?) to hopefully reduce my data hoarding habits.

Here's a few that I can think of.

For a few years, I've set up something resembling a "RAM disk":

tmpfs /scrapdisk tmpfs rw,nodev,nosuid,size=4G 0 0

I had to create a new separate tmpfs here since /tmp was already taken and full of application junk. I'm not sure what article inspired me, but it was probably this one. This line especially stood out:

Imagine working on your computer and having to download files off the Internet - files that will be used once (for instance, ZIP files) but can be tossed away on the next reboot. In fact, Safari on OSX, and, more generally, OSX tends to pollute the ~/Downloads folder with a bunch of files downloaded off the Internet that, unless explicitly deleted, will end up taking up loads of disk space even though they have no more use.

Anyway. Basically, this "RAM disk" is exactly what it sounds like: it lives in RAM (and swap) so it resets on every boot up. And as that quote advised, I use it like this:

1. On every browser I have, I set its Downloads folder to this /scrapdisk folder.
2. Exported stuff for pasting into chats and whatnot goes there, if not the clipboard.
3. I only move stuff out of there if I decide I will look at it later, or if it's too big.

So that's fine and dandy, but I'm thinking of extending this concept to the hard drive:

As I said before, I've got a problem with hoarding, essentially the inability to decide what to keep… so I just keep them all. Companies have these data retention policies that cover way more than lifetime and seem to be more for legal reasons, but I'd figure I take a cue from them for personal and "storage efficiency" purposes.

Volatile: This is what/scrapdisk effectively is. It can really last as long as I'd like (even months, if I'd so inclined), so long as my computer isn't shut down or rebooted.

Medium term: Anything in here would be deleted monthly—anything goes, at the start, at the middle, or the end… just a fixed time that I know I have to check out. Or let my computer remind me that there's things in there I'd like to check.

Long term: This is the one I'd want to back up. Yeah, I know, space is relatively cheap and all especially with SSDs around, but transfer speeds and how much I end up backing up (more than necessary!) is still a big problem.

There really isn't anything here, files can be moved around between these three tiers at any given moment. Although it's kind of hard to imagine a case where I'd want to move something from "long term" to "volatile" instead of just plainly deleting it.

Now this is just an idea, I'm not sure how best to implement this. Ideally it's something that the computer itself can enforce whether through scripts or file system technicalities. At the moment I'm just mulling over the layouts and stuff.

Some hidden files turn out to be important. Depends on your point of view of course. Real problem is the fact that .cache (something not very useful to backup) and .config (something kinda useful to backup) are both hidden files ("dotfiles", because they start with a dot). And of course grrrrrrrrrr .mozillaaaaaaaaaaaaaaaaaaaaaaa

Now I leave them on in my file manager views for completeness but sometimes it bothers me with the amount splattered on my home directory. And some (hopefully old) programs just leave dotfile trails even when I try said program just once, leading to some amount of analysis paralysis whenever I try to choose dotfiles to back up.

So then I just said fuck it, and did two things:

1. Use Stow to have a ~/dotfile folder and symlink everything in there to ~ automatically.
2. Ran a script to delete every dotfile in ~ and re-stow everything immediately before I log in.

Stow by default symlinks as-is—that means "hidden" files stay "hidden", and so I still have to have folders like .docker in there. With "show hidden files" disabled on my favorite file manager the dotfile folder seems empty. With --dotfiles I can just say screw that, as on anything prefixed with dot- it'll just replace that with an actual dot, so the folder still has Things™ on it.

As for the script:

#!/usr/bin/env bash

for i in `find $HOME -maxdepth 1 -name '.*' ! -name '.local' ! -name '.' ! -name '..'`; do
  rm -rf $i
done

stow -d $HOME/configuration --dotfiles .

And the systemd user service:

[Unit]
Description=Clean home directory

[Service]
Type=oneshot
ExecStart=/home/zumi/.local/bin/reset-home

[Install]
WantedBy=default.target

user enable --now, yadda yadda.. And loginctl enable-linger $USER for some reason.

All I'm trying to say is, dotfiles are worth something if I can just see them. And if I can, there's probably going to be a chance of consciously picking and choosing things. You might have noticed that .local is an exception. I mean, I can treat .local the same way and then symlink a visible local folder to it.

About the inability to decide what to keep… it applies to the physical as much as the digital. Somehow everything, even the littlest things seem to have vague or even unspecified "sentimental value". Like, for example, a specific blank piece of paper. Thing is, I can just buy another piece of paper. But there is sentimental value in this one that I end up just keeping the damn thing.

It's something that I can't explain, because if I try to do so, whoever's listening might just double down on the "throw it away, just buy another one". "Sentimental value", however vague it is, is I believe something that only I can judge, by definition. I've had an old inactive ID card trashed once just because it didn't match someone else's definition of "sentimental value", which would probably be just family photos or whatever.

Sometimes I ask myself really if I'm just what I am now, or if I'm the sum of what I've done in the past. Surely it's the latter, or else you wouldn't ask me for my track record or whatnot.

But then what's worth to keep? The fact that I'm asking this at all means I must be questioning whether or not I can even afford to keep things I think to be sentimental. I don't really know what's worth holding on to really. Maybe throw some of it at the Cloud™ (the clown) but even there, there's a limit.

Or, the actual title: My dumb error-handling idea.

It's ${CURRENT_YEAR}, and error handling is STILL not a "solved problem", especially when it comes to balancing ergonomics (how good it feels to code with) and performance. It's still very much debatable what approach you use, whether it be exceptions, error codes, Result/Option types, either monadic (Rust etc.) or tuple (Go, Odin).

I have made the mistake of reading programmer "Disc Horse" (particularly on Hacker News and Lobsters) where ✨the enlightened✨ makes fun of the Blub programmers for having such a error-prone and inefficient way of handling errors, among other things. All of this hasn't added clarity, only confusion. Enough for me to add fuel to the fire, especially as someone completely unqualified to yap about any of this.

So let's first enter these rough points from said discourses into "evidence", of which I won't elaborate or argue for:

1. The design of exceptions is bad and you should feel bad, prefer error codes and result types.
2. Error handling should be front-and-center and not just afterthoughts, therefore they should be part of your code.
3. "Careful thought and strategically-placed print statements" still reign supreme.
4. The if err != nil: return nil, err ("simple re-throwing") pattern in Go.
5. fmt.Errorf("%w", err) ("error wrapping") in Go.
6. Log and rethrow can be considered an anti-pattern.
7. Premature abstraction is the root of all evil.
8. It is still useful to have different types of error, because not every error is equal.

Next, consider this log of an unhandled exception of a web service:

ValueError exception: Can't parse "blerp" as Date
from: std/parsedate:100
from: /home/zumi/dev/myDumbProject/src/parsetools/object:245
from: /home/zumi/dev/myDumbProject/src/fetch/object:124
from: /home/zumi/dev/myDumbProject/src/controller/getObject:24
from: /home/zumi/dev/myDumbProject/src/handleRequests:24

You can probably infer from this that there has been an error when accessing some Object, and that it has received a nonsense value for a date input. But there's two bits of context I'm missing here:

1. What kind of date was it trying to parse, exactly? As in, which database column? (and it'll most likely be from a database)
2. Which object ID did it choke on?

In a trivial database, you can probably go to the line and then go to the database to find the odd value. But all "serious" web services will have to talk about scale. You can imagine that plus a wild request, and suddenly it becomes a bit harder to rectify.

And you could argue that I should have caught it at the lowest level I could. A bit easier with a language that has checked exceptions, because the compiler will tell you which ones to catch.

Besides, the whole point of exceptions is so that you can get the error-handling cruft out of the way, right? So then, you'd be likely to wrap your entire function inside of a giant try statement rather than try-catch individually. In some languages, try is its own block. Imagine if you want to initialize variables this way, unless blocks can also be expressions.

But what if I can have this instead:

[!!] can't parse object's creation date. e: "ValueError: Can't parse "blerp" as Date". value = "blerp". [std/parsedate:100]
[!!] can't parse object. [parsetools/object.src:245]
[!!] can't fetch object. id = 99. [fetch/object.src:124]
[!!] can't display object. id = 99. [controller/getObject.src:24]

Right out of the gate, I have the answer to the questions of context, and I can fix the issue faster:

1. blerp for some reason was the value of the creation date in the database.
2. The issue was with ID 99.

This can be achieved anyway you like. But what if the log is the stack trace?

There's a practice I know of that simply has strings as the error value, so basically Result[T, string]. I think the idea being that, they're just values, so as to keep the function pure and you have the option of logging it or not.

What if I, like a dumbass, don't care about "keeping it pure"? I just assume that a logger package is present, initialized, and I utterly depend on it, and I don't want the option of not logging the error?

And instead of having only one or maybe two kinds of error, results or error codes or exceptions or whatever... what if I have all of them??

I came up with these error types:

• Outcome
• DifferentiatedFail
• Option[T]
• Result[T, DifferentiatedFail]
• Result[T, ContextualFail]

It's important to note that this is NOT a general error-handling strategy. This shouldn't be used to "take over the world", but instead to ONLY be used in YOUR code. The libraries you use can do whatever, exceptions, results, whatever. The strategy described here should be used in code that YOU care about. It's really a way of marking boundaries between app code and "library" code. A.k.a, "app code that faces the user" where this approach can be used vs. "app code that can be used elsewhere" where e.g. exceptions can be used.

And instead of the focus being on the function itself, the focus is in whoever calls it. I think it fits into the "code what you need" mindset instead of prematurely preparing abstractions that will not hold up.

For the pseudo-code in this article let's assume something that looks kind-of-like C, and has exceptions.

A value of either Fail, or Ok. Implementing this can simply be a boolean true/false—whichever denotes the error value depends on what kind of standards you have. For example, C functions usually say "return true if there is an error", which results in "non-zero value denotes an error".

An enum that just says what kinds of errors are possible that you care about. They can be literally anything depending on what you need, perhaps in the form of these constants, which map neatly to HTTP errors:

InternalFail // 500
ExistsFail // 404
PermissionFail // 403

Your bog-standard "safe nullable reference" optionals type that you need to "unwrap" to use, forcing you to check the thing. In absence of this, you could just use the nullable reference type, but you carry the risk that comes with it.

Here it communicates that either there is only one possible way that the call can error out, or that the caller should not care what kind of error occurred. After all it just needs to know whether it would receive the value or not.

The ok-or-error monad that Rust convinced me was the One True Way to Go. And speaking of, yeah, Go fits this model, although it's like the "standard nullable reference" version of this model, since you can forget to check it. oooOOOOooOOo BILLION DOLLAR MISTAKE!!!!!! BAD!!!!!! or something. Hate this type of antagonism.

The closest thing to "exception objects", but it's lighter because it just contains this:

struct ContextualFail
{
    kind: DifferentiatedFail;
    message: string;
};

• Errorless functions obviously either return some value of type T or has nothing to return (void/never, some call these functions "procedures" instead)
• One type of error to care about? Either Outcome or Option[T].
• Multiple types of error? DifferentiatedFail or Result[T, DifferentiatedFail].
• If and only if your caller needs a custom message that they themselves can't generate, then you should go for Result[T, ContextualFail].

In short:

The thing about DifferentiatedFail is—again—it can be anything, even function-specific:

UsernameVibeCheckFailed
PasswordTooStrong
ConfirmationPasswordMismatch

This is fine for standalone console apps or whatever, but if you're writing a web service, you may as well fold it into a ContextualFail:

ContextualFail{
    .kind: ValidationFail,
    .message: "Invalid user name, must have at least one em-dash in it"
}
ContextualFail{
    .kind: ValidationFail,
    .message: "Password too strong, must have a maximum of 3 characters"
}
ContextualFail{
    .kind: ValidationFail,
    .message: "Password confirmation doesn't match password input"
}

The idea here being that ValidationFail would map to a 400, and then the message can be a message that can be flashed alongside the re-thrown form.

Alright. I'm a bit more familiar with Go so that's what I'll use as reference. In Go, you might do:

func callee() (T, error) {
    // ...
    if err != nil {
        // you can choose to bury errors here
        return nil, errors.New("can't do x")
    }
    //
    return ...
}

func caller() error {
    a, err := callee()
    if err != nil {
        // if callee uses this pattern, you can easily
        // get a mile long string...
        return nil, fmt.Errorf("can't get a: %w", err)
    }
    return nil
}

func main() {
    err := caller()
    if err != nil {
        // you could log only in here, but then you
        // don't get WHY it happens...
        log.Fatalf("%w", err)
    }
}

In my dumb error handling proposal, I'd do, in pseudo-code:

Option[T] callee()
{
    try
    {
        x = // ...
        return x.some()
    }
    catch SomeError as e
    {
        log.errorf("cannot get x: %s", e.message)
        return none(T)
    }
}

Outcome caller()
{
#if you_prefer_pattern_matching
    match (callee())
    {
        some(a)
        {
            // do stuff with a
            return Ok
        }
        none()
        {
            // note how you don't return this log itself
            // as a value, but it's just a log
            log.errorf("cannot get a")
            return Fail
        }
    }
#else
    a = {
        result = callee()
        if (a.isNone())
        {
            log.errorf("cannot get a")
            // this is an early *explicit* return,
            // exits the entire function
            return Fail
        }
        // *implicit* return assigns to `a`
        // and continues
        result.get()
    }
    // do stuff with a
    return Ok
#endif
}

int main()
{
    if (caller() != Ok)
    {
        log.errorf("cannot do thing")
        return 1;
    }
    return 0;
}

You'd get:

cannot get x: some internal error
cannot get a
cannot do thing

Notice how you don't need to:

• Carry around strings
• Have an existential crisis over what to log, because you log everything
• Care about telling WHY it failed, because it's already been said

How might an error string be used, aside from logging, anyway? Go already has a hard time differentiating errors (need to preallocate beforehand and having to use errors.Is), so I think that says something about returning error strings as a concept.

Exception object implementations are varied, but they usually need to have a stack trace. The common complaint is that they need heap allocations for composing the error messages and such. No difference here if you do formatting for everything, but you could define a bunch of strings in .data to mitigate that. Again, what do you plan on using all of that for?

In some languages you might even want to create new Exception types because at a high level you don't need to care about the internals of whatever it is you're calling, e.g. a controller shouldn't need to care about a DbError. I think here even that would fit.

If you want to use this in a web service in particular, there's another point that could be in favor of this:

Users don't need to know the precise error logs. But the server admin does.

"Precise error logs" in this model, then, is an explicit opt-in, rather than an opt-out. Let's assume the standard handler-repository pattern.

Here's some rough, contrived "repository" code. In an exception-laden language, you might want to try-and-catch at the most granular level. But sometimes you don't want to handle things like DbError which can happen at every step of the way.

// as in this tiny example the kind of error we're expecting is only
// a server-side error, we can use an Option[T] here.
Option[T] getPostDate(int postId)
/** error kind: internal **/
{
    db = getGlobalDatabase()
    try
    {
        q = db.prepareStatement(makePostQuery(postId))
        s = db.execute(q)
        r = db.getColumn(s, "post_date")
        try
        {
            return r.asDate().some()
        }
        catch ValueError as e
        {
            // you can tell the problem value here
            log.errorf("cannot parse post date: %s. value=%s", e.message, r)
            return none()
        }
    }
    catch DbError as e
    {
        // since this is a catch-all, we won't get which one
        // of these db calls (prepareStatement, execute, getRowCol)
        // errored out. so we might wanna print its stack trace anyway.
        log.errorf("%s", e.getStackTrace())
        log.errorf("db error: %s", e.message)
        return none()
    }
}

And here's the "handler" that takes this function.

void endpointGetPost(HttpServer s, int id)
{
    // assume the post's availability is handled differently
    date = {
        i = getPostDate(id)
        if (i.isNone())
        {
            log.errorf("unable to get post date of id %d", id)
            s.reply(status=500, body=makeErrorPage(500))
            return
        }
        i.get()
    }
    // ...
}

The user only sees the 500 error, while you see:

[!] cannot parse post date: Can't parse "null" as Date. value=null [controller/post:163]
[!] unable to get post date of id 19 [controller/post:145]

Or:

std/dbtools/private/backwarddb:140
std/dbtools:1100
fetch/object:154
[!] db error: The database server didn't respond [fetch/object:174]
[!] unable to get post date of id 19 [controller/post:145]

From the handler code's PoV, you as a caller don't need to know WHY the lower layer failed, like a DBError or whatever. But you want to know WHAT the effects of it is. like, whether it's an "internal error" or a "does not exist error", so you can return a 500 or a 404.

From the administrator's PoV, you want to know WHY the lower layer failed, because WHAT the effects of it is already reported to the user of your service.

But in case you're convinced this is a good idea, consider the following.

Again, you need some global log object that is always available, and accessible by every function. If you work in something that expects pure functions, tough luck, you need to pass that logger object around like hot potatoes.

For Outcome and Option[T] in particular, since they communicate ONE kind of error, you still need to diligently document that yourself.

// wait, does this error or not?
Option[T] derpity()
{

// ahh okay, i see.
Option[T] derpity()
/** error kind: internal **/
{

You find that you suddenly need to differentiate between an internal error and a validation error, but what you called was an Outcome or Option[T]. Okay, let's change that to DifferentiatedFail or Result[T, DifferentiatedFail]. Oops, turns out there's 10 other functions that use it, and you have to change them too. That's right, it's viral! But the solace is in the fact that it's not happening in libraries, only your own code. Besides, abstractions that work do tend to take a while.

Depending on how granular you made your functions, you can't easily shut off some errors from your callers. Your only option here is to set the log levels from the called functions themselves. Incidentally, that also means you can't just "swallow" errors like you might be tempted to do with reg'o exceptions.

And let's not forget the fact that you now have FIVE types to choose before writing anything, instead of just the one. In which case, you should refer to the flowchart of what to choose.

XKCD 2119.

I'll have to put this into practice for my aforementioned web app, and see how I like it. But until then it's just a theory.

A GAME THEORY. THANKS FOR FLAMING.

As I find more things to bitch about, this may expand. Couldn't be a Twitter thread because you can't see it anonymously without a proxy now.

Something must give when you join a "community".

I've had to leave my own principles at the door when joining a company, so that I may serve the company's principles. In that same way, some part of me must change when I'm in a community or a clique (let's be frank). Whether it be sooner or later. When I spend a lot of time with a community (especially if I find myself most comfortable in it), it can be quite a challenge to "be myself" again.

Something as simple as password hashing. At my job, my boss wanted me to use standard hashing algorithms on the basis that they are already well-proven cryptographically. Things like argon2 aren't needed, he says, SHA2 and up are already enough. Even when the algos are actually designed for passwords. At home, I made myself think twice before using argon2, despite being the thing it was made for.

It's probably because of wanting to avoid the "personality A with friend group B" problem. The less there is to remember the easier it will be for me. The horrifying part about it is that this can be socially engineered to steer people towards a certain personality and outlook, whether actually beneficial or not.

The one thing I hate about its logical conclusion is that I see a lot of people, in a bid to become "unique", actually be the opposite—they are carbon copies of each other. I don't mean that in the usual "oh everyone are sheeple and I'm the only enlightened one" but no, they are literally predictable. And it is so frustrating. I really do not want to lump people into stereotypes but here they are actively becoming the stereotype. I know some of why that is, but it still perplexes me.

Good art has the ability to successfully evoke—in others—the feeling, impression, or emotion that you want.

From what I've observed, it seems to fall within how successfully someone can imitate a certain style or aesthetic. Even when talking about what "traditionally" counts as art—you know, nature, anatomy, portraits and the like. In those cases you're considered "good" when you make something that to others are perfectly life-like. Exceptional, if your art is practically indistinguishable from a photograph.

In my neck of the woods, that's chiptune… Some people make chiptune that could pass off as a genuine chart-topper. No, the things that literally sound like Skrillex circa 2012 despite being made on an actual microwave. THAT kind of good. On another—retro design—some people make stuff perfectly embodying an era with none of the things that can give away that it's made today with rose-tinted glasses.

The people that I know who do this kinda thing do it consistently, I guess that's why everyone knows them. I admire them greatly, but also secretly jealous. The kind of drive needed for me to up my craft, even if I beat myself up constantly for not meeting their standards. In fact, to me they are unapproachable even when they otherwise aren't, or aren't meaning to be. I think to myself, "They must be inundated with utter peasants that I wouldn't even be worth their time", even when they aren't. Better safe than sorry.

So I found this video which claims to use a patched version of the Microsoft GS Wavetable Synth (MSGS), a.k.a the default Windows MIDI synth, a.k.a basically synonymous with MIDI at this point. Here the guy says that they've patched the driver to run at 48000 Hz instead of the 22050 Hz it came with. I looked through this guy's social media links and GitHub to find nothing of the sort, so I did what I always did and went "fine, I'll do it myself."

Okay, where do I start? For one, I'll have to figure out where the driver even is. As Windows XP is frequently said to be the best incarnation of MSGS (the newer versions apparently is said to have nerfed it for whatever reason), that's what I'll be using as a base.

From Device Manager, I looked for the Microsoft Kernel GS Wavetable Synthesizer. I might have needed to load a MIDI for it to appear, even with "Show hidden devices". Going to Driver Details I'd find that it's in C:\WINDOWS\system32\drivers\swmidi.sys. So I went there.

I extracted the file off the disk and opened it in Ghidra. As it is, it's just a bunch of meaningless data. Fortunately, there's always the public debug symbols for Windows that can help me—someone archived the ones for Windows XP.

Next, I did a Scalar Search for 22050, and it does seem to be used in a couple of routines. At this point I barely have any clue what I'm doing so I just went in and changed stuff to 44100 for a start.

This ControlLogic::SetSampleRate function is called by ControlLogic's constructor, so that must be where it was determined. The decompiler says this 22050 (0x5622) value is set in two different places, but the disassembly says this value ultimately comes from the EAX register, which was set earlier. So I changed that to 44100 (0xac44).

Upon saving and then replacing the driver in Safe Mode, Windows complained about not being able to load the driver, giving me a Code 39. Because this is Windows, no amount of search is going to help me on that one.

Until I realized that the PE checksum was a thing, and it wasn't adjusted yet. With the help of this FixPEChecksum, I got it working!

…Almost. Okay, what do I do now? Through certain hints, apparently I still had to change the audio format it's outputting through. It's a table (a bunch of data) called PinDigitalAudioFormats. There's two values near the end that is the same 0x5622 number I'm looking for. Except it's backwards, because in the actual data, this is little endian.

Bingo!. Compare that tiny bit to the vanilla output. Doesn't that sound like you unplugged your ears and cleaned out your ear wax? It's no SC-55, but it's close. And you get the bonus of preserving the quirks of this particular synth!

I managed to take it up a notch up and matched the guy's 48000 Hz sample rate, except that I needed to change ControlLogic::SetSampleRate because it clamps the sample rate to 44100, indicating a design assumption. Changing this clamp value "works", but some other assumption I think would be violated somewhere else. But again, it works.

…At least, that's what I thought at first. It was funny listening to tracks through this spruced-up MSGS as if it had come out of some other synth with a soundfont derived from GM.DLS.

I was then shown a couple of tracks that had pushed MSGS to its limits—and these were tracks that do not work anywhere else than with MSGS.

Strobe's track in particular exploits MSGS's aliasing to produce something that sounds a bit like extratone. Sound warning—and I had to abbreviate it—in order: 22khz, 44khz, and 48khz. The tone is heard most clearly for the 22khz version, still discernable but more painful for the 44khz version, and completely ruined for the 48khz version.

In the case of General Serum, it's not ruined that much (the 44khz version is used here), but there is enough differences in mixing/"EQ" the function of Windows XP not supporting multi-patch in one channel (which General Serum relies on) resulting in less punchy bass. The crunchiness of the original synth seems to have helped it.

Those two examples… made me realize something.

This is a small modification, and yet I have taken away a differentiating characteristic of this synth.

There exists MIDIs specifically making use of this quirk, and here I am "fixing" it.

The point of those MIDIs marked specifically as "MSGS MIDI" is that they only work with MSGS. On any other synth, it won't sound as magical—it won't have the "intended sound". At best, some instruments will be wrong, or the mixing will be different (as shown here). At worst, it will completely break or give your synth a heart attack trying to process it.

Effectively, I had made this version of MSGS cease to be MSGS.

The low sample rate is part of MSGS, and by removing it, it simply isn't MSGS anymore.

I'm reminded of Kyle's custom 2A03 chip. In that video, he explained how he fixed the triangle channel of the NES's 2A03 so that it's an actual triangle wave. It's an interesting and fun experiment, but a quirk of the NES had changed—it's possible that some track depended on the steppy wave of the 2A03, and now that it's smoothed out it doesn't quite "work" anymore, because it isn't a "2A03" anymore. In that case, because the NES and NES music means something specific, the modified chip (even if it's just an emulator patch) is "at fault" here.

Meanwhile, MSGS is one implementation of the overarching MIDI standard. MIDI and MIDI music means something more general. If MSGS specifically is exploited, it's an MSGS MIDI—different in category than a regular MIDI (good in all synths) and a MIDI for some x soundfont or other driver. I think MSGS MIDIs are more impressive because it:

• breaks people's perception of the "shitty, good-for-nothing synth" that is MSGS
• breaks on anything that isn't MSGS, ensuring its exclusivity.

But those tracks break with my modifications to MSGS. It's not MSGS anymore.

My so-called ““““““MSGS”””””” tracks… don't. Little to nothing is sacrificed if I play those tracks in any other MIDI player that supports GS properly. The MIDIs don't exploit anything in particular but the most basic of MSGS stuff, so it isn't as "exclusive". In this case, it's my tracks that are "at fault", not the synth per se—it's just another MIDI synth now.

In contrast to my previous MIDI post, this may be the usual case of "I think I knew something, but I actually didn't" (Dunning-Kruger, valley of despair, etc.), but the conclusion is that most of my ““““““MSGS”””””” MIDIs don't deserve that distinction yet. Because they are "not exclusive" to MSGS, because they don't break outside of MSGS. (So they're not really MSGS MIDIs are they?) As a result, I have stripped the "MSGS" qualifier on most of my MIDI uploads on YouTube. The "Carve Your Own Path" cover, I feel isn't compromised that much, but still will break on other synths, so I think I'm gonna keep it in.

There's also a reason they categorize the Commodore 64 SID as two different variants (generally speaking): the 6581 which has a volume bug exploited for samples (among other things), and the 8580 which fixed the bug and consequently broke samples. A track made for one may have an unintended sound when played with the other.

Maybe I'm wrong about this, and that extremely talented people, along with the specific circumstances of this synth, have massively warped my expectations to the point of questioning myself. I've "known" for a bit that I just make average MIDIs for your average synth/driver, but I think it's only now that I realized what that actually means.

I'm fine.

Oh, uh. Right.

You don't need a debugger for this because all that's been done, but you'll need a hex editor. Got your swmidi.sys (from Windows XP SP3 32-bit, specifically) ready? Good.

1. Change address 0x138 from 63 79 to CA 7B.
2. Change addresses 0x2AF2, 0xA75C, and 0xA760 from 22 56 to 44 AC.

1. Change address 0x138 from 63 79 to B9 B8.
2. Change address 0x2874 from 44 AC to 80 BB.
3. Change address 0x287E from 0A to 09.
4. Change addresses 0x2AF2, 0xA75C, and 0xA760 from 22 56 to 80 BB.

Because why the hell not.

I like it. Sure, it may have some surface-level goofiness, but I'd say it's been pleasant so far. A syntax as approachable as Python while remaining absolutely performant. It sounds too good to be true, but it really is that good.

The reference implementation of Nim compiles to C. "Hold on, but doesn't that make Nim Not a Real Compiler™?", you say. I like to think that Nim is cheating, but it's a good kind of cheating—piggybacking off of the many C compilers makes it possible to build things with it to all kinds of systems. And you get smooth interop!

Aside from the obvious "we made it work on microcontrollers", someone made it work on a GBA and developed a whole SDK for it. That made me wanna try it with the one (widely-used) C compiler that compiles to Game Boy, that is GBDK-2020. It's very specialized, and it does take a bit of muscle to get there, but it surprisingly works pretty well.

A bit of a disclaimer here, I've been using Nim for like a week at the time of writing, so I don't know everything there is to it. This is just my noobie take on doing a Thing™.

Since Nim compiles to C, it needs a nimbase.h to mostly help out with compiling for different platforms. Now there's no way in hell the default nimbase.h would ever work with SDCC (therefore GBDK), so I made a pretty minimal one that contains exactly only what's needed for the generated C code to compile.

First, I tried to echo something, thinking that it would magically work. Well, since Nim strings are not the same as strings in C, that doesn't work—when that's encountered, the Nim compile will bring in its entire standard library with it. Of course, that won't fly, considering the Game Boy isn't POSIX (or any kind of OS, really), and there's probably not enough useful memory to store all the necessary "safe" structs.

I then made a wrapper around printf. I referenced the Nim manual among other sources for some pointers on making C bindings. After taking a sample Makefile from the basic GBDK template and modifying it with some notes from Natu:

Okay, turns out it was simple enough. I then tried some basic constructs like this small routine:

proc printf*(s: cstring) {.importc, varargs, header:"<stdio.h>".}
proc delay*(s: uint16) {.importc, header:"<gb/gb.h>".}

proc addNumbers(): int =
  result = 0
  for i in 1..32:
    result += i
    printf "%d\n", result
    delay 43

Of course, that works as well:

You'll notice that it wraps around. That's because at this point (for whatever reason) I set NI (Nim's integer typedef in C) to signed char. It's a typedef anyway, can be changed at any time. So I continued.

Now, as I was trying out Nim's features, one that doesn't work well is sequences. Unlike arrays, sequences can be appended to. That means yet another special struct to deal with, so you'll have to provide ways for Nim to make one that fits the hardware (by defining newSeq), if you're using the os:standalone and mm:none compiler options, as I'm using here. I'm sure it can be done (GBDK's stdlib actually contains malloc among other things), and GC/RC can probably be enabled, but I don't really feel like it lol. Especially considering the small size of the Game Boy's WRAM... yeah.

One of my personal highlights is Nim's ability to define bit fields using sets. I really couldn't believe the following works, but it does:

proc printf*(s: cstring) {.importc, varargs, header:"<stdio.h>".}
proc joypad*(): set[JoypadKey] {.importc, header:"<gb/gb.h>".}

type
  JoypadKey* {.size:1.} = enum
    jRight,jLeft,jUp,jDown,jA,jB
    jSelect,jStart
  JoypadKeys = set[JoypadKey]

# ------------------------
while true:
  if jStart in joypad():
    printf "Start!\n"

It's beautifully self-explanatory. "Repeat this forever: if it detects that Start is among the buttons currently pressed, then do the following". And that's exactly what it looks like! This works also for detecting key combos: if joypad() == {jStart, jA, jUp}.

...Okay, maybe for that one you'd expect something like "START + A + UP", in which case, yes: if joypad() == ({jStart} + {jA} + {jUp}). It's a bit more noisy, but that'll do.

To help me aim while making more GBDK bindings, I decided to port some of the GBDK examples. I started out with the phys example, which is just moving a sprite around the screen with some momentum to make it not-so-robotic.

It was initially a straight port, including defining spriteData exactly like the C version. Again, making bindings were really straightforward, just need to map the names to the corresponding header file. c2nim exists for this exact purpose, but the thing is that GBDK's header files are very specialized towards SDCC. What with the BANKED, NONBANKED, CRITICAL and the rest of the keywords that c2nim doesn't recognize (if there's a way to set it then I just don't feel like finding out), I had to port them over manually.

One of the things I stumbled on was when I was creating the bindings for joypad_init and joypad_ex. At first, I thought that having the correct return type of joypad_init—which makes discarding the results of joypad_init necessary, since the phys demo calls it standalone—makes it not work, somehow. Nim doesn't like it when you call a proc (a function, really) that returns something, but then you don't use it.

I had a look at the C that it generated. Turns out it was wrapping joypad_init and joypad_ex into their own functions. This adds overhead that both of those functions weren't meant to sustain, and so it can't poll the joypad in time. What gives? Well... I soon realized it wasn't the return type that was the problem:

proc joypadInit*(npads: uint8, joypads: ptr Joypads): uint8 {.importc: "joypad_init", gb.}
proc joypadInit*(joypads: Joypads, npads: uint8): uint8 =
    joypadInit(npads, unsafeAddr joypads)

See that second proc? I was attempting to overload joypadInit so that I can use it Nim-style, like so: myJoypads.joypadInit(1). But in doing so, what making that a proc is doing is... well, exactly how the C outputted: wrapping the inner function. That's the problem right here.

What I should have used is template. It's like a more powerful version of #define FUNCTION(a,b) used in C and C++. You can use templates like a function, but Nim substitutes it at compile time so that it's actually the thing inside the template.

proc joypadInit*(npads: uint8, joypads: ptr Joypads): uint8 {.importc: "joypad_init", gb.}
template joypadInit*(joypads: Joypads, npads: uint8): uint8 =
    # now it works again
    joypadInit(npads, unsafeAddr joypads)

Emulating an incbin is also very much useful. This was my initial implementation:

{.emit:"#include <gbdk/incbin.h".}

{.emit:""""INCBIN(my_gfx, "res/my_gfx.2bpp")""".}
var myGfx {.importc:"my_gfx".}: uint8

Yeah... Hooking it up to GBDK's INCBIN define... not the best idea. I checked the Natu repo again, and found this. Now that's more like Nim! But... where does it read the file from? Apparently, the file where the template is defined, that's where. However, it's an easy fix: prepend the path of the currently compiling source file. So the new implementation is a template:

# adapted from Natu
template incBin*(filename: static string): untyped =
    const data = static:
        const str = staticRead(getProjectPath() & "/" & filename)
        var arr: array[str.len, uint8]
        for i, c in str:
            arr[i] = uint8(c)
        arr
    data

And I get to use it like incBin("../res/my_gfx.2bpp"). Sweet! BUT! There's a catch.

If I use let myGfx = incBin("../res/my_gfx.2bpp"), what that does is copy the entire file to memory! I've already mentioned the small size of the Game Boy's RAM, so yeah that's definitely a no-go. (That being said, I've dealt with cases like this, it's definitely a weird GBDK thing)

Nononono, what you wanna do is use const myGfx = incBin("../res/my_gfx.2bpp") instead.

In that same kinda vein, I'm debating whether or not I should also decouple constants in gbdk/io so that it's just raw addresses instead. But I've got a feeling that GBDK can optimize if it's given in its native define, so... I don't know.

Next up is the scroller example. Now, this demo uses a interrupt handler, and adding that handler requires the necessary code to be inside of a CRITICAL {} block, which SDCC interprets it as "enables should be disabled before running this, and then re-enabled afterwards". As you saw before, Nim has an emit pragma that can be used to manually insert stuff into the generated C code. But... where is it placing it, exactly? There's three options documented: type section, var section, and include section. None of them can be used, however, since I'm using it to wrap a function. I managed to implement a critical: macro, but as it says, there's a catch—have to use it inside a function that only has the critical block inside it. Everywhere else, you'll find a lone CRITICAL{} nowhere near your function. :/

Same. How about I try porting the GB ASM tutorial example instead? It looks kinda nice, I think. For my port, I called it "ReBRICK'D" because I'm so fuckin original. :p

Anyway, it's got all the things you'd expect, control a paddle, destroy bricks with the ball, earn the high score. I find enums to be very handy for organizing graphics:

type
  BgGfxParts = enum
    bgBorderTopLeft = 0'u8, bgBorderTop, bgBorderTopRight
    bgBlack, bgBorderLeft, bgPaddleLhalf, bgPaddleRhalf,
    bgBorderRight, bgWhite, bgBorderBottom

Which means I can write code like:

    # check for collision with the top of the screen 
    if currentCrossedTile.BgGfxParts in [
        bgBorderTopLeft, bgBorderTop, bgBorderTopRight
    ]:
        # ...

In assembly or even GBDK, you could pretty much do the same. It's just that I find it "friendlier" here (probably attributed more to the fact that I'm familiar with Python...)

Another useful feature is the ability to overload operators in Nim. That means I can hook up the score to automatically update it on the screen when it's being changed:

proc `+=`(a: var Score, b: int) =
    a = Score(a.int + b)
    updateScore()

So, adding 1 to a Score makes it call updateScore as well. As a side note, Python lets you overload existing operators as well, but it's not quite as "direct", as to overload + you'd have to define __add__.

I've made a main-game-loop sorta model here, where different parts of the game run in different modes. The game runs in a single loop that hands it off to the update routine of the currently-running game mode. Switching between them should automatically call the respective init function, before handing it off to the main loop again. I've got it handled with the switchGameMode proc, which is:

proc switchGameMode(mode: GameMode) =
  gameModeInits[mode.ord]()
  gameMode = mode

Which brings me to another feature of Nim. Unlike in Python, you can actually define custom operators! I wanted gameMode -> gmGame to mean exactly the proc above, so I replaced it with:

proc `->`(gameMode: var GameMode, newMode: GameMode) =
    gameModeInits[newMode.ord]()
    gameMode = newMode

I switched it back to switchGameMode, however—so, no custom operator. I felt like having this as an operator might reduce readability. But still, it's really cool that it's possible in Nim.

You'll notice that I used the movement code from GBDK's phys demo, which gives it a bit of a smooth-looking control (if not slippery). But that's the good news. The bad news is that this demo still has major bugs, notably in my really lazy eyeballed paddle collision (instead of just following the tutorial), and also the really wonky wall/brick collision. Out of 33 bricks, the highest I've managed to achieve is 29, before promptly getting stuck inside a wall. :p

Outside of playing, you'll find .2bpp files instead of neat little PNGs. At this point the Makefile is getting a bit crowded, and I'm too lazy to work in an asset conversion tool down the chain (png2asset converts it to .h files which I don't wanna have to import). Maybe for next time, when I convert the Makefile into Nimble tasks.

Besides the obvious performance hit you might get with GBDK, you might find yourself using ptr, addr and cast very often. Maybe a bit of type juggling, and looking at the generated C (or ASM) code when things go south (which is not pretty at all).

Also, the GBDK bindings aren't even complete. Maybe it'll be useable one day, but right now I don't think it's ready to rumble Nimble. I just ported over what I needed, and the documentation is near verbatim to the original GBDK (for convenience, should someone run nim doc --project docs.nim on the gbdk folder)

Not to mention, I haven't checked if banking is possible. It might be as simple as adding a codegenDecl pragma to mark it as BANKED, and then somehow adjusting lcc to make it compile at some bank, I don't even know.

But overall, it's been a pretty fun and satisfying experiment.

This is a continuation of a previous post on GBS ripping.

Ah, here we go again.

GBS is definitely one of the game music formats of all time. Other formats have some sort of tagging standard, whether it be a new format or an extension. NSF has NSFE and NSF2. SID has the STIL. What about GBS?

Let's look at the GBS specification for a moment. One thing you'll notice is that GBS leaves no space for anything else but the bare basics for song playback. There's not even any "reserved" space defined here, unlike those other formats. Basically no room for anything else, especially the possibility of tagging. In the meantime, here's the "standards" I'm currently aware of:

I've already touched on this on the aforementioned previous post, but here's a refresher for GBS rips:

• It's basically a 7zip archive.
• The archive contains the GBS itself, and a bunch of M3U playlists.
• The M3U playlists are plain text files using NEZPlug's "Extended Playlist" format as defined in the in_nez documentation.
• The M3U is split into a "main playlist" listing all the tracks in the GBS, and several M3U files for each of the tracks.
• The M3U has comments (usually only in the main playlist) that list information about the rip, for example:

  # @TITLE       Pokemon Picross
  # @ARTIST      Jupiter
  # @COMPOSER    Toshiyuki Ueno
  # @DATE        1999
  # @RIPPER      Zumi
  # @TAGGER      ppltoast, Zumi
  

  (Worth noting that that these comments are not part of the NEZPlug spec and isn't parsed by it—instead it's what's being used de-facto for some reason. Knurek's rips follow this format)

Now, this format assumes you're playing it using NEZplug and Winamp. While Winamp users are based llama lovers, who else uses it (and especially with NEZplug) to listen to some video game music? I feel like most people switched to foobar2000 and its Game Emu Player (foo_gep) plugin—me included.

There are some oddities of playing these kinds of M3U playlists with foo_gep. First, you don't drag the main M3U file into foobar. Instead, you drag the GBS rip and it'll pick up tags from the M3U file automatically. That's file—singular—file, because dragging the other individual M3U files will not work. Second, there seems to be indexing differences between the two, which is quite annoying. That is, song indices seem to start at 1, while NEZplug's documentation specifies that GBS stuff should be indexed at 0 (and in_nez does act accordingly).

Figured it was time to investigate…

foo_gep's backend is a library called Game_Music_Emu (GME), I believe originating from blargg with the latest version at 0.5.2, continued by kode54 (apparently at 0.6.x) and as of this writing the latest version is Michael Pyne's version 0.6.3.

It's a solid backend for playing video game music, but for this purpose, I care about how it reads M3U files. Some inspection (and experimentation with both foo_gep and GME's provided demo player) says that unlike NEZplug, this only has 1-indexed track numbers for every format. More consistent, sure, but it does present problems for people who still use NEZplug for whatever reason.

The sensible thing to do here is file a bug and ask if this is intended behavior. Problem is, the issues tab of the maintained repo seems to be completely overrun with spam.

Then I went to foo_gep and found out that it's an archived version (since the maintainer has quit from foobar2000 extensions) and its linked source code repo is a dead link. However—looking at the link through the Wayback Machine—I found a commit detailing about "fixing M3U with 1-based indexes into formats with 0-based indexes", but I can't look at the commit itself, so I'm not sure.

And I'm not about to fork GME and try to fix it myself, so I suppose it's time to look at some other formats.

Ew, VGM. Okay, so VGM is usually associated with Sega video game music… probably because there's not much else (GYM only does YM2612 and poorly) and the hardware can handle these fine. It's like the opposite situation where there's no reliable dumping format (except maybe for games that use SMPS) so this is the next best thing.

VGM has support for several chips but is more of a register dump / log. Yes, it has support even for systems that already have ripping formats like the NES (nsf) and Game Boy (gbs). Seeing VGM "rips" for those systems feels cursed. (Apparently it's less preferred anyway since emulation bugs may creep in to the VGM log, though hardware playback may help)

But the one thing VGM has is an actual, "well-formed" tagging system known as GD3. Consistent, somewhat reliable, covers most of the relevant bases.

Again, VGM is kinda… eh (and I also don't currently have emulators to dump them in the first place). It's understandable why it's been a popular choice—you don't need to disassemble or hunt down sound engine playback code, so they're easy to make. Just hit "Dump VGM", wait, and perform post-processing like finding looping points and whatnot.

An idea I had is to provide VGM with the M3U+GBS, but it's not a good idea, I think. Why provide two copies of the same thing in one package?

There's yet another option to consider, however…

GBSX was the proposed extended header format for GBS files. It was supported by GBSPlay up until commit eaaa3714 (2021-01-06), after which it was removed due to lack of any traction. Sad, really.

I came up with my own GBS tag format but it was really a poor ID3 ripoff and had zero proof-of-concept players.

Today, GBSPlay still supports loading tags, but only for VGM. (GD3 tags are only read when loading a VGM)

To think that literally nobody is even aware of GBSX, which is such a shame as this could have been very useful. It may not be perfect, it may contain more basic info, but goddamn it would be much better than having to load up real ancient plugins!!

So, I was determined to make this work.

I checked out and built the final commit that had GBSPlay support and cooked up a script that would embed GBSX tags in the GBS file. Now, a few comments on building these tags:

1. The spec says that GBSX can be separated into its own file, and there's a proposal on how one can do that. However, I can't seem to find any indication that this is actually implemented into GBSPlay. Disappointing, but I'm fine with embedding it into the GBS anyway.
2. Seems like extra hassle to require serializing a CRC32 checksum. I get why it's in the design, but like… why?

   For reference—CRC32 checksums (standard algorithm, like ZIP) are needed for the original GBS file and the entire extended header with its CRC slot zeroed out.

3. Adding the GBSX header requires padding the GBS length to some multiple of 16. The offset to the GBSX, minus the 0x70 of the GBS's standard header, then divided by 0x10—is then written to at address 0x6e. (For whatever reason, before writing it, 0x100 seems to be added to this value)
4. (Insert something about encodings here). Speaking of encodings, I just said fuck it and used \r\n for the new line endings.

After all that:

Not much info is displayed. But it's a good sign. Although yes, it seems to ignore the track time and fades out where it would be by default. I'm not sure why, but whatever, at least it actually reads them.

So, knowing what I know now I decided to try these three things to my new distribution—a rip of the unreleased Katakis 3D (done by zlago):

1. Add GBSX tags to the GBS rip. Any player (if any) can read these tags if M3U support is not available.
2. Use 1-indexed track numbers in the main M3U playlist. The rationale is that foobar2000 reads this file as 1-indexed. Yes, it will assign the wrong tracks when loaded in NEZPlug.
3. Use 0-indexed track numbers in the individual M3U playlists. Rationale being that only NEZPlug can load these files, so… drag all of them into Winamp than the one main M3U I guess.

Last two may make the distribution feel cursed, but I'm trying dammit lmao.

As before, I generate these with the help of Python script to process a few JSON files to make the process more streamlined. Here's the tools used for this rip, a bit more versatile than the one I used previously: gbstools

The tools have only two real dependencies: jsonschema and py7zr, both of which are a pip install away.

This is a continuation of my previous post on MIDI. Now originally I was gonna make two continuation posts, but I think I'll combine it into one long post.

So here are some notes of interest I had when writing more covers for my MIDI page. They're not much, but I just thought I'd write it anyway. In advance, sorry this section is so short. I wrote it a long time ago, so the excruciating details might not come back to mind as of yet.

Drum patch 24 (electronic) is used here to approximate the original samples, assuming the "default MIDI soundset" (as described in the first MIDI post) is used. Unfortunately, since bg-sound operates from Gravis Ultrasound patches (and therefore a GUS approximation of gm.dls), you won't hear this drum set in the browser and instead get the standard one.

Y'know, it always felt kind of odd to me that Roland's definition of an Acoustic Snare in this patch sounded much more like a tom—and so is the bass drum. It's nearly as odd as the presence of two power-pads and sound effects in the GM instrument set.

There is a bit of an oversight in the program changes on one of the echo channels, but I don't feel like fixing it. Now that I let you know, you'll probably gonna want to notice it, if you haven't before.

What I noticed about this song is that it's got differing time signatures. The intro and the end has a time signature of 3/4, while the main part has a 4/4 time signature. In this case, it's an opportunity to insert time signature meta events.

It's interesting how different players deal with the time signature meta. I believe I inserted it only on the first channel, the same one where tempo changes are defined. Anvil Studio inserts this meta event on every channel, SynthFont 1 ignores the mid-song time signature change, while Rosegarden did just fine.

This was a request, and I think I'm happy with it. Generally I've just experimented a little to see what sounds would fit the best for the samples. But these are just approximations. I've said while making this, that the more I work with MIDI the more I find myself sympathizing with Neil Biggin due to its admittedly limited palette, and the electronic instruments being stuck in italo disco forever.

Maybe you should use a different patch for that lead, it sounds different.

Huh? This one sounds pretty close...

Not with my soundfont.

grr...

Also at the time of making, I'd say I've outdone myself here, mainly because the source material is just so rich with sound that even a "watered down" MIDI version of it still sounds pretty nice. It's so fire that Windows Media Player can't even handle the large amounts of polyphony in that thing. ...Then again, that might just be Windows 10's reduced polyphony here.

I also found that the master pitch in FL Studio actually shifts all pitch slides in the MIDI by that amount, which is handy considering all the samples in this module are tuned slightly higher. While making this cover, I actually had to tune down all the samples in the original s3m.

I've thought of using a meta message to set the tuning, but as with everything, I'm not sure if soundfont engines support that. So a global pitch slide it is.

For these ones I looked into MIDI lyrics some more. This isn't the first time I inserted lyrics into MIDI, that one was the cover of Oceania (Literally 1984). I found Rosegarden to be useful here because you can insert syllables for every note in one channel.

As for how it's typed, there's different ways. I tried newlines, dashes after each syllable, spaces... The one that seems to work is "new line before every phrase and spaces after every word". Whether or not dashes after syllables are removed seems to depend on the particular player, but generally I find they lack dashes anyway.

Then I found out several different karaoke formats exist. Oh, MIDI.

SoftKaraoke's first track must be text events that mark it as a KMIDI KARAOKE FILE, and its second track must be lyrical content. (I don't really agree with splitting lyrics onto its own track, would much prefer it being in the same track as the melody...)

For splitting new lines, the SoftKaraoke format specifies the \ (back slash) character should be used.

VanBasco's player seems to have the best tolerance for multiple karaoke formats, though it still acts weird with multiple lyrics channels like the one in the Twitter for Android cover (despite having "allow multiple karaoke channels" checked), since it just mashes the two together leaving a confusing mess. Maybe it's not good practice, anyway.

Meanwhile, Midica isn't quite as "tolerant" as VanBasco. If it doesn't see a SoftKaraoke style header/format it doesn't split the slashes into new lines. It also parses 0x0D \r as "down one line", while 0x0A \n is "down two lines".

The karaoke formats thread had this sad bit that I think can be noted:

I'm not aware of any MMA compliant software apart from my own: Midica.
At least I think that Midica is compliant to RC-026. I cannot test it due to a lack of compliant example files.

Anyway, playing with newlines like that unfortunately also breaks Serenade:

There's another software called WinKaraoke and it has this neat bouncing ball thing, but it seems to be only compatible with SoftKaraoke files. My "karaoke" MIDIs aren't real "karaoke" MIDIs, so no bouncing ball for them. 😔

I did that drum stick 1.. 2.. 1, 2, 3, 4 thing with the Losers Club cover to emulate what a "real" karaoke MIDI would do, but now I know why that was done. Apparently I can't just jump straight into it, because VanBasco cuts the first syllable of the Oceania cover. That didn't happen with TiMidity.

Then outta nowhere, there's YAOTA by Kot. Somehow EDM is possible using only the Windows default MIDI synth. Yeah. My initial reaction is that this midi was being made at high speed, everyone else who listened to it thought it was filters. But nope. It was recorded straight outta Windows Media Player. You can't imagine such a thing coming from MIDI, but here we are.

But as it turns out, the MIDI was made at a reasonable BPM (180, to be exact). The real meat however, lies in CC event spam and the wider range of sounds made available through the GS standard (that's the "GS" in "Microsoft GS Wavetable Synth"), of which it takes advantage of through clever sound design. For example:

• There's a bass sound using the machine gun SFX and the baritone sax to emulate rough bass synths.
• Sine waves for kick + click SFX to give it that extra punch.
• Masterful use of pitch bends everywhere. And I do mean everywhere, including the drums.
• Really short note lengths for some other harsh synth sounds.

The end result is pretty mindblowing. But quoting the track's author, "that's nothing compared to HueArme". And indeed, HueArme's work is even more mindblowing, and more so because their entire discography is nothing but MSGS that sounds a lot more like DAW output. Who knew you can squeeze this much out of such a shit synth that even Microsoft left to rot. Here's hoping there's a "MSGS scene" somewhere (probably there already is but it's just really obscure).

Well, even though I'm nowhere near their levels, I'd thought I'd still try and up my own game. And so I learned some new stuff to add to my MIDI-making toolbox: the expression CC and SysEx.

The expression CC (CC #11) is yet another way to control MIDI volume. Unlike velocity, which is set per note, this CC is continuous. Other advantages by using this CC is that you can simply let the track volume be, which saves the headache of having to adjust dynamic track volumes. In this case, you can use it for a sidechaining effect by fading it from 0 to 127 every beat.

The next thing is SysEx. I've always avoided it since I didn't think it to be all that relevant, but in this case, it is! MSGS actually parses this to permit stuff like drum channels other than ch. 10 and bank switching. And you want that if you want to use the full GS palette.

There are several others that I tried, but apparently not all synths support these:

• The filter CC's, seems to have resonance (CC #71) and cutoff (CC #74). If a synth supports these, even richer sounds can be made, and more opportunity to abuse patches to sound like something they're not.
• Envelope CC's, usually attack (CC #73) and release (CC #72). Assuming the samples in the soundfont or DLS aren't hardbaked to have any envelopes (not a good idea either way), you can modify how fast it fades in. Again, can be useful for helping to make wild sounds out of patches that are not even intended for it.

As far as I know, BASSMIDI VSTi does support these. But not with MSGS. I have yet to test if VirtualMIDISynth has support, but I'm gonna guess it does.

And now, to apply them. Trying my existing tooling first, here's what I came up with—a short snippet of Carpenter Brut's Roller Mobster. I daisy-chained a bunch of automation clips together and applied it to a bunch of MIDI Out controls. I have two automation clips: one which is just the sidechaining stuff, and then the other which is a slow fade out for the sweeping down SFX.

I added CC #11 to the Midi Out channels I wanted, then using the Link to controller... option I linked the lead channels to the Sidechain automation clip directly. For the sweep channel I linked it to the Sweep automation clip, but not before modifying the Sweep clip by linking its max level to the Sidechain automation channel. In effect, it's like multiplying the Sidechain by the Sweep, because I don't think you can do that directly. I think I did try X-Y controller and the Peak Controller to achieve the same goal, but it just ended up being a barely-working mess.

If I were to draw a rough graph of that setup, it'd look somewhat like this:

Now, this approach matches what you would actually do if you were making EDM (and not stinky MIDI files). It's quite easy to control. The downside however, is that it will generate very granular CC event spam since each change gets resolved into something like a single tick in the MIDI file. You can see it here, where the output is really smooth:

As for SysEx... Haha. No. Coupled with Midi Out not being able to perform bank switches (you can have only one patch bank for the entire song), this effectively locks me using General MIDI. Which is good for interoperability (and what MIDI is really designed for), but certainly limits the possible timbres you can use. If you're actually creative and don't have any skill issues (unlike me), then you definitely could work with it and make a masterpiece out of it.

For me though, I'm just gonna try something else.

The other choice I have here is Rosegarden, and that's a native app. But Rosegarden is just... weird to work with. I just use it to clean up MIDIs after they've been created with FL Studio and then export it again to be postprocessed with midicsv and back again. I'm not confident enough in using it as a main authoring tool.

And while we're at it, let's talk synths shall we? The only two software synths natively available as a MIDI driver on my computer are TiMidity and FluidSynth. I tried playing YAOTA through both of them. Sadly, neither can seem to deliver GS well.

In TiMidity's case, it can process GS SysEx in file mode, so YAOTA's bank switching and extra drums work. However, it doesn't seem to take them when using it as a MIDI port, so I just hear the basic square chorused lead instead of actual square waves and sine waves. Either that or it's a problem with Wine's midi drivers. Also for some reason, when using any gm.dls-derived soundfont, its drums are at a significantly lower pitch than usual. This one I do know is a problem with TiMidity

As for Fluidsynth, it handles soundfont volumes and pitches better, but I can't get it to read SysEx in midi files even when I forced it somehow (tried synth.midi-bank-select=gs). I haven't tried using it as a MIDI port but I doubt it'll get any better.

Also, as briefly mentioned, Windows 10's MSGS is simply nerfed. Lower polyphony limit and seemingly more lag. VirtualMIDISynth is an excellent substitute, but I wanted the distorted sound of the ancient MSGS.

So, a virtual machine running Windows XP it is.

Kot told me they used Domino, a Japanese MIDI authoring tool. I've heard of it before from the Black MIDI community favoring it due to its speed, but Kot did tell me that it does make SysEx handling quite easy.

The first hurdle is finding a translation, since this program's entirely in Japanese. There's the version 1.44 English translation, but apparently the version 1.43 translation is more complete. At first I'd assumed that each translation builds off of each other, but these seem to be separate. Either way, they're really hit-or-miss, but at least I've got some idea of what I'm doing.

The second is, well, just getting used to it. Mostly ways to work with notes. Found some notable features by reading the also best-effort translated manual, like double right-clicking on the piano roll switches you between select mode and drawing mode. Multiselection is done with the Ctrl key. I do find it funny how it can draw multiple selection boxes on top of others, though.

Domino definitely gives off the impression of a first-class MIDI editor. You have the option of drawing several types of curves in the event editor, changing playback speed, and I especially love the "mass change" feature that lets you change values (including notes and tick offsets) by giving it a math expression (say, 60%+10 velocity).

But the real gold is being able to set a "sound source definition", which unlocks the possibility of being able to set SysEx events easily through Domino macros... or whatever it is. When you set a definition through the Midi out settings (F12), it can start a new project with a couple of these already added so you can make use of it immediately. You don't even need to touch any hex data or a MIDI implementation details sheet, just click on the presets on the left of the window (if you can even make out what it's trying to say...). As Kot said, it did make SysEx stuff trivial.

I figured Carve Your Own Path would make an excellent choice to start flexing what I learned. Considering that at the time of creationm Furnace (which is what was used to make it) had no meaningful Real™ export option for Namco 163 (besides wav), so it had to be placed in "allgear", where stuff literally made in DAWs can make it in. So it makes it all the more impressive that it got top 5 in that category.

And yes, the irony of covering a song called "Carve Your Own Path" (which itself already has several really cool covers) is not lost on me.

Seems more like monkey see, monkey do than using creative energy if you ask me.
—Inspector Gadget, "Minecraft with Gadget"

Either way, I was quite determined to make it my test case. Check it out:

I would've put this in my MIDI page, but considering what I said about synths above... yeah. I'll need to wait until I implement something that excludes some tracks from playing in the browser.

For now, uhh... I don't know where to begin, really. Maybe let's go track-by-track. For starters, if you inspect the MIDI yourself, I named each track based on what it's used for, so it can give you some hints as to where to start looking. I'll still explain, for the heck of it:

1. Bass, sidechained rubber bass. Applied the equivalent of a 12 0 arp macro on a typical MML or chiptune tracker here.

2. Kick+sub, pure sine wave. I'm pretty sure this drives the entire song, as should be the case with any typical EDM (where kicks must take center stage). Whereas YAOTA has the kick and bass in different tracks, I combined both kick and sub bass in one track since the bass is on the off-beats.

   I know there's some songs that make use of a reversed kick drum, and they sound great. So I added some here, and I think it works just as well.

3. Chords1, sidechained pure saw wave. Originally I was just gonna leave it as plain notes, but I decided to add a rising pitch bend to the start of each measure to spice things up a bit.

4. Chords2, sidechained pure chorused wave. Most of the time it's a copy of Chords1. I silenced it during the solo to increase the prominence of the piano and bass. It didn't contribute much to the sound at that part anyway.

5. triangle, pure sine wave. Its channel allocation is shared with the Kick+sub channel, so there may be some cuts. Sloppily extended some parts.

6. Square thing, sidechained pure square waves. The clav sounded similar to a 12.5% square wave, so I used that alternating with the 50% square found in bank 1. I had some fun with the panning here. Also applied the arp trick here, too.

7. Square thing echo, sidechained pure square waves. Ditto

8. Lead, multiduty. What it says on the tin. Tried to go a little wild with the pitch bends here, as the original also kinda went to town with it too. I'm happy with what I did with the Expression CC during the solo, was trying to simulate what an actual sax or trumpet player might do, where the volume goes down and then up again.

9. Lead echo, multiduty. Ditto, except delayed by a couple of ticks. During the solo, I removed the delay and made it exactly an octave higher than the lead to act as a second voice.

10. Something, multiduty. Meant as a sort of additional accompaniment. Some parts are inspired by Zyl's cover and others are analogous to components in the original.

11. Snare, percussion. Placed in Ch. 10. Well, this is where the snare was, but I was too lazy to change the name. Instead you have a bunch of clicks to increase prominence of the kick, some claps, and a weird riser made of open triangles (found through trial and error)

12. hi hat2, percussion. Placed in Ch. 11. Hey, here's where the snare went! I tried the same pitch bended drums trick as YAOTA but clearly it's a bit more amateur. But hey, it kinda works.

13. regular drums, percussion. Placed in Ch. 13. Yes, these are everyday boring MSGS drums. Limited to cymbals and hi-hats and the occassional agogo.

14. woosh, sidechained SFX. Woosh indeed. Here, I tried the automation trick described above but manually. First, by drawing the base curve where it rises and falls, and then drawing the sidechain curves manually such that its maximum on every beat corresponds to wherever the base curve falls at that point in time, giving this effect:

15. solo piano, sidechained. Really just repeating the saw chords in a more "interesting" pattern. Sidechain is only half the time, though—I wanted the low notes to be heard, I guess.

There's also the part where all sound stops for a quarter of a beat (made by setting the expression CC to 0), that was also quite interesting to try. Hopefully it adds to the EDM kinda vibe, although at the same time it still sounds very much like a MIDI. Yeah, skill issue on my end here.

And as I've said, the problem with using MSGS to make "impressive" sounds is that it only works with MSGS. And again, VirtualMIDISynth with GMGSx.sf2 or Scc1t2.sf2 can absolutely handle it. I'm afraid I can't say the same with other engines or soundfonts, since... y'know, this isn't what MIDI is designed for.

OpenMPT seems to handle YAOTA fine, but some events have reduced accuracy when I try it with mine, even at the fastest import speed settings. Probably because YAOTA is more polished than my attempt lol.

Although I said in the last post that I assume everyone's gonna be using Windows Media Player to play my MIDIs, I've always treated MIDI as the "universal standard" it is and... you know, actually consider situations like soundfonts not having any SFX patches, anticipate shitty pitch bend playback (stares at FL Studio), attempting to tag it properly, etc. But the thing is, MIDI is an entire rabbit hole. The fact that there are several different ways of playing the same thing that are "supposed" to sound the same but it doesn't. Several competing standards all based off one file standard based off of a unified protocol.

MIDI definitely ain't no HTML, yet I somehow treat it as such. The parallels of different user agents receiving the same standardized data and what to do with it. Pulling off something with MSGS feels like using Tailwind, tossing print and other media as well as responsive design down the bin, or tagging for the OSM renderer...

But in this case, it's oddly worth it. Like chiptune (think NES, C64, Amiga), you're working with only a few soundsets, and those soundsets only, which gives you some freedom to go absolutely wild with it, with the assurance that it's gonna sound—mostly—the same everywhere. No need to think about how other playback methods may mangle your song, as far as you're concerned it's their problem, not yours.

Overall, I've enjoyed the process, and I'm quite happy with the result. That being said, is a comparison between "Best played with MSGS" and "Best viewed with Internet Explorer Netscape Chrome" fair?

RGBDS is considered the premier assembler for developing games for the Game Boy, indeed it's seen popular usage and has grown into a powerful suite over the many years it has existed. It's pretty accessible, and there's loads of genuinely useful things in there, including native character maps, rgbgfx for png-to-2bpp conversion, easy linker file syntax, among other QoL improvements. But as much as I love it, I feel like something weird is up. The same software culture being applied here, rapid releases which deprecate a lot of stuff really quickly. That becomes a problem when dealing with old projects which I may not have the time to rewrite... funny how it goes needing to rewrite an assembly program just to keep up with the pace of things!

Despite the main GB dev community positioning RGBDS as The Standard Environment™ for developing Game Boy programs, several alternatives exist. They may or may not be of taste if you're coming from RGBDS like I am, and some even require different assumptions on assembly programming. Just cursory comparisons, and may not even be fair to their actual capabilities. I'm also not really looking at things like ROM mapping and symbol file creation (useful for debugging). If you're using one of these in a pipeline, your mileage may vary.

A multi-arch assembler, aiming to be portable. Z80 (and Game Boy ASM) are included as an official module among many other architectures including well-known ones like 6502, M68k and PowerPC, as well as some out-there ones like RPi VideoCore, PDP-11 and the TR3200.

For an assembler, it's got quite a choice when it comes to number prefixes. You can prefix hexadecimal not only with $, but 0x and &. You can also do 2Ah instead of $2A. In other words, it's got something for everyone.

Its docs describe the basic syntax of things, but not much examples. It's probably just me, but I'm having a bit of trouble with what's supposed to be bss sections that doesn't start at 0 (like WRAM needing to start at $C000), so I'm using "virtual sections" provided by the dsect feature. Also with regards to wrangling the sections to fit the Game Boy's memory layout and bank switching, I'm using rorg for them instead.

Since org specifies entirely new sections, I decided to use align 3 for all the rst and interrupt vectors. align 14 is quite useful for filling in the rest of the bank with a huge swath of nothing.

It's worth noting that ChibiAkumas / Akuyou uses this assembler for his delightfully multi-platform assembly tutorials.

Sample code here.

Another multi-arch assembler, written by you-know-who. Supports the Game Boy target architecture under the name of gb.cpu. Aims to be a straight-forward, no-frills, extensible assembler. No linkers, etc, just a plain assembly compiler seemingly following the Unix philosophy. Perfect if you want to make patch ROM hacks (since there's a "modify" switch), though it's also because of this also that you might need to do manual linking.

Funny now that I've said I was trying to use RGBDS for binary hacking, then this exists…

Compared to the others, the writing style for this target is more C-influenced. There's things like scoping, functions as assembler directives, and then some. It supports anonymous local labels up to two within a single scope (any more than that and it refuses to work, because your code smells, probably). Defining memory is a bit tricky here. It does, however, support expressions like while, which is useful for making unrolled loops. Be wary of gotchas like the fact you need to skip the spaces when doing indirect addressing. Also, I can't seem to find an include directive?* At least, not what I think might be.

All in all, if you target this assembler, coding feels a bit like you're writing inlined assembly in C. You may need to get or write helper tools if you want to work with it, especially for header checksums.

Sample code here.

NOTE: I've been told that the include equivalent is called insert. I'll want to try this out later.

Yet another multi-arch assembler. But it's a very popular alternative for RGBDS, and for good reason. Although Game Boy ASM is just one of the many architectures it supports, it doesn't just support its assembly language — but also has its own specific features — like being able to define headers easily through .gbheader and automatically calculating checksums at link time.

Although I probably shouldn't be surprised, since this was originally a Game Boy development suite, which then was extended to all sorts of systems because it was that good. SNES developers can enjoy the same luxuries WLA-DX offers, since it supports all 3 architectures you have to program for it: 65816, SPC-700, and SuperFX.

It has support for all sorts of funny memory layouts. Although it can feel a bit daunting about having to manage these things, it makes it possible for you to specify weird memory layouts if you have to program some other funny systems. It also has a lot of things that RGBDS even aspires to have, like structs (very much useful for virtual sprite management, inventories, or party member data), tables (useful for quickly asserting some fixed data structure), and frankly a lot of other things.

There's loads of options if you want labels. Although local labels are specified with a _ prefix, you can have multiple levels of "local" by specifying a number of @ prefixes. Sometimes, a piece of code may have multiple different parts. A one-level local label is bound to have names like: .okay, yep, .ok2, .done, which may be confusing in a large file. Multi-level labels might be able to help with this. Anonymous local labels are also available, useful for simple loops.

Point is, this thing is all about control. It's got excellent documentation, and I do recommend reading it. It's got explanations of everything and some examples, which helps if you want to unleash the full power of this beast.

Sample code here.

This is where it all began. Or at least, part of where it all began. RGBDS's history notes that SurfSmurf / Carsten Sørensen developed ASMotor first in 1997, and then in 1999, Otaku no Zoku / Justin Lloyd uses ASMotor to create a compiler for Game Boy assembly, called RGBDS.

Looking around a bit, it would seem that things are a bit confused. Straight from the horse's mouth, it appears that (like WLA-DX) Sørensen made RGBDS first, then extended it as a suite of assemblers called ASMotor. The RGBASM (xAsm) history page mentions 1996-10-01 as the first release, while first ASMotor release is specified as 1997-07-03. Perhaps the fact that ASMotor (and xSomething) is mentioned more than RGBDS gave the impression that ASMotor came first. If it's not that, then an alternative interpretation of ASMotor being RGBDS's "spiritual successor" possibly could be the fact that ASMotor supports more architectures than just the Game Boy. There's also this thread in which both Sørensen and Lloyd explains it from their own sides.

Either way, ASMotor today is very similar to how RGBDS was in its early days. Whereas RGBDS as we know it now deviates a ton and is a lot more modern, ASMotor remains "stable" and faithful to its roots… perhaps a little too faithful.

A comparison to a modern RGBDS workflow is inevitable. For starters, if you want each section to be in its own bank, you'll have to specify them manually (as with other assemblers). In this regard, rgblink has the upper hand, since its link file can also double as a bird's eye view of the ROM. Some oddities exist, like that endm has to be indented first. Syntax differs still, such as jp [hl] but not jp hl, and ldi [hl], a ld [hl+], a but not ld [hli], a. And as described in my sample code, unused space between sections will always be filled with FF regardless of what you set with -z, unless you explicitly use ds to specify unallocated space. Group names like HOME and CODE do need to be in all caps (HOME == ROM0, while CODE and DATA == ROMX). Also, importing/exporting symbols manually is a thing you do here, though you can also use the global export marker ::.

Symbol files are generated with xLink's -m flag, and while its syntax is compatible with bgb's implementation of the .sym format, not every label is present — seems to be only the ones that are used somewhere else. This does not appear to include local labels.

Its docs help a little bit, but they fall a little short. For example, the HRAM and VRAM group symbols don't appear to be documented, and the closest thing would be the GB examples that Sørensen gives in a separate repo. Even the old docs were more complete. Although unlike the old xLink, modern xLink doesn't seem to accept WLA-styled linkfiles anymore.

There is, however, one cutting edge of today's ASMotor: the ability to inline code and data literals. So instead of

Test:
    ld de, Text
    ld hl, wTilemap + (20 * 2) + 5
    jp PrintText

Text:
    db "Hello", 0

you can do:

Test:
    ld de, { db "Hello", 0 }
    ld hl, wTilemap + (20 * 2) + 5
    jp PrintText

Very much useful if all you want is to display text and not have to name every single string that appears.

Sørensen also provides xgbfix separately as part of the ASMotor suite. If you do want to ditch RGBDS entirely (and you're also familiar with the RGBDS of old), this should be a valid choice since all you need (except for rgbgfx) has its equivalents here.

Sample code here

OVERALL NOTE: If you're feeling salty, right click and view page source to read what I really feel…

I work with multiple devices, and I need to work on something using any of those devices at any one time. Sure, I could copy the files, but how do I make sure that any change I make won't conflict with one another? For most people, this is a solved problem. They can sync up anything just by using Google Drive or version control systems or something. In fact, who needs file systems when you've got the Cloud™, and it's very convenient?

But… what happens if I don't want to use third party services? Maybe I have some something that's more "private" that I don't wanna risk getting leaked out because it's on the Internet, even if vendors say that they have the strongest locks ever totally you guys, pinky-promise.

Sure, I can do it offline, and I have, via a USB flash drive. But the thing with them is that it doesn't "feel quite at home" having my projects all stored in there, and since they're so small it might be easy to lose them. And that's when I thought of self-hosting.

Self-hosting is kind of a noble thing to aspire for, isn't it? The idea of being on your own two feet and all that. Well… it gets pretty expensive, especially if you want to make sure that everything is yours. A decent VPS that lets me do things is not quite in my budget, and so would a Raspberry Pi. And even then, there would be concerns with both: A VPS I'd still rent from someone else (i.e. I'm still trusting the vendor with my data), and going with a Raspi server from my home Wi-Fi isn't guaranteed to be reliable anyway, in terms of accessing it from outside.

Given that, I'd thought I'd try some other compromise… using my phone as a server. I didn't think it would be a walk in the park, but I didn't really expect to jump through so many hoops as well. So the thing is that this is not a "public" file server, so it might be alright to just host something on a local Wi-Fi. The mobile hotspot feature seems fitting here.

There are some apps on my phone which do provide some sort of server functionality, like Amaze which has the ability to start an FTP server. But syncing up project files through FTP isn't very convenient and feels clunky. Plus, I might need to set up a server both on my computer and on my phone, which might not always be possible.

And then I thought of Git. I knew that it can basically do retrieve-and-sync from some upstream, and that anything can set up a Git remote that everyone can pull and push to, so why not try just that? I have Termux installed, which lets me have a very Unix-like environment within Android, and which has a wide variety of packages, among them Git (and SSH, which will be used for authentication).

On the phone, installing those works as you'd expect. Create a new directory, run git init --bare on it, set up SSH and starting sshd. So far so good.

What to do next should be straightforward… on a computer.

But, because it's a phone, of course it's gonna be a trial by fire because of it wanting to lock everything down. For one, look at how the home directory is represented:

/data/data/com.termux/files/home

Since this is where everything accessible is under, I have to type this whole thing everytime I do something via SSH that has something to do with files in the home directory. Which, speaking of, is the place I dumped git-bare folders on. Just to add to that, they use a different port for SSH. Not 22, but 8022. So that's another thing to remember: to use -P 8022 and explicitly specify that port number in Git URLs.

Right, so I've generated my key and copied it over to my phone, minding that port number. But then... how do I clone it? The thing is, I need to know what my phone's IP was when devices are connected to its hotspot. Running a few localhost services myself, I found that out initially by accessing my computer's local IP address and then checking the site access logs. Quite roundabout, but gets the job done.

Once I found out my phone's IP address, I could then run a git clone from that IP, followed by the port number, followed by that annoying directory name and then the git repo. After that, I could just push and pull to my git repo on my phone, so long as I'm connected to its hotspot. Fortunately, it doesn't change every time I deactivate the hotspot.

But unfortunately, it will whenever I restart the phone. Another way of illustrating how phones (especially in Android) lock people down more and more each passing year. Obviously, this means that it might not be a good long term solution, but I'll want to force this, whatever it takes.

It was at this point that I remembered to use the ipconfig command to check what my hotspot's IP address is, and then just use it in the git remote configs. But then I have the problem of using raw IP addresses on disparate git folders, in disparate machines. So then if my phone restarts, I'd have to change all of these addresses.

And then, I remembered another trick: using the /etc/hosts file to specify my phone's IP address and "alias" it into some readable "domain name". Well, not exactly in the sense of something.com, but more along the lines of myphone. Since I often dualboot, this configuration needs to be synced manually across all machines twice.

All of this works, of course, in the hackiest possible way. It feels like quite an event whenever I push or pull from the git repo hosted on my phone, but other than the hellish maintenance… it works? Kind of? I feel like… if anything, it goes to show how much more limited phones are by design, unless you root it or jailbreak it (which comes with its own risks but hey).

It doesn't have to be this way, right? Can users be trusted to do anything? I know, security risks and how unlikely your average Joe will hack around to make his devices do what he wants by any means possible… but, if you declare the PC dead, why replace it with something so locked down that you can't run a serious workflow in it? And why'd you then rebuild the PC, but with this exact same pattern? And even then, no one likes it, even forcing better options on it.

Ask your local gamer if they can install 8 high end video cards on a cell phone. Ask your local manufacturer if they can run their industrial supply line off of an iPad. Ask a movie junky if they can put their 60 terabyte RAID array in an e-Reader.

Some idiots realized all they ever used a computer for was looking at pictures of cats and random clips from "Family Guy", and replaced it with a mobile device. That does not, by any stretch of the imagination, mean that the PC is dead.

— Nathan Lineback

I used to think Android was the "freer" platform, only because it had Bluetooth on it (I hated how "exclusive" AirDrop is and how everything is opaque in iOS) and that it's "open-source". But what I have aren't those decidedly open-source versions. Despite rooting being piss-easy, I'm still scared that I could brick something. I'm using the same stuff pretty much everyone else uses.

To be fair, at least some vendors still let you see your device file system in Android instead of throwing everything into some weird cloud and pretend that's all you need to know. I've had a hard time helping someone move arbitrary files in iOS. I could segue into how Apple ruined everything for everyone and singing praises for any GNU/Linux phone (Purism, Pinephone, etc.) and all that… but I digress.

…In the end however, after all that, I just said fuck it and just used Syncthing instead. :p