geekAbyte

The use case is as follows: You have some tasks you want to execute concurrently, but you want the results to be returned in the same order the tasks were defined to be executed.

As an example, let's say you have some records returned from the database, as specified by some query. But before returning the records, some transformation needs to be done, and these transformations can be done concurrently. Due to the nature of concurrency, the order in which the transformations will be done is nondeterministic, meaning the order of the transformed records can and will end up being different from the original order of the records as retrieved from the database. This is something you do not want; you want the transformed records to be in the same order as specified in the original query that retrieved them from the database.

This sounds like the perfect task for Rust's FuturesOrdered type from the futures crate.

Reading the documentation, it states:

"...it imposes a FIFO order on top of the set of futures. While futures in the set will race to completion in parallel, results will only be returned in the order their originating futures were added to the queue."

Just what we want.

Let us illustrate with some code.

[dependencies]
futures = "0.3.30"
tokio = { version = "1.37.0", features = ["full"]}
rand = "0.8.5"

async fn transform(number: i32) -> i32 {
    let mut rng = rand::thread_rng();
    let random_number = rng.gen_range(1..=5);
    tokio::time::sleep(Duration::from_secs(random_number)).await;
    println!("Done from {number}");
    return number;
}

#[tokio::main]
async fn main() -> std::io::Result<()> {
    let mut tasks = FuturesOrdered::new();
    for n in 0..9 {
        tasks.push_back(async move { transform(n).await })
    }
    let mut results = vec![];
    while let Some(result) = tasks.next().await {
        results.push(result)
    }
    println!("{results:?}");
    Ok(())
}

The transform function represents the concurrent task. It sleeps for a random amount of seconds between 1 and 5, prints it is done, and then returns the transformed value, which in this case is just the original value passed to the function.

Running the code above should give something like:

Done from 1
Done from 3
Done from 6
Done from 9
Done from 5
Done from 7
Done from 0
Done from 2
Done from 4
Done from 8
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

The Done from n shows that the tasks are finished in different ordering, but the final printing of the results is still in order, showing that even though the tasks get executed concurrently and finishes out of order, when the result is retrieved via calling `tasks.next().await` it is returned in other.

#[tokio::main]
async fn main() -> std::io::Result<()> {
    let mut tasks = FuturesOrdered::new();
    for n in 0..=9 {
        tasks.push_back(transform(n))
    }
    let results = tasks.collect::<Vec<i32>>().await;
    println!("{results:?}");
    Ok(())
}

With this, the tasks would also execute, concurrently, finish in nondeterministic order, but the results will be collected in order as the tasks were started.

I took a break from exploring development in Solana, and the first thing I hit when I recently resumed again was failing tests when using the Anchor framework.

Running anchor run test leads to this error:

  1) calculator
  Is initialized!:
  Error: failed to get recent blockhash: FetchError:
  request to http://localhost:8899/ failed, reason:
  connect ECONNREFUSED ::1:8899
  at Connection.getLatestBlockhash
  (node_modules/@solana/web3.js/src/connection.ts:4555:13)
  at processTicksAndRejections
  (node:internal/process/task_queues:95:5)
  at AnchorProvider.sendAndConfirm
  (node_modules/@coral-xyz/anchor/src/provider.ts:147:9)
  at MethodsBuilder.rpc [as _rpcFn]
  (node_modules/@coral-xyz/anchor/src/program/namespace/rpc.ts:29:16)

The important part of the error message is:

Error: failed to get recent blockhash: FetchError:
request to http://localhost:8899/ failed, reason:
connect ECONNREFUSED ::1:8899
      at Connection.getLatestBlockhash

Googling the error, I found this suggestion here which says to "Try using node 16 instead of 17 if you're on 17".

I tried this and sure this works. But it does not feel right. node's current version is 20.x, dropping down to version 16 does not feel like the most appropriate way to solve this problem.

So why is the issue happening in the first place? Maybe if we understand that, we can come up with a better solution.

Well, the reason why dropping down to version 16 works, is because, starting from version 17, node now by default resolves "localhost" to IPv6's ::1, not IPv4's 127.0.0.1. Version 16 was the last version were node resolves localhost to 127.0.0.1.

And you guessed it, when you run the local Solana test validator, via solana-test-validator it is automatically listening on 127.0.0.1. So Anchor test is trying to connect to the validator using "localhost", but this resolves to an IPv6 address, but our Solana validator is listening for connections on an IPv4 address, which is 127.0.0.1 

Now that we know the problem, a better solution then is to configure the Solana test validator to listen on the IPv6 address, or better still configure it to listen on 0.0.0.0 which means it will be able to serve request that comes in both as IPv4 or IPv6.

To do this, stop your Solana test validator if it was running and then run the following command:

solana config set --url http://0.0.0.0:8899
// or to set to IPv6
solana config set --url http://\[::1\]:8899

Start your test validator and run the tests, everything should work fine.

Types, loosely defined, are about capabilities. What are the capabilities available on a value of a certain type? Programmers experienced with strong and statically typed languages will probably have a more visceral appreciation of this, but even programmers of weakly and dynamically typed languages like JavaScript, encounter the reality of types and their capabilities; only that they do so at runtime.

For example, a JavaScript developer might type out the following code:

> "3".toExponential()

But this will fail at runtime because the "3" is of the String type and it does not have the capability to be shown as exponential. The following, on the other hand, will work seamlessly:

> (3).toExponential()
'3e+0'

Because 3 is of the number type and the number type has the capability to be formatted in exponential form.

Rust extends the idea of types to now include capabilities that relate to memory management. These capabilities ensure memory safety in Rust.

Dealing with memory while programming is an area that is mostly hidden from developers who have mostly worked with languages with garbage collectors. This section quickly provides a brief overview of some of the key aspects of memory management, as moving to Rust will require a deeper understanding of what is happening behind the scenes.

Values within a program take up memory. There are various memory regions in a computer, where values can be placed, but two of the most common regions are the stack and heap.

You can think of the stack as a temporary memory location for values needed by an executing function. This includes function parameters and locally defined variables. Since this is more of a scratch area, access to it is fast, and once a function is done executing, the values can be discarded/overridden. Hence, the typical lifespan of a value on the stack is tied to the function that needs it for execution.

The heap, on the other hand, is a memory location that is more durable, where values that are not tied to the lifetime of an executing function can be placed. Dealing with the heap is more involved for obvious reasons. The heap is not as simple as the stack, and finding space to put new values might require additional low level memory management operation needed to make such space available. The stack does not have this complexity.

The idea is to limit the use of the heap region as much as possible, as it is not as fast as the stack memory region.

The type of a variable determines the type of values such a variable can contain. A consequence of this is that types also dictate how big or small the memory required by such a value is.

A variable of type u32 can contain numeric values in the range of 0 up to 4,294,967,295, while a variable of type u8, on the other hand, has a smaller size and can contain numeric values between the range of 0 and 255. This means a variable of u32, will require a memory that is 32-bit long, while u8, requires memory that is 8-bit in length.

Most types have a particular size, hence required memory length, that is knowable at compile time. u8 and u32 fall into this category of types. These types are called Sized Types. They are guaranteed to remain uniquely the same. A type of u32 will always need a 32-bit length of memory regardless if it contains 0 or 4,294,967,295. The same applies to all other sized types.

On the other hand, there are other types whose sizes cannot be uniquely known at compile time.

One example is an array represented by [T]. This type represents a certain number of T in sequence, but we don’t know how many there are; it could be 0, 1, 132, or 1 million T's. Hence, it is not possible to ascribe a unique size to these types at compile time. These types are known as Dynamically sized types (DST).

The str type is another example of DSTs. It represents a slice of strings. But for the same reason why we cannot uniquely determine the size of [T] at compile time, we also cannot determine the size of all slices of strings. Because a slice of string, represented by the type str could be 0, 1, 131, or 1 million long.

The idea with DSTs is not that the size is not known at compile times; they are, but they can vary, hence no unique size can be ascribed to them at compile time.

The Rust compiler generally prefers to know the size of types at compile time, for various reasons such as better management and optimization. So, we have a problem here with dynamically sized types, since creating a value and annotating it with a DST won't compile.

This:

fn main() {
   let dst_value: str = "hello world";
}

Will fail compilation with the following error

error[E0277]: the size for values of type `str` cannot be known at compilation time
  --> src/main.rs:70:9
   |
70 |     let dst_value: str = "hello world";
   |         ^^^^^^^^^ doesn't have a size known at compile-time
   |
   = help: the trait `Sized` is not implemented for `str`
   = note: all local variables must have a statically known size
   = help: unsized locals are gated as an unstable feature

The Rust compiler is kind enough to tell us why it is refusing to compile the code. A very instructive reason it gave is that "all local variables must have a statically known size."

So, how do we deal with this?

To fully understand how we first take a look at the concepts of ownership and the borrow checker in Rust.

Let's first look at copying and referencing. We can illustrate using Javascript.

Copying

This is one of the atomic operations performed in any programming language. You have value in one variable, you copy it into another. In Javascript:

> let a = 1
undefined
> let b = a
undefined
> b
1
> b = 2
2
> b
2
> a
1

Basically, once you have b = a the value of a is copied to b, hence when b is modified, it does not affect the value still contained in a.

Referencing

Referencing is when multiple variables do not have their unique copy of a value, but instead point to the same value. This means that if one of the variables changes the value, the other variable will also be changed.

> let a = [1,2,3]
undefined
> let b = a
undefined
> a.push(4)
4
> a
[ 1, 2, 3, 4 ]
> b
[ 1, 2, 3, 4 ]

To prevent creating a reference, a clone will have to be created.

The concepts of copying and referencing also exist in Rust, but Rust handles them differently. Before discussing copying in Rust, let's take a look at the concepts of ownership and moving of values, which on the other hand, are unique to Rust.

Ownership and moving of values

Rust introduces the concept of ownership and the moving of values. Simply put, it means values are not assigned to variables, rather values are owned by variables. For example:

>> let a = 1;

Should now be seen as the variable a was granted ownership of the value 1.

It is also possible to copy a value owned by a variable, which creates another value that is owned by the other variable.

>> let a = 1;
>> let b = a;
>> a
1
>> b
1

This should be interpreted as variable a owning the value 1, then with let b = a, a new copy of the value owned by the variable a is created and handed over to be owned by variable b.

The above is more or less the same behavior as the copying we saw with JavaScript.

The additional feature in Rust is the concept of moving values.

This means, instead of copying, a variable yields ownership of a value to another variable. As a result, once the value is moved and ownership is transferred, the old variable can no longer be used.

fn main() {
   let a = String::from("Hello world");
   let b = a; // value has moved from a to b
   println!("{b}");
   println!("{a}"); // this line won't make it compile
}

The above code won't compile. This is because with the values of type String, the let b = a line means the value from variable a is moved to variable b. That is, variable a has transferred ownership to variable b. Hence, trying to use the variable a later on in the code by attempting to print it out won't compile.

What could be confusing is the fact that in the previous example where we worked with values of type u8, the behavior was copying, which is similar to what we had with JavaScript. But, when the value is of type String, the value was moved!

What gives?

Well, it comes down to the idea of types again and their capabilities. The general idea is, types allocated on the heap generally default to moving values, while most primitive types that are placed on the stack copy instead. String is a type that manages memory on the heap hence why its default capability is not to copy. To create a distinct copy of String it will have to be cloned instead.

It is illustrative to point out that Rust further distinguishes between copying and cloning. Where copy can be interpreted as a simple bit-wise transfer of bits from one memory location to another while clone, apart from moving bits, also require additional logic to be executed.

This whole concept of moving values is something unique to Rust and new programmers coming to Rust need to be aware of this.

Borrowing and referencing of values

Now to recreate the scenario we had in Javascript where mutating an array in one variable led to the mutation on another variable, we have the following Rust code:

fn main() {
   let mut a = vec![1,2,3,4];
   let b = &mut a;
   b.push(5);
   println!("{a:?}");
}

Variable a holds a vector, but it was mutated via variable b and 5 was added, and this got reflected in variable a. Essentially, the same reference scenario we had with JavaScript arrays.

The only difference, however, is that Rust makes referencing explicit.

Let's go through the syntax. let mut a = vec![1,2,3,4]; grants ownership of the vector to variable a. Most data structures are immutable by default in Rust, so we use the mut keyword to indicate that the value owned by a can be modified.

Then let b =&mut a; is where the magic happens.

This, instead of moving the value from a to b, allows b to borrow the value instead.

This form of borrowing is achieved by creating a reference and handing that over to b.

The syntax &mut is used to achieve this, and the mut keyword indicates that it is possible to mutate the value through this reference.

It is possible to borrow a value, i.e. create a reference to a value but not be able to mutate the value. Sort of a read-only. In such a scenario, you only use & for example:

fn main() {
   let mut a = vec![1,2,3,4];
   let b = &a;
   println!("{b:?}");
}

But there are certain things you can and cannot do depending on if a value is owned, moved, borrowed, mutable, or immutable. These whole classes of restrictions are there to ensure memory safety, and this is what the borrow checker enforces.

In summary, the borrow checker ensures: you can have multiple read-only references to the same memory location as long as there are no mutable references. If a mutable reference is created to a memory location, it is only this mutable reference that can read from and write to the memory location.

To get a more elaborate exploration of the rules the borrow checker enforces, see Rust Ownership Rules

DSTs were already introduced as types that do not have a unique size at compile time, and as mentioned above, Rust prefers to deal with types it can know their size at compile time. So, how does Rust work with DSTs? To understand this, we need to talk about pointers.

Pointers and References in Rust

Pointers are variables, just like any other type of variable, but their values hold the memory address of other variables. They are an indirection that points to the memory location where another value can be found.

In Rust, pointers also have types, which makes sense since one can imagine, specific capabilities are only available to pointers. Pointers can also be found in languages like C and C++, but they can be difficult, and unsafe to work with due to the potential for misusing the unhindered access to memory locations.

This is the reason in Rust, you hardly deal with pointers directly. Instead, you have references, which are pointers with safety or liveness guarantees.

You can think of them as a protective layer that makes working with pointers safer. Pointers that are not references, i.e. that don't have this protective guarantee in Rust are usually called raw pointers.

The thing with references is that they also have types, and the good thing about them, is that they have a known size. This is because references hold memory addresses, hence it is possible to have a constant bit length that will always be assigned to references, big enough to enable it to store whatever memory address it needs to store.

Sometimes extra meta-information about the value in the memory address they point to is also stored, when this is the case, the reference is usually called a fat pointer.

The syntax for creating references is &T. To create a reference to a type T, use &T. For example, in the following code:

use std::collections::HashMap;
fn main() {
   let phone_code: HashMap<String, u8> = HashMap::from([("NL".to_string(), 31), ("USA".to_string(), 1)]);
   let ref_to_phone_code: &HashMap<String, u8> = &phone_code;
   dbg!(ref_to_phone_code);
}

phone_code is of the type HashMap<String, u8>, but ref_to_phone_code, a reference, is of the type &HashMap<String, u8> - notice the ampersand now in the type.

Also, the value of the type &HashMap<String, u8> was created by appending & to the variable it will reference, that is &phone_code in the code above.

Rust uses references to work with DSTs. Since references are types with known size at compile time, the trick is to only allow interaction with DSTs via references to them. This is why annotating a type with str, a DST, will cause a compilation failure, but &str, a reference to a DST, compiles fine.

As mentioned at the beginning of this post, types are about encoding capabilities allowed with a value. Rust extends this to encompass capabilities related to memory management and layout.

So a type T can exist in one of two flavors: One where its memory size is always known to be of a particular bit length; these are called Sized Types, and the ones whose size/bit length is not unique and can vary; these are called Dynamically Sized Types.

Then you have references, which are pointers (variables that hold memory location) with guarantees that ensure safe memory access.

These guarantees are enforced by the borrow checker.

These references could be of type &T or &mut T, depending on whether the reference is mutable or not. If type T is a DST, then a variable of type &T (or &mut T) can be used to reference the DST.

• The 5-Minute Guide to C Pointers 
• Pointer Types 
• Rust Ownership Rules 
• Visualizing memory layout of Rust's data types

This post will give a quick overview of declarative macros in Rust. If you are not familiar with the concepts of macros in Rust, then read Introduction to Macros in Rust first.

Declarative macros work similar to how pattern matching works. It involves taking inputs, then pattern matching to determine what logic to execute.

The macros definition for creating declarative macros, then contains rules which determine the Rust code that gets generated/executed based on the pattern.

This is why declarative macros are also sometimes referred to as Macros By Example, as the code that is generated depends on the example of patterns that matches.

Rust provides the macro_rules! macro for defining declarative macros. You use the macro_rules! to specify the input string pattern and also to specify the code that gets generated/executed.

The syntax for macro_rules takes the following form:

macro_rules! $name {
    ($matcher) => {$expansion} ;
    ($matcher) => {$expansion} ;
    // …
   ($matcher) => {$expansion} ;
}

That is, the macro_rules definition contains a list of rules, the particular rule, represented by {$expression} that will be executed depends on which of the ($matcher) matches the input. To see this syntax in action, create a new cargo project by running

cargo new declarative_macros

Open the main.rs and edit it.

The $matcher can be a simple literal match. For example to create a macro that prints different message based on whether the input passed to it is "morning" or "evening" , add the code below to main.rs:

fn main() {

    #[macro_export]
    macro_rules! greet {
        ("morning") => {
            println!("Good morning! Have a nice day!");
        };
        ("evening") => {
            println!("Good evening! Sleep tight!");
        }
    }
}
// using it
#[test]
fn run_greet() {
    // prints "Good morning! Have a nice day!"
    greet!("morning"); 
    // prints "Good evening! Sleep tight!"
    greet!("evening");
}

The $match can be more versatile than just matching a literal. For example the match can be used to define that the input should be an expression, a block of code etc. It can also be used to capture values passed in, that are then used within the macro logic.

To demonstrate how using a more versatile match pattern looks like when defining declarative macros, we write one that retrieves the balances for specified Solana accounts.

For this to work, update the dependencies section in Cargo.toml as follows:

[dependencies]
solana-client = "1.8.1, < 1.11"
solana-program = "1.8.1, < 1.11"

Then update the main.rs file as follows:

use solana_client::rpc_client::RpcClient;
use solana_program::pubkey::Pubkey;
use std::str::FromStr;

fn main() {
 #[macro_export]
 macro_rules! sol_balance {
  ($item:expr) => {
    {
     let mut result: Vec<u64> = Vec::new();
     let rpc = RpcClient::new("https://api.devnet.solana.com"
               .to_string());
     let item_str = format!("{}", $item);
     for i in item_str.split(",") {
         result.push(rpc.get_account(
         &Pubkey::from_str(i.trim()).unwrap()
         ).unwrap().lamports);
       }
       result
     }
  }}

}

#[test]
fn run_macros() {
    let result = sol_balance!(
        "4zMMC9srt5Ri5X14GAgXhaHii3GnPAEERYPJgZJDncDU,
        HEvSKofvBgfaexv23kMabbYqxasxU3mQ4ibBMEmJWHny"
    );
    dbg!(result);
}

Running the test would print out the sol balance at the specified account. Let's go over the code above. The core structure of the macro is:

    #[macro_export]
    macro_rules! sol_balance {
        ($item:expr) => {
        // body of macro
    }}

The ($item:expr) is an example of a more versatile match. It still corresponds to the ($matcher) in ($matcher) => {$expansion}. The $item is the input that is being matched, but this time around it is not just a literal. This syntax also captures the input making its value available for use in the macro code. This is called Metavariables. It also contains a qualification part using :expr. This is used to categorize the input. The :expr is an example of a fragment specifier. :expr specifies that the input is an expression. Other fragment specifiers exist. See Fragment Specifiers for more information.

The body of the macro is then defined as:

{
  let mut result: Vec<u64> = Vec::new();
  let rpc = RpcClient::new("https://api.devnet.solana.com"
            .to_string());
  let item_str = format!("{}", $item);
  for i in item_str.split(",") {
      result.push(rpc.get_account(
      &Pubkey::from_str(i.trim()).unwrap()
      ).unwrap().lamports);
     }
  result
}

Note that we wrap this within {} to allow using multiple statements.

It is also possible to support repetition when defining the matcher. This take the form of:

$ ( $(matcher) ) separator repmode

You can read this as $(matcher) as the pattern to be repeated, the separator that delineates the repetition, and repmode determines the number of repetition that is expected.

The currently supported repmode are:

?: at most one repetition *: zero or more repetitions +: one or more repetitions

The same syntax is used to process each item that is repeated. To see this repetition in practice, we can update the sol_balance macro to take a list of accounts separated by ";" instead of taking a single string separated by ",". This will look like:

use solana_client::rpc_client::RpcClient;
use solana_program::pubkey::Pubkey;
use std::str::FromStr;

fn main() {

 #[macro_export]
 macro_rules! sol_balance {
   ($($item:expr);*) => {
     {
       let mut result: Vec<u64> = Vec::new();

       let rpc = RpcClient::new("https://api.devnet.solana.com"
                 .to_string());
         $(
           let item_str = format!("{}", $item);
           result.push(rpc.get_account(
           &Pubkey::from_str(item_str.trim()).unwrap()
           ).unwrap().lamports);
          )*
          result
       }
    }}
}

// usage

#[test]
fn run_macros() {
  let result = sol_balance!(
    "4zMMC9srt5Ri5X14GAgXhaHii3GnPAEERYPJgZJDncDU"
     ;
     "HEvSKofvBgfaexv23kMabbYqxasxU3mQ4ibBMEmJWHny"
   );
  dbg!(result);
}

Running the test with the code modified as above should also print the sol balance at the given accounts.

There is more to writing a declarative macros that is not touched upon in this overview. For example we did not talk about the #[macro_export] attribute, neither did we talk about things like Metavariable Expressions. Even at this, this introduction should be enough to understand the essence of declarative Macros in Rust.

The next post in this series will then be an introduction to the function-like procedural macros.

This guide will present an overview of the fundamentals a new developer approaching macros in Rust needs to know. If you have read other materials online and you still find the concept of macros in Rust vague, then this post is for you.

The guide will be broken into 4 series of posts. The first post, which is this post, will be a quick overview of macros in Rust. The following 3 posts will then look more deeply into the different types of macros in Rust and how to create them.

The posts in this series are:

• Introduction to macros in Rust (this post)
• Introduction to declarative macros in Rust
• Introduction to attribute-like procedural macros in Rust (yet to be published)
• Introduction to function-like procedural macros in Rust (yet to be published)
• Introduction to custom-derive procedural macros in Rust (yet to be published)

Macros are mechanisms for generating or manipulating code programmatically at compile time. In essence, it allows us to treat source code as values. Meaning that, at compile time, we can take inputs: where the input could be another source code or some defined string pattern, and from these inputs, manipulate and generate source codes that form the final compiled code that will be executed.

Macros as a concept are not specific to Rust, as they can be found in languages like Lisp, C/C++, Scala, and a host of others. Although one clear distinction of macros in Rust is that it provides type safety. Meaning that it provides mechanisms to prevent generating programs or source code that are not type-safe.

That is what macros are in a nutshell, a mechanism for taking inputs (other source codes, or just string patterns) and using these to generate source code/program at compile time.

So now that we know what macros are, we talk about how macro usage looks like.

Before looking at how to create macros in Rust, it is instructive to see how already created macros are used.

There are two distinct ways of making use of macros in Rust. macros are either used via A syntax I call bang function syntax or via attribute syntax.

The bang function syntax is the syntax that looks like a function call but with an exclamation mark at the end of the function name. For example things like: println!(), dbg!(), format!().

These are used to invoke macros and are not normal function calls.

Attributes syntax, on the other hand, is the syntax used for annotation. It takes the form of #[..] or #![..] and they can be placed on various language elements like functions, structs, enums, etc. This is another syntax that can be used to invoke macros.

A real-world example of attribute syntax being used to invoke macros can be found in the web framework called rocket. For example in the code below, gotten from the documentation here where we see the definition of a handler for the / HTTP path:

#[get("/")]
fn index() -> &'static str {
    "Hello, world!"
}

The #[get("/")] syntax is invoking a macro defined by the rocket framework. This invocation means that at compile time, the custom get macro will take the function index() as input, process it, and generate the exact source code, the framework needs to be able to call this function in response to an HTTP GET request to the path /. Essentially fulfilling what a macro is: a mechanism to generate source code at compile time.

The exact syntax used to call a macro depends on the type of macro and how it has been created. Some macro types are called via the bang function syntax, while other kinds are called via the attribute syntax.

In Rust, there are two types of macros: Declarative macros and Procedural macros. There are then 3 subcategories of procedural macros: Custom-derive macros, Attribute-like macros, and Function-like macros.

Basically types of Rust macros:

• Declarative macros (also referred to as macro by example or mbe for short)
• Procedural macros
  • Function-like
  • Attribute-like macros
  • Custom derive macros

The first thing to note in this categorisation is what makes declarative macros different from procedural Macros.

The difference lies in their inputs and outputs, and also how they are created and used.

Declarative macros take what can be described as token fragments while procedural macros take TokenStream as inputs. Token fragments are like string patterns that can further be qualified by a fragment specifier that tells what kind they are. Fragment specifiers can be used to specify the inputs are block, expression, statement, etc. See Fragment Specifiers for an overview of the supported fragment specifiers.

Procedural macros, on the other hand, take in TokenStream which is a more low-level representation of the source code.

The other difference between declarative macros and procedural macros (also the difference between the 3 kinds of procedural macros) is how they are created and used.

Declarative macros and function-like procedural macros are used by a syntax that looks like function invocation but ends with !. for example do_work!(). Meaning when looking at usage it is not possible to tell if a declarative macro is being used or a function-like procedural macro is.

Attribute-like procedural macros on the other hand are used to create a custom annotation, so they are used with the syntax of applying attributes. That is: #[my_macro]. They can also be defined to take helper attributes. That is #[my_macro(attr)]

While custom derives macros are used to define new values that can be passed into the #[derive(...)] macro. For example, if MyMacro is defined to be a custom derive macro, it can then be used as #[derive(MyMacro)]

This completes the quick overview of macros in Rust. The following posts in the series will then go into the details of how these different types of macros are created and used and their different peculiarities.

This post shows how to query Solana's cluster rpc endpoint using Rust. Most of the examples of how to perform this task that I ran to uses Javascript/Typescript via the Solana Web3js library, so I decided to do a quick post showing how to do same but in Rust. It uses querying for the sol balance of a Solana account as the main example of interaction with the rpc endpoint.

The task is simple: given a Solana address, we retrieve the sol balance for that address.

To get started we will need the following dependencies:

[dependencies]
solana-client = "1.14.3" // update to the current latest version
solana-sdk = "1.14.3" // update to the current latest version

The code snippet below then shows how to call the rpc endpoint and retrieve the Sol balance for an account

use solana_client::rpc_client::RpcClient;
use solana_program::pubkey::Pubkey;
use std::str::FromStr;

fn main() {
 let rpc = RpcClient::new("https://api.devnet.solana.com");
 let pubkey_str = "4zMMC9srt5Ri5X14GAgXhaHii3GnPAEERYPJgZJDncDU";

 let balance = rpc
      .get_account(&Pubkey::from_str(pubkey_str).unwrap())
      .unwrap().lamports;

 println!("Sol balance of {pubkey_str} is {balance}");
}

Solana addresses are base58 encoded strings, hence they need to be first converted into an instance of Pubkey before they can be used in the rpc.get_account call, which expects a &[u8] byte array. To help with this conversion, Pubkey implements the std::str::FromStr trait. This wmeans, if the trait std::str::FromStr is in scope, then it is possible to convert a string to Pubkey, which is what was done in the code snippet above.

This post was inspired by a recent question I answered on Solana stack exchange here. The answer also mentions another verbose alternative that involves manually converting from base58 to bytes array.

In Rust Iterator pattern with iter(), into_iter() and iter_mut() methods I explained why attempting to use a variable holding a Vec after iterating through it using the for … in syntax leads to a compilation error.

The post explains why the following code won't compile:

fn main() {
   let some_ints = vec![1,2,3,4,5];
  // iterating through a vec
   for i in some_ints {
       dbg!(i);
   }
// attempting to use the vec will 
// lead to compile error after the iteration
   dbg!(some_ints);
}

I then showed 3 methods that can be called before iterating using the for … in and how 2 of these methods allow the Vec to still be used even after iteration.

These 3 methods are into_iter(), iter(), and iter_mut(). That is:

#[test]
fn into_iter_demo() {
    // the .into_iter() method creates an iterator, v1_iter 
    // which takes ownership of the values being iterated.
    let mut v1_iter = v1.into_iter();

    assert_eq!(v1_iter.next(), Some(1));
    assert_eq!(v1_iter.next(), Some(2));
    assert_eq!(v1_iter.next(), Some(3));
    assert_eq!(v1_iter.next(), None);

    // If the line below is uncommented, the code won't compile anymore
    // this is because, after the iteration, v1 can no longer be used 
    // since the iteration moved ownership
    //dbg!(v1);
}

The two other methods that allow the Vec to still be used after iteration via for … in are:

#[test]
fn iter_demo() {
    let v1 = vec![1, 2, 3];
    // the .iter() method creates an iterator, 
    // v1_iter which borrows value immutably 
    let mut v1_iter = v1.iter();

    // iter() returns an iterator of slices.
    assert_eq!(v1_iter.next(), Some(&1));
    assert_eq!(v1_iter.next(), Some(&2));
    assert_eq!(v1_iter.next(), Some(&3));
    assert_eq!(v1_iter.next(), None);
   // because values were borrowed immutably, 
   // it is still possible to use 
   // the vec after iteration is done
    dbg!(v1);
}

And

#[test]
fn iter_mut_demo() {
    let mut v1 = vec![1, 2, 3];

    // the .iter_mut() method creates an iterator, 
    // v1_iter which borrows value and can mutate it. 
    let mut v1_iter = v1.iter_mut();

    // access the first item and multiple it by 2
    let item1 = v1_iter.next().unwrap();
    *item1 = *item1 * 2;

    // access the second item and multiple it by 2
    let item2 = v1_iter.next().unwrap();
    *item2 = *item2 * 2;

    // access the third item and multiple it by 2
    let item3 = v1_iter.next().unwrap();
    *item3 = *item3 * 2;

    // end of the iteration
    assert_eq!(v1_iter.next(), None);

    // this will print out [2,4,6]
    dbg!(v1);
}

In this post, we are going to dive a little bit deeper into understanding some of the machinery that makes the above work.

We start again by talking about the Iterator trait.

An Iterator represents the ability to retrieve elements from another data structure in sequence. In rust, it is any data structure that implements the Iterator trait.

It is important to note that the Vec data structure by itself is not an Iterator and hence cannot be iterated.

To make this more obvious, let us forget about the for … in syntax for a second, and try to perform an operation that should be possible on a data structure that supports being iterated.

An example of such an operation is the for_each method.

In the code example below, we attempt to loop directly over a Vec of numbers using for_each and print each item. The code won't compile:

fn main() {
   let some_ints = vec![1,2,3];
   // calling for_each directly on a Vec won't compile
   some_ints.for_each(|item| {
       dbg!(item);
   });
}

The compile error below gives us a clue as to why the code does not compile:

error[E0599]: `Vec<{integer}>` is not an iterator
  --> src/main.rs:60:14
   |
60 |      some_ints.for_each(|item| {
   |                ^^^^^^^^ `Vec<{integer}>` is not an iterator; try calling `.into_iter()` or `.iter()`
   |
   = note: the following trait bounds were not satisfied:
           `Vec<{integer}>: Iterator`
           which is required by `&mut Vec<{integer}>: Iterator`
           `[{integer}]: Iterator`
           which is required by `&mut [{integer}]: Iterator`

For more information about this error, try `rustc --explain E0599`.
error: could not compile `playground` due to a previous error

The compile error contains the line:

Vec<{integer}> is not an iterator; try calling .into_iter() or .iter()

Proving the point that a data structure like Vec by itself is not an iterator.

But instead of using the for_each method, we can actually perform an iteration directly on the Vec using the for … in syntax.

For example, the following code compile and runs as expected:

fn main() {
   let some_ints = vec![1,2,3];
   for item in some_ints {
       dbg!(item);
   }
}

What gives?!

Did we not just prove that a Vec is not an Iterator on itself?

We even showed this by trying to call a method that should work on an iterator and confirm it fails. But here we are still being able to iterate over something that should not be an Iterator using the for … in syntax.

How is that possible?

To understand why this works, we need to look into another trait called IntoIterator.

The IntoIterator is a trait that specifies how a data structure can be converted into an Iterator. The basic structure of the trait looks like this:

pub trait IntoIterator {
    type Item;
    type IntoIter: Iterator
    where
        <Self::IntoIter as Iterator>::Item == Self::Item;

    fn into_iter(self) -> Self::IntoIter;
}

As can be seen, the main method specified by the trait is into_iter(). The result of calling it with is an Iterator. That is, Self::IntoIter the return type, is of type Iterator, given it is an associated type defined to be an iterator in the body of the trait. This is what the line type IntoIter: Iterator above means.

So any data structure, that by itself is not an iterator, can define how it can be transformed into an iterator by implementing the IntoIterator.

The Vec data structure defined in the Rust standard library implements the IntoIterator, which means it has the method into_iter() which when called returns an Iterator.

To see this in action, let's go back to the code that did not compile, where we directly called for_each on a Vec, but instead of calling for_each directly, we first call into_iter(), before calling for_each.

fn main() {
   let some_ints = vec![1,2,3];
   // first calling into_iter() works
   some_ints.into_iter().for_each(|item| {
       dbg!(item);
   });
}

This works, because the first call to into_iter() returns an Iterator, which allows iterating over the underlying Vec.

So how does this help answer why it is possible to use for … in directly on a Vec without first turning it into an Iterator by calling into_iter?

Well, the answer is that when the for … in syntax is used, the compiler automagically first calls into_iter(), getting an Iterator back and using that to do the iteration.

According to the documentation, the for … in syntax actually desugars to something like this:

let values = vec![1, 2, 3, 4, 5];
{
    let result = match IntoIterator::into_iter(values) {
        mut iter => loop {
            let next;
            match iter.next() {
                Some(val) => next = val,
                None => break,
            };
            let x = next;
            let () = { println!("{x}"); };
        },
    };
    result
}

Where the into_iter is first called on the Vec value, and then the iteration is continuously done, calling next(), until None is reached, signifying the end of the iteration.

So the into_iter method, which is part of the IntoInterator traits explains how the for … in syntax can be used for iterating over a Vec. And this is because the compiler by default calls the into_iter when for … in syntax is used.

But what about the other two similar methods that we saw at the beginning of this post? That is iter() and iter_mut().

It is also possible to use these two methods to turn a Vec into an iterator.

That is:

fn main() {

   let some_ints = vec![1,2,3];
   // first calling into_iter() works
   some_ints.iter().for_each(|item| {
       dbg!(item);
   });
}

and

fn main() {
   let mut some_ints = vec![1,2,3];
   some_ints.iter_mut().for_each(|item| {
       dbg!(item);
   });
}

What is going on when iter_mut() and iter() are used? And how is this different from the into_iter() that comes from the IntoIterator trait?

Iterators can come in different forms. Nothing is stopping a developer from implementing an Iterator that has other custom behavior that defines how it iterates.

In Rust's standard library, most collections have 3 different kinds of Iterators. We can have one which takes ownership of the value being iterated, one that borrows the value immutably, and another that borrows the value and can mutate it.

An iterator that takes ownership can be created by calling into_iter(), one that borrows immutable can be created by calling iter() and the one that borrows value with the ability to mutate can be created by calling iter_mut(). This is the crux of the Rust Iterator pattern with iter(), into_iter() and iter_mut() methods post.

It turns out that the Rust compiler by default goes for the into_iter() version when it de-sugars the for … in syntax.

One important thing to point out here is the fact that it is possible to have a custom data structure, that is an iterator, i.e. has all the familiar iteration-related methods: map, for_each etc but which is not usable with the for … in syntax. This will be the case if such data structure implements Iterator but does not implement the IntoIterator trait. Because without implementing the IntoIterator trait, there will be no into_iter() method for the for … in syntax to call.

Another interesting point is what happens if we manually call any of into_iter(), iter(), or iter_mut ourselves as part of usage in for … in syntax. Basically what was shown in the Rust Iterator pattern with iter(), into_iter() and iter_mut() methods post.

How come these works:

fn main() {
   let mut some_ints = vec![1,2,3];
// manually calling iter in a for … in 
   for i in some_ints.iter() { 
       dbg!(i);
   }

// manually calling iter_mut in a for … in 
   for i in some_ints.iter_mut() {
       dbg!(i);
   }

// manually calling into_iter in a for … in    
   for i in some_ints.into_iter() {
       dbg!(i);
   }
}

We are manually converting the Vec ourselves to an iterator by calling iter(), iter_mut(), and into_iter() and yet it works.

Why does this work?

Was it not already stated that the for … in syntax works with anything that implements IntoIterator which allows it to call into_iter(). And here we are, manually converting the Vec into an iterator ourselves, and yet the for … in works. How come?

The answer is in a little trick in the standard library. Which is the fact that the standard library contains this implementation for IntoIterator:

impl<I: Iterator> IntoIterator for I

This basically means any Iterator implements IntoIterator and the implementation is such that the Iterator returns itself when into_iter() is called. Which makes sense if you think about it. If something is already an Iterator, what else can be done when you attempt to turn it again into an Iterator other than it returning itself?

And this is what happens. Even though the iter(), into_iter() and iter_mut methods are called directly, for … in, still work, because the iterator created by calling this method automatically has an implementation of IntoIterator which returns itself and which the for … in syntax needs.

The above shows how implementing the IntoIterator trait can be done in such a way as to provide interesting functionalities.

Another interesting utility that is achieved via providing different implementations for IntoIterator is how it is possible to use for … in with a collection and yet be able to still use the collection after iteration, without having to call the iter() or iter_mut.

This is a more succinct syntax to the solution provided in the Rust Iterator pattern with iter(), into_iter() and iter_mut() methods post

We look at this, in the next section.

There are three different implementations of IntoIterator for the Vec type. These 3 different implementations are for 3 different variants of the Vec type depending on its memory access.

There are implementations of IntoIterator for the bare Vec<T> type, the immutable referenced &Vec<T> type and mutable reference &'a mut Vec<T> type.

The implementation of IntoIterator for bare Vec<T> returns an Iterator that takes ownership of the values as they are iterated. The implementation of IntoIterator for &Vec<T> borrows the value being iterated immutable, while the implementation of IntoIterator for &'a mut Vec<T> makes it possible to mutate the value as part of the iteration.

This means one can iterate over a Vec type and still be able to use it afterward if the iteration is done over &Vec<T> or &'a mut Vec<T>. For example:

fn main() {
   let some_ints = vec![1,2,3];
   for i in &some_ints { // same as calling some_ints.iter()
       dbg!(i);
   }
}

and

fn main() {
   let mut some_ints = vec![1,2,3];
   for i in &mut some_ints { // same as calling some_ints.iter_mut()
       *i = *i * 2;
   }
   dbg!(some_ints);
}

The above syntax can be used as a more succinct way to iterate over a data structure like Vec using the for … in syntax without taking ownership of the Vec.

• The IntoIterator is a trait that defines how an Iterator can be created for a data structure. It defines an into_iter() method that when it is called, should return an Iterator 
• The for .. in syntax requires there is an implementation of IntoIterator because the compiler automagically first calls into_iter() to retrieve an Iterator it uses for its iteration. 
• The Rust standard library also contains an implementation of IntoIterator for an Iterator. This implementation just returns the Iterator. This makes sense, because if something is already an iterator, then returning it, satisfies the contract defined by IntoIterator 
• The fact that there is an IntoIterator for Iterator that returns that iterator means that methods like iter() or iter_mut() can still be used within the for .. in syntax. The for .. in syntax, can call the into_iter() gets the Iterator itself and uses that for its iteration. 
• The standard library also contains 3 different implementations of IntoIterator for Vec<T>, &Vec<T>, and &'a mut Vec<T>. The implementation for Vec<T> takes ownership of the value being iterated, the implementation for &Vec<T> borrows values being iterated immutably, and the implementation for &'a mut Vec<T> borrows values mutably. 
• Given a variable some_var holding a Vec one can iterate over it using for … in and still be able to use it after the iteration if the iteration is done over &some_var and &mut some_ints.

Let's create a vec of integers, iterate through and print the individual values, and then afterward, print out the whole vec. Here is the code:

fn main() {
   let some_ints = vec![1,2,3,4,5];
   for i in some_ints {
       dbg!(i);
   }

   dbg!(some_ints);
}

Simple right? Well nope. Trying to compile the above code will fail with the following errors:

error[E0382]: use of moved value: `some_ints`
   --> src/main.rs:9:9
    |
4   |    let some_ints = vec![1,2,3,4,5];
    |        --------- move occurs because `some_ints` has type `Vec<i32>`, 
    which does not implement the `Copy` trait
5   |    for i in some_ints {
    |             --------- `some_ints` moved due to this implicit call to `.into_iter()`
...
9   |    dbg!(some_ints);
    |         ^^^^^^^^^ value used here after move
    |
note: this function takes ownership of the receiver `self`, which moves `some_ints`
help: consider iterating over a slice of the `Vec<i32>`'s 
content to avoid moving into the `for` loop
    |
5   |    for i in &some_ints {
    |             +

For more information about this error, try `rustc --explain E0382`

Why is this the case? Why is the borrow checker preventing the use of a vec after a simple iteration?

Well, the answer to that question lies in Rust's implementation of the Iterator pattern - which by the way, is what makes it possible to use the for…in syntax.

Iterators are not special or unique to Rust. The concept can be found in a handful of languages. I wrote about the Iterator pattern as it exists in JavaScript in the post Iterables and Iterators in JavaScript.

The unique thing about the Iterator pattern in Rust is its interaction with the borrow checker.

If this interaction with the borrow checker is not taken into consideration then it is possible to bump into certain confusing compile errors while attempting to use the iterator pattern.

So to get started answering the question of why the borrow checker prevents what looks like a legit code above, let's take a look at the Iterator pattern and what is special about it in Rust.

An Iterator is a data structure that allows retrieval of elements from another data structure in sequence. In rust, it is any data structure that implements the Iterator trait.

#![allow(unused)]
fn main() {
pub trait Iterator {
    type Item;

    fn next(&mut self) -> Option<Self::Item>;

    // methods with default implementations elided
 }
}

The Iterator trait defines a next method that when called, returns an Option of type Self::Item, which ends up being whatever the type of elements in what is being iterated over. Once the iteration reaches the end, a None is returned. For example:

#[test]
fn iterator_demo() {
 let v1 = vec![1, 2, 3];
 let mut v1_iter = v1.iter();

 assert_eq!(v1_iter.next(), Some(&1));
 assert_eq!(v1_iter.next(), Some(&2));
 assert_eq!(v1_iter.next(), Some(&3));
 assert_eq!(v1_iter.next(), None);

}

The above is simple enough. The part that interacts with the borrow checker is how the next method treats the value it is getting from the underlying structure it is iterating over.

It is possible that the next method of the iterator trait be implemented in such a way that it borrows the value immutable, borrows the value mutable, or even takes ownership of the values being iterated.

These three behaviors define 3 different distinct types of iterators.

• An iterator that borrows the value being iterated immutably
• An iterator that borrows the value being iterated mutably.
• An iterator that takes ownership of values being iterated

To demonstrate these 3 iterator types, we can take a vec as usual and create these three distinct iterators and observe their behavior.

#[test]
fn iter_demo() {
    let v1 = vec![1, 2, 3];
    // the .iter() method creates an iterator, v1_iter 
    // which borrows value immutably 
    let mut v1_iter = v1.iter();

    // iter() returns an iterator of slices.
    assert_eq!(v1_iter.next(), Some(&1));
    assert_eq!(v1_iter.next(), Some(&2));
    assert_eq!(v1_iter.next(), Some(&3));
    assert_eq!(v1_iter.next(), None);
   // because values were borrowed immutably, it is still 
   // possible to use the vec after iteration is done
    dbg!(v1);
}

In the code above, the .iter() method that is called on the vec, v1 creates an iterator, v1_iter which borrows value immutably. Because the values of v1 were borrowed immutable by the iterator v1_iter it was still possible to use the vec v1 after the iteration by dbg! printing it.

#[test]
fn iter_mut_demo() {
    let mut v1 = vec![1, 2, 3];

    // the .iter_mut() method creates an iterator, 
    // v1_iter which borrows value and can mutate it. 
    let mut v1_iter = v1.iter_mut();

    // access the first item and multiple it by 2
    let item1 = v1_iter.next().unwrap();
    *item1 = *item1 * 2;

    // access the second item and multiple it by 2
    let item2 = v1_iter.next().unwrap();
    *item2 = *item2 * 2;

    // access the third item and multiple it by 2
    let item3 = v1_iter.next().unwrap();
    *item3 = *item3 * 2;

    // end of the iteration
    assert_eq!(v1_iter.next(), None);

    // this will print out [2,4,6]
    dbg!(v1);
}

In the code above, the .iter_mut() method that is called on the vec, v1 creates an iterator, v1_iter which borrows value mutable. That is, the value borrowed can be updated. And the code does exactly that, mutating each value iterated by multiplying it by 2. Because the values of v1 were borrowed mutably and not owned by the iterator v1_iter it was still possible to use the vec v1 by printing it. Printing also confirms that the values in the vec have been updated with each item multiplied by 2.

#[test]
fn into_iter_demo() {
    // the .into_iter() method creates an iterator, 
    // v1_iter which takes ownership 
    // of the values being iterated.
    let mut v1_iter = v1.into_iter();

    assert_eq!(v1_iter.next(), Some(1));
    assert_eq!(v1_iter.next(), Some(2));
    assert_eq!(v1_iter.next(), Some(3));
    assert_eq!(v1_iter.next(), None);

    // If the line below is uncommented, the code won't compile anymore
    // this is because, after the iteration, v1 can no longer be used 
    // since the iteration moved ownership
    //dbg!(v1);
}

In the code above, the .into_iter() method creates an iterator, v1_iter which takes ownership, because of this, v1 is no longer usable after the iteration has ownership has moved after the iteration. Uncommenting the dbg! line would lead to a compile error because of this.

So now we have seen that Iterators in Rust can come in 3 forms depending on how ownership is handled when the iterator accesses the values of the data structure being iterated.

How does this then help us answer why the following code:

fn main() {
   let some_ints = vec![1,2,3,4,5];
   for i in some_ints {
       dbg!(i);
   }

   dbg!(some_ints);
}

which looks legit get flagged by the borrow checker?

The answer lies in the fact that the for…in syntax, by default, calls the .into_iter() method, which returns an iterator that takes ownership of values being iterated.

The code above is essentially the same as the following code:

fn main() {
   let some_ints = vec![1,2,3,4,5];
   for i in some_ints.into_iter() {
       dbg!(i);
   }

   dbg!(some_ints);
}

Compiling both will lead to the same error.

So to use the for…in syntax and still be able to use the underlying vec, after iteration, we will have to manually call either the iter() or iter_mut() method instead.

For example, using the iter() method with for..in will look like this:

fn main() {

   let some_ints = vec![1,2,3,4,5];
   for i in some_ints.iter() {
       dbg!(i);
   }

   dbg!(some_ints);

This time around, the code compiles fine. And if the code is executed, the contents of some_ints will get printed fine even after the iteration. Something that was not possible before.

Using the iter_mut() method on the other hand enables us to mutate the values as they are being iterated. For example:

fn main() {

   let mut some_ints = vec![1,2,3,4,5];
   for i in some_ints.iter_mut() {
       *i = *i * 2;
   }

   dbg!(some_ints);

}

The above code compiles fine, and when run, will print [2,3,6,8,10] confirming that it was possible to mutate the values while iterating.

To further explore how all the above works, check IntoIterator and the for ... in Syntax in Rust

• An Iterator is a pattern that allows retrieval of elements from another data structure in sequence. In Rust, it is represented by the Iterator trait.
• A data structure implementing the Iterator trait interacts with the borrow checker based on how it handles access to the values in the data structure being iterated.
• It is possible for the iterator to borrow the value being iterated immutably. This type of iterator is created by calling iter().
• It is possible for the iterator to borrow the value being iterated mutably. This type of iterator is created by calling iter_mut().
• It is possible for the iterator to take ownership of the value values being iterated. This type of iterator is created by calling into_iter()
• By default, the for..in syntax calls the into_iter() which takes ownership of the value being iterated. This is why after iteration, the underlying variable that holds the original vec becomes unusable. This is because ownership has moved away from it.
• To use the for..in syntax and still be able to use the variable that holds the original vec, after iteration, then either call iter() or iter_mut()

In this post, we would be looking at digital signatures. It will be a quick, straight-to-the-point overview of some of the essential things a developer should know about digital signatures. How to think about them and what problems they solve. It is targeted at the working developer who needs to be familiar enough with digital signatures to use them, but who does not need to know the gory details of how they are implemented, or how they work internally.

This post contains the following sections: 

1. What is a digital signature 
2. Components of Digital Signature Schemes 
3. Digital Signature Algorithms 
4. Last note: Digital Signatures versus Message Authenticated Codes.

Signatures in real life are used to ascertain that a piece of document originated and is authorized by the signer.

Similarly, in the digital world, digital signatures are cryptographic primitives used for the same purpose. That is, to ascertain that some digital data came from, and has been authorized by an entity that claims to have originated it.

The digital data could be in the form of an email, a PDF document, blockchain transactions, etc. the point of digital signature remains the same: ascertain that some digital data is legit by ascertaining it originated from the right origin. This function is called origin authentication.

Apart from origin authentication, digital signatures can also be used to ascertain that the data signed has not been tampered with. This is called message authentication or Integrity.

It is also worth mentioning that digital signatures are cryptographic primitives that fall under the asymmetric cryptography category because they make use of two key pairs: private and public keys.

Let's now take a look at the components of a digital signature scheme.

Digital signature schemes are usually made up of three distinct components or algorithms:

• Key generation: This component of the scheme consists of the algorithm that takes care of generating the key pair (private/public) that the signer needs to generate the digital signature.
• Signing Algorithm: This component is the actual algorithm that generates the digital signature. It takes the message to be signed and uses the private key generated by the key generation algorithm, to create a digital signature.
• Verifying Algorithm: This checks the authenticity of digital signatures. It takes the public key generated via a key generation algorithm, together with the signature, generated by the signing algorithm, and returns either success if the signature is valid or an error if invalid.

Having listed the three main broad components of digital signatures, let's now take a look at some of the popularly known digital signature schemes. For completeness' sake, it would also mention schemes that were once used but are no longer recommended due to security reasons.

RSA (Ron Rivest, Adi Shamir, and Leonard Adleman - last names of the cryptographers who proposed the scheme) was the first digital signatures scheme to be articulated in 1977.

RSA was introduced as a scheme for encryption in Introduction to Asymmetric Encryption for the Working Developer. It is asymmetric cryptography primitive, hence uses private/public key pair. When used for encryption, the public key is used to encrypt a message that is then sent to the owner of the corresponding private key. The private key can then be used to decrypt the message.

When RSA is used for digital signature, the signing algorithm involves taking a hash of the message to be signed, and then encrypting the hash to create a ciphertext that serves as the signature. The message and the signature (ciphertext of the message) are then shared.

The verifying part of the scheme involves two steps. The first step is to use the same hashing algorithm to hash the message received from the signer. The second step is to use the corresponding public key to the private key of the signer to decrypt the received signature (that is the ciphertext the signer published with the message). If the hash value generated by the verifier is the same as the one revealed after using the public key to decrypt the signature, then the signature is valid. If it is not equal, then the signature is invalid.

As mentioned in Introduction to Asymmetric Encryption for the Working Developer RSA is usually used with a padding scheme to make it more secure. The first padding scheme that can be used with RSA to create digital signatures is PKCS1-v1_5. This gives rise to the RSA-PKCS1-v1_5 digital signature scheme. This scheme is no longer recommended due to security vulnerability.

A newer standard RSA-PSS which makes use of PSS encoding algorithm is the preferred scheme to use when using RSA for digital signatures.

Schnorr Signature is another digital signature scheme that was proposed in 1990 by Claus P. Schnorr. It is efficient and secure, with no known vulnerability. Although historically, It is not widely used because it was covered by a patent. The patent has since expired in 2008 so adoption will probably see an uptick. This is observed already has Bitcoin recently added Schnorr signatures as part of the digital signature it supports.

In 1991, the National Institute of Standard and Technology NIST proposed the Digital Signature Algorithm (DSA). For all intent and purposes, DSA is a variant of the Schnoor signature. The sole reason why it was proposed, was to avoid Shnoor's patent restriction. It has no known vulnerability and enjoyed great adoption, although it has since been replaced with the elliptic curve implementation of the scheme, which is known as the Elliptic Curve Digital Signature Algorithm (ECDSA)

EdDSA, which stands for Edwards-curve Digital Signature Algorithm, was proposed in 2008 by Daniel J. Bernstein. It takes inspiration from Schnoor's signature it is also an elliptic curve-based scheme since it makes use of the Edward curve

It is worth mentioning that these schemes mentioned above are by far not the only digital signature schemes in existence. Although, arguably, they are the most common. There exist other schemes like ElGamal signature scheme, the Rabin signature algorithm, or NTRUsign just to mention a few.

The code snippets below show how to use RSA-PSS using Deno's implementation of Web Crypto API. Three methods will be used: generateKey, sign, and verify. Which corresponds to the three components of digital signature schemes.

// Key Generation
const keyParams = {
    name: "RSA-PSS",
    // The length in bits of the RSA modulus
    modulusLength: 2048,
    // The public component e. Recommended value is 65537
    publicExponent: new Uint8Array([1, 0, 1]),
    // hash function required by PSS
    hash: "SHA-256",
};

const keyPair = await crypto.subtle.generateKey(keyParams, true, ["sign", "verify"])

// Signing

const message = new TextEncoder().encode("increase interest rate by 2%")

const signParams = {
    name: "RSA-PSS",
    // A long integer representing the length of the random salt to use, in bytes
    // Value can be either 0 or the length of the output of the key's digest algorithm. 
    // For example, if you use SHA-256 as the digest algorithm, this could be 32
    saltLength: 32
}
const signature = await crypto.subtle.sign(signParams, keyPair.privateKey, message)


// Verification

const verifyParams = {
    name: "RSA-PSS",
    // A long integer representing the length of the random salt to use, in bytes
    // Value can be either 0 or the length of the output of the key's digest algorithm. 
    // For example, if you use SHA-256 as the digest algorithm, this could be 32
    saltLength: 32
}

const verificationResult = await crypto.subtle.verify(verifyParams, keyPair.publicKey, signature, message)

console.log(`Signature verification is: ${verificationResult}`);


// Confirm if message is different verification fails

const differentMessage = new TextEncoder().encode("increase interest rate by 20%")

const failedVerification = await crypto.subtle.verify(verifyParams, keyPair.publicKey, signature, differentMessage)

console.log(`Signature verification for different message is: ${failedVerification}`);

Having the above code snippet in index.js file and running via the following command deno run index.js would print out the following:

Signature verification is: true
Signature verification for different message is: false

In Hash Function in Action: Message Authentication Codes I introduced Message Authenticated Codes (MAC). Which involves two parties making use of a shared secret key to ascertain the integrity of messages.

It is worth mentioning that digital signatures are the asymmetric version of message authentication code, with the extra convenience that it does not require parties to perform any key exchange.

In this post, we are going to look at asymmetric encryption. As mentioned in the previous post on symmetric encryption, encryption is the cryptographic primitive that guarantees confidentiality, by which we mean the ability for two or more authorized parties to communicate, without an unauthorized person being able to decipher the messages being communicated.

In symmetric encryption, both parties need to use the same key for encryption and decryption. This leads to the problem of how to securely get the communicating parties to use the same key. A solution to this is the Key exchange protocol that was discussed in Introduction to Key Exchange for the Working Developer

Asymmetric encryption is different. Unlike symmetric encryption, it does not have the requirement that the same key needs to be used for encryption and decryption. Hence why it is called asymmetric. It makes use of key pairs instead. One is called the private key, the other called the public key.

In this post, we will be looking at how to use asymmetric encryption and what other general information to be aware of. As always it is targeted at the working developer who needs a grounded understanding of these cryptographic primitives without necessarily covering the internals of these primitives.

Oftentimes when private/public keys are mentioned, it is often done so in the context of encryption, that is, asymmetric encryption. The reality is that the use of public and private keys is not restricted to encryption and decryption alone, there are other primitives in cryptography that make use of private/public keys.

For example, the Diffie-Hellman Key Exchange, which I wrote about in Introduction to Key Exchange for the Working Developer is an example of a cryptographic primitive that makes use of asymmetric keys but has nothing to do with encryption.

Asymmetric cryptography is the broad term that refers to all the various cryptosystems that do not depend on having two identical secret keys to function. Asymmetric encryption falls under asymmetric cryptography but so do other primitives. I believe this tidbit needs to be highlighted to prevent the assumption that when public and private keys are used, they are always used in encryption and decryption.

With that clarification out of the way, let us dig into asymmetric encryption by taking a broad look at the different asymmetric encryption algorithms.

Asymmetric encryption makes use of key pairs. A private key and a public key. This pair of keys can be used to achieve confidentiality. This is because messages encrypted by a key in the pair can be decrypted by the other key.

This is how it usually works. The public key, as the name implies, is made public. This can be shared at the beginning of confidential message exchange, or it can be made available free via any other means: email, on a website, etc. The private portion, on the other hand, is kept secret. To communicate confidentially, the public key is used to encrypt a message, which is then sent to the entity that controls the corresponding private key. This encrypted message can only be decrypted by the owner of the corresponding private key.

A two-way encrypted communication then involves the communicating parties using each other's public key to encrypt messages. That is Alice uses Bob's public key to encrypt messages she wants to send to Bob. Bob can then decrypt these messages from Alice. To reply to Alice's confidentiality, Bob also makes use of Alice's public key to encrypt replies that are sent to Alice.

There are a couple of algorithms that implement the ability to encrypt and decrypt using private/public key pairs. For example ElGamal, Paillier cryptosystem, and Cramer–Shoup cryptosystem but by far, the most popular and most used algorithm is RSA cryptosystem.

The acronym "RSA" is from the last names of the three cryptographers Ron Rivest, Adi Shamir, and Leonard Adleman, who proposed the algorithm in 1977.

Like with most asymmetric cryptosystems, the underpinning of RSA and its security is based on mathematics. But since the aim of this post is to provide the necessary information needed for a working developer to form a working mental model of asymmetric encryption, we will avoid a rigorous take but gloss over the mathematical details instead. Readers who are interested in digging more into mathematics can check the additional section of this post. There, I share links that cover some of the mathematics that is fundamental to RSA.

For now, we take a bird's eye view of the RSA algorithm.

RSA is based on factorization problem. Factorization is the mathematical operation of splitting an integer into products of smaller integers. For example, the numbers 21, 7, and 3 are its factors.

The factorization problem is the observation that it is easy to find an integer given its factors, but the reverse, that is finding the factors of a given number is hard. A small number like 21 in the example above does not demonstrate this difficulty, but given a substantial large integer, finding its individual factors is extremely hard, with no algorithm to speed up the process, and finding such factors essentially involves brute force.

"Can the reader say what two numbers multiplied together will produce the number 8,616,460,799? I think it unlikely that anyone but myself will ever know."

The above challenge was posed by William Stanley Jevons in his 1874 book, Principles of Science. The number becomes known as Jevon's number. It is essentially a factorization problem challenge. It took 15 years to crack. It was factored by Charles J. Busk in 1889 who found that 89,681 and 96,079 are factors of 8,616,460,799.

Factoring is the underlying, presumably hard problem upon which the RSA algorithm is based. An in-depth explanation is beyond the scope of this post, but the general algorithm is outlined below.

Note that the outline below is often referred to as textbook RSA and this is not how RSA is implemented in practice because it is insecure. There are slight improvements mainly in the area of padding that fortifies the algorithm.

The general outline of the algorithm is as follows:

• Choose very two large prime numbers. Let's call them p and q
• Let N = p * q N is usually referred to as the modulus
• Find the Euler totient of p and q, that is T = (p-1)(q-1)
• Choose another number e where 1 < e < T and e do not share any factors with N and T
• e is used for the encryption process.
• Choose another number d where e * d mod T = 1
• d is used for the decryption process.
• The pair N and e is the public key. This is published
• The pair N and d is the private key. This is kept a secret.
• To encrypt, a message, represent the message as a numeric value, say m
• Then calculate m ^ e mod N = ciphertext c
• To decrypt ciphertext c back to m calculate c ^ d mod N

That is the general outline. To see this in practice, let's try it out with 2 prime numbers that are small enough to make demonstration possible. We choose p = 2 and q = 7

• Choose very two large prime numbers. Let's call them p and q
  • p = 2 and q = 7
• Let N = p * q N is usually referred to as the modulus
  • N = 2 * 7 = 14
• Find the Euler coefficient of p and q, that is T = (p-1)(q-1)
  • T = (2-1)(7-1) = 1 * 6 = 6
• Choose another number e where 1 < e < T and e do not share any factors with N and T
  • e = 5
• e is used for the encryption process.
• Choose another number d where e * d mod T = 1
  • d = 11 because:
  • 5 * 11 mod 6 = 55 mod 6 = 1
• d is used for the decryption process.
• The pair N and e is the public key. This is published
  • Public key is (5, 14)
• The pair N and d is the private key. This is kept a secret.
  • Private key is (11, 14)
• To encrypt, a message, represent the message as a numeric value, say m
  • Let the message be B which in numeric form we have as 2
• Then calculate m ^ e mod N = ciphertext c
  • 2 ^ 5 mod 14 = 4
  • Cipher text is 4
  • Which in our encoding scheme is D
• To decrypt ciphertext c back to m calculate c ^ d mod N
  • 4^ 11 mod 14 = 2
  • Plain text is 2
  • Which in our encoding is B - which is the original value encrypted

The underlying security of this algorithm is the fact that given the value N which is public, it is practically impossible to figure out the factors p and q. If it were possible then it would be possible to derive d which would make the whole scheme insecure.

As mentioned earlier, textbook RSA is not used in practice as it is not secure. For example, using textbook RSA to encrypt small messages is insecure, because an eavesdropper can encrypt all the small numbers and quickly observe if any of their encrypted numbers match the ciphertext. This is why in practice, RSA makes use of padding schemes that pad the message before encryption.

The very first such a padding scheme in PKCS#1. This padding scheme has been shown to also be insecure and should not be used. Hence in practice, the padding scheme recommended to be used with RSA is the Optimal Asymmetric Encryption Padding (OAEP). Hence RSA-OAEP

Let's now see how RSA-OAEP looks with a working code example.

In the code sample below, we generate a private/public key pair, use it to encrypt a text, and then decrypt it to retrieve the original plaintext. The code makes use of web crypto API so it should work in most browsers, Deno, or the latest version of node.

let plainText = "Hello world!";

const keys = await window.crypto.subtle.generateKey({
   name: "RSA-OAEP",
   // The length in bits of the RSA modulus
   modulusLength: 2048,
   // The public component e. Recommended value is 65537
   publicExponent: new Uint8Array([1, 0, 1]),
   // hash function required by OAEP
   hash: "SHA-256",
},
   // A boolean value indicating whether it will be possible
   // to export the key using SubtleCrypto.exportKey() or
   // SubtleCrypto.wrapKey().
   true,
   // An Array indicating what can be done
   // with the newly generated key
   ["encrypt", "decrypt"]
)

// We encrypt
const ciphertext = await window.crypto.subtle.encrypt(
   { name: "RSA-OAEP" },
   keys.publicKey,
   new TextEncoder().encode(plainText)
)


// We convert ciphertext to a hex string and console logged it
console.log(buf2hex(ciphertext))

// We decrypt
const decrypted = await window.crypto.subtle.decrypt(
   {
       name: "RSA-OAEP"
   },
   keys.privateKey,
   ciphertext
)

// We convert to string
const decryptedPlaintext = new TextDecoder().decode(decrypted);
console.log(`Encrypting and decrypting was successful: ${plainText === decryptedPlaintext}`)

// utility to convert ArrayBuffer to hex string
function buf2hex(buffer: ArrayBuffer) {
   return [...new Uint8Array(buffer)]
       .map(x => x.toString(16).padStart(2, '0'))
       .join('');
}

Executing the following code, for example using Deno should give the following output:

deno run index.ts
Check file:///Users/dadepoaderemi/Documents/delete/blogging/assymmetric/index.ts
3b69df6eee76c660b8c9c307f89c4a023f455b06cf8da6622c9ff2073...
Encrypting and decrypting was successful: true

RSA is inefficient. It can only encrypt a small amount of plaintext at a time, and the speed of encryption and decryption is quite slow. This drawback is not limited to RSA but applies to most asymmetric encryption algorithms. This is mainly because these algorithms implement math operations.

Because of these drawbacks, asymmetric encryption is usually not used to encrypt messages. Instead, it is used as part of what is usually referred to as hybrid encryption constructions.

The general idea behind hybrid encryption is this: use symmetric encryption for the actual encryption since symmetric encryption is more efficient while asymmetric cryptography is used to exchange the symmetric keys. The part where asymmetric cryptography is used to ensure the parties communicating have the same key is usually called key encapsulation, while the actual encryption via symmetric encryption is called data encapsulation.

Asymmetric encryption is an example of asymmetric cryptography that can be used for key encapsulation. In this case, if Alice wants to communicate confidentially with Bob, she generates a symmetric key and then encrypts it using Bob's public key. This ensures Bob can decrypt with his private key and retrieve the symmetric key to be used for encryption.

Another scheme is to use Diffie Hellman key exchange, as the asymmetric cryptography system used in exchanging the symmetric key used for actual encryption and decryption.

The idea of hybrid encryption is very practical, this has led to the creation of well-defined schemes that makes use of hybrid encryption but only expose the typical asymmetric encryption interface to the use. This means the user of such schemes only need to worry about the private and public keys, while internally the scheme makes use of symmetric encryption. This also means the user does not have to manually perform multiple steps that consist of key encapsulation and data encapsulation but only uses the private and public keys to encrypt and decrypt.

There are many hybrid encryption schemes in existence, although arguably the most widely adopted standard is Elliptic Curve Integrated Encryption Scheme (ECIES). Hence I will quickly show how ECIES looks in practice.

Unfortunately, Web Crypto API does not support ECIES, so for this demonstration, I will use nodejs and the eciesjs package.

Create a new node js project and install eciesjs by running the following command

npm init ecies_demo -y
npm install eciesjs

Then in the index.js have the following code:

import { encrypt, decrypt, PrivateKey } from 'eciesjs'

const prvKey = new PrivateKey()
const pubKey = prvKey.publicKey
const message = Buffer.from('message to b')

// encrypt the message
const encryptedToB = encrypt(pubKey.toHex(), message);

// console log the encrypted message as hex
console.log(encryptedToB.toString('hex'))

// decrypt the message
const decrypted = decrypt(prvKey.toHex(), encryptedToB).toString()

// confirm original message and decrypted message is same
const isSame = message.toString() === decrypted
console.log(`decrypted and original message is same: ${isSame}`)

This code uses a generated public key to encrypt a message which is then decrypted with the corresponding private key and confirmed that the original message and decrypted message are the same.

As can be seen, the interface only exposes the usage of public-private key pairs and hides the fact that symmetric encryption is being used. In fact, the ECIES scheme is more complex and makes use of other cryptographic primitives like key-derivation function (KDF) and HMAC,

A functional diagram of ECIES can be seen below:

Source https://core.ac.uk/download/pdf/36042967.pdf

The good thing is that the end-user does not need to deal directly with this complexity. A simpler interface based on private and public keys is exposed. This is the advantage of using a hybrid encryption scheme like ECIES.

Here are some of the link that covers some of the fundamental mathematics that makes asynchronous encryption work

• GCD
• Relative Prime 
• Primitive Root
• Euler’s ϕ-function
• Modular Arithmetic
• Prime Numbers & RSA Encryption Algorithm - Computerphile

The truth of the matter is, the async/await features in other programming languages seem to be more straightforward than it is in Rust. For example in JavaScript, you have the syntax that you use, and that is about it. But in Rust, there seem to be more moving parts.

Tokio? Future? There is a futures crate? Runtimes? How do all these fit into the async/await picture in Rust?

This is a short post to help clarify some of the terms a developer would encounter when they start exploring asynchronous programming in Rust. It would explain the main moving parts, which are: the standard library's std::future::Future, the futures crates, and asynchronous runtimes like tokio and async-std.

Let get started:

The crux of asynchronous programming is to maximize computing resources, especially IO-bound computation. One of the ways this is done, is to code in such a way that the computation is not directly executed but written in a suspended form that can be later executed or paused, or retried.

The type used to express such a computation is std::future::Future. For example, a print statement that is not executed directly can be written is as follows:

use std::future::Future;
fn future_print() -> impl Future<Output = ()> {
    async {
        println!("Hello World!");
    }
}

The return type impl Future<Output = ()> means this is a function that returns any type that implements the std::future::Future trait.

The std::future::Future is part of the standard library and there is no need to depend on any external crate to have access to it.

To confirm that the future_print function does not execute directly, let us try and call it in main to see what happens:

use std::future::Future;

fn future_print() -> impl Future<Output = ()> {
    async {
        println!("Hello World!");
    }
}

fn main() {
    future_print();
}

The output should be something similar to:

warning: `async_await` (bin "async_await") generated 2 warnings
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/async_await`

Process finished with exit code 0

As you can see, there is no Hello World printed to the console. Confirming that the std::future::Future  denotes computation that has been described but that is not executed directly and this is the crux of asynchronous programming in Rust.

In practice, there is usually no need to use the std::future::Future type directly as the return type. Rust has a syntactic sugar in the form of the async keyword that makes the code less noisy. The above code snippet can be re-written as:

async fn future_print() {
    println!("Hello World!");
}

fn main() {
    future_print();
}

This is less noisy, even though it is essentially the same as the verbose code that directly uses the std::future::Future type.

The question now is, how do a method that returns a future be executed? To answer this we look at the next piece of the async/await story in Rust. The futures crate, which can be found here

Formally futures-rs. This crate provides the other core foundational components needed for asynchronous programming in Rust.

The components it provides, are not included in the standard library like std::future::Future, and hence to bring them in, the crate needs to be defined as a dependency in Cargo.toml. It can be done as follows:

[dependencies]
futures = "0.3.21" # change to the current version

Even though it is not included in the standard library it is worth noting that the futures-rs crate is developed by the Rust core team, as can be seen from the fact that the code repository is under the rust-lang namespace on Github

The futures-rs crates include key trait definitions like Stream, as well as utilities like join!, select!, and various futures combinator methods that enable expressive asynchronous control flow.

This post won't look at the utilities like join!, select!, instead we go back to the initial question we are yet to answer: We have a function that returns a Future how do we run it?

To run a Future in Rust, there is a need for an asynchronous runtime. This runtime is usually called the executor. The executor is a machinery that is tasked with the management of all the resources required for executing futures. It takes a computation described as a Future and ensures it is executed asynchronously.

The futures-rs crate comes with such an executor which can be accessed via futures::executor::block_on.

The following code snippet shows its usage:

use futures::executor::block_on;

async fn future_print()  {
    println!("Hello world");
}

fn main() {
    block_on(future_print());
}

Now if the following code is executed, the output should look like the following:

/Users/dadepoaderemi/.cargo/bin/cargo run --color=always --package async_await --bin async_await
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/async_await`
Hello world

Process finished with exit code 0

Confirming that we can execute the future code with the usage of futures::executor::block_on

At this point, we have looked at std::future::Future which is part of the standard library and is used to describe computation to be executed later by a runtime. We also looked at the futures-rs crate, which provides such a runtime in the form of the futures::executor::block_on executor.

The only problem is that the futures::executor::block_on executor is quite primitive and not powerful enough to be used in a real life production environment.

To fill the gap for more powerful, feature-rich asynchronous runtimes, various crates have spurned up within the Rust ecosystem. Some popular ones include: Tokio, async-std, and smol.

They all do the same thing: Provide a sophisticated and more powerful runtime that executes asynchronous computations.

We will use Tokio to demonstrate. Update the dependencies as follows:

[dependencies]
futures = "0.3.21" # change to the current version
tokio = { version="1.18.2", features = ["full"] } # change to the current version

Note that using external runtimes like tokio does not necessarily remove the need for the futures-rs crate, as it contains useful utilities that can still be used alongside the runtime.

Using the tokio runtime would look something like this:

use tokio::runtime::Runtime;

async fn future_print() {
    println!("Hello world")
}

fn main() {
    Runtime::new().unwrap().block_on(future_print());
}

This works. The only problem though is that this is a bit verbose. To make the code less noisy, we use another Rust language keyword related to async/await, the .await, and a macro from Tokio: #[tokio::main].

In code, it looks like this:

async fn future_print() {
    println!("Hello world")
}

#[tokio::main]
async fn main() {
    future_print().await;
}

Running the above code would print something like this into the console:

/Users/dadepoaderemi/.cargo/bin/cargo run --color=always --package async_await --bin async_await
    Finished dev [unoptimized + debuginfo] target(s) in 0.02s
     Running `target/debug/async_await`
Hello World!

Seeing the Hello World! confirms that we were able to use a runtime provided by Tokio runtime to execute the computation.

Using async/await in Rust is not as straightforward as just using the language syntax as it is with languages like JavaScript or C# etc that have similar asynchronous constructs.

More moving parts might not be obvious to the beginner Rust developer. This post gave a quick overview of the crucial components:

The std::future::Future type is part of the standard library that forms the crux of the asynchronous programming model as it is used to denote computation that can be executed asynchronously.

The futures-rs crates, maintained by the Rust team, provide more of the foundations for asynchronous programming in Rust. It comes with a simple executor that can be used as a runtime to execute asynchronous code.

And finally, we looked at Tokio, as an example of an external crate that provides a more sophisticated asynchronous runtime that is usually used in a production-like environment.

Other well known crates providing asynchronous runtimes include smol, and async-std.

In summary, the core async/await landscape can be described as comprising the standard library type of std::future::Future which denotes asynchronous computation. The futures-rs crate provides additional foundational components: traits, macros utilities, etc and then finally the 3rd party runtimes that are external to the standard library which provide sophisticated executors used for running asynchronous code.

This post does not attempt to go into details about how to use async/await in Rust. It seeks to just be the introductory text that explains the lay of the land. To learn more about asynchronous programming in Rust, you can check the following resources:

• Crust of Rust: async/await
• Async/await in Rust: Introduction
• Asynchronous Programming in Rust

In Introduction to Symmetric Encryption for the Working Developer, we saw how to encrypt and decrypt data. Being symmetric encryption, both the process of encryption and decryption needed the same key. This poses a challenge, which was not addressed in the post: How do one get across the same key to both parties who want to engage in encrypted communication?

Basically how do we perform key exchange? How do we get the same key to parties who want to encrypt and decrypt messages?

An approach is perhaps for the parties involved to first meet in person and exchange the key?

Another approach could be to use another communication channel to first share the key? But then again, if the other communication channel is digital, and it uses symmetric encryption, how should its encryption key be also exchanged? This becomes a catch 22 real quick.

This is the problem Key exchange schemes seek to solve, and thankfully there is a secure way to have two parties exchange secret keys without the need to physically meet in person. This is what this post is about. We will be looking at two popular mechanisms for key exchange: Diffie-Hellman key exchange procedures and application of RSA for key exchange.

As always, as with the other posts in this series, the idea is to provide the basic information needed by the working developer to be able to understand and use these cryptographic primitives without going into the thick of their internal details or implementation.

This post contains the following sections

• Entering the realm of Public Key Cryptography
• Introduction to Diffie-Hellman Key Exchange: An Intuition.
• Whirlwind tour of the Mathematics
• Diffie-Hellman in code
• Diffie-Hellman Standards
• Using RSA for key exchange
• Conclusion and References

Public key cryptography (or Asymmetric key cryptography) is a class of cryptography primitives that makes use of dissimilar keys. This is different from Symmetric key cryptography which makes use of similar keys for its operations.

Public key cryptography also relies more heavily on Mathematics, especially number theory, and the security properties and guarantees it provides is often as a result of a direct consequence of some mathematical theory. Hence in discussion of cryptographic primitives that can be classified as public key cryptography there is more encounter of mathematical notations and theorems.

This post will not go into the mathematical underpinning of the cryptography primitives discussed. Instead, I will highlight the mathematical theories in this post and point to resources, motivated readers can check out to learn more.

Let's start by exploring one of the two schemes that uses public key cryptography for key exchange: that is: Diffie-Hellman.

The Diffie-Hellman key exchange is a method that allows two parties, communicating in a public, insecure channel, to exchange messages that allows them to derive the same secret key. But the method has the property that eavesdroppers observing the messages being exchanged won't be able to use it to construct the secret key derived by the parties engaged in the key exchange.

How is this possible? To have two parties exchange two numbers, that anyone can see, and then, separately, these two parties use the numbers they receive from each other to compute another number privately, and the number they compute will end up being the same (ie the secret key) but the people that observed the numbers that was exchange cannot do the same computation to arrive at the secret key?

A good intuition to have about how the process works is this. It is a process for two parties to first exchange two secret numbers. But they do not just exchange this secret number, they first mix it up with another number both parties agree with.

So each party comes up with a secret number, mixes it up with a publicly agreed number and shares.

After the sharing, both parties have the secret number generated by the other party mixed up with an agreed number. Anyone observing the exchange can only see the results of the mixed number (that is, party A's secret number mixed with an agreed number which is not secret and party B's secret number mixed with the same agreed number). The security of this exchange boils down to the inability to break apart the result of this number mixing to reveal the secret number that was mixed with it. In Diffie-Hellman, this is impossible to achieve based on the discrete algorithm problem which will be briefly explored later on. But for this intuition, we agree that, parties mixed numbers and share, and anyone observing cannot break this number apart.

After the mixing and sharing what do we have? We actually have both parties having each other's secret number. Only that, this secret number they have of each other is not "bare" but mixed up with an agreed number.

What then happens is that both parties perform a computation using their secret number on the other party's secret number (but mixed up in an agreed number) and they will both arrive at another secret number which can be used as the secret key.

That is the intuition. The Diffie-Hellman is a process for two parties to first initially exchange a secret number mixed up with an agreed and non-secretive number, and then use this to perform a computation. What ends up happening is that the inputs to the computation on each party's side are the same. That is for party A, it is compute(secret A, M1 = (mixing number + secret B)) while for party B, it is compute(secret B, M2 = (mixing number + secret A)).

Even if the mixing number is known, as long as it is impossible to break M1 or M2 apart to retrieve secret B or secret A, then the scheme works.

That is the intuition of the process. Now let's dig deeper into how things work, without also going into too much of the mathematics.

The idea of this section is to give an overview of the key concepts from mathematics that come together to make Diffie-Hellman possible. For obvious reasons, an in depth coverage of these concepts is not possible in this one post, but links would be sprinkled in pointing to where more information can be gotten.

So let's start.

We all know about sets from elementary mathematics. When sets fulfil certain characteristics, they are said to form a group. These characteristics include Closure, Associativity, Presence of an Identity Element and presence of an Inverse element. The study of these characteristics is called group theory.

A key part of a group is the presence of a binary operation. That is an operation that operates on two inputs to produce an output that is also a member of the set. Examples of binary operations include addition, or multiplication or Modulo multiplication. Modulo operation are mathematical operations involving numbers that wrap around. It is also often called clock arithmetics.

The two flavors of Diffie-Hellman algorithms that we have, Diffie-Hellman in a modulo multiplicative group and Elliptic Curve Diffie Hellman (ECDH) both make use of groups and binary operations.

In the first flavor, the group includes the set of strictly positive integers 1, 2, p – 1 for p a prime number, along with modular multiplication, where the modulo is a prime.

The second flavor makes use of groups that form an elliptic curve, while the binary operation is addition. The algorithm for ECDH is a bit involved when compared hence a quick overview of the first flavor is given in this post.

So as stated, the first (or classic) Diffie-Hellman algorithm makes use of groups made up of positive integers, with binary operation being multiplication with the modulo being a prime P.

The algorithm involves picking a number that forms a primitive root with the modulo prime. This number that we pick when multiplied with itself ad-infinitum, in modulo prime, will generate a cyclic subgroup, ie when it is raised to power n, where n = 1,2,3, (infinity - 1), under modulo prime P, will produce a cyclic group. This extra characteristic is important in order to ensure the discrete logarithm problem is sufficiently hard enough to ensure the security of the algorithm.

And the discrete logarithm problem is a class of problems that is sometimes referred to as one way function, i.e. functions easy to compute given inputs, but hard or almost impossible to retrieve the inputs given the results of the computation. The discrete logarithm problem simply states that, given a number x and y where y is greater than x, it is hard to find the number of times n, x needs to be multiplied by itself to result into y, under modulo of another number p.

In the classic Diffie-Hellman algorithm, a prime number P and an integer g, the generator is agreed upon by the two parties. This is what they use to mix the secret number they want to pass on to each other that would be used to derive the private key.

Party A generates a secret number sa, and then mixes it up with prime p and generator g by computing = AK = g^sa mod p

AK is referred to as a public key and Party A sends this value to party B.

Party B generates a secret number sb, and then mixes it up with prime p and generator g by computing = BK = g^sb mod p

BK is referred to as a public key and party B sends this value to party A.

At this point, party A has BK, while party B has AK.

Party A now computes the secret key by computing SK= BK^sa mod p and party B also computes the secret key by computing SK= AK^sb mod p. They both arrive at the same value because:

BK^sa = (g^sb)sa = g^(sb)(sa) = (g^sa)sb = AK^sa mod p

The two values publicly transmitted are AK = g^sa mod p and BK = g^sb mod p.

g and p are known, AK and BK are also known, as they are communicated in the clear. But anyone observing these numbers won't be able to derive what sa or sb is. Because of the discrete logarithm problem.

It is worth mentioning that in practice, the SK arrived is not used directly as the secret key, but usually passed through a Key derivation function.

Let's see how this procedure looks in code.

Before showing the code, let's go over the steps again.

As stated, the objective is for two parties to arrive at the same secret key that can be used for symmetric encryption, without having to directly share the key.

A rough step by step outline of how it works is as follows:

• Both Alice and Bob agree on using Diffie-Hellman. This involves agreeing on two numbers. One small usually referred to as g, and another big, which is a prime number, usually referred to as p.
• Alice generates a secret number. Performs a modulus computation with it together with g and the prime number p. This computation derives another number called the public key. This is why this procedure falls under Public Key Cryptography.
• Bob also does the same.
• Alice gives Bob her public number, while keeping her secret one. Bob also gives Alice his public number and keeps his secret number. Now Alice has her secret number and technically Bob's secret number mixed within Bob's public number. Bob also has his secret number and Alice's secret number mixed in the public number he received from her.
• Alice uses her secret number and Bob's public number to calculate another number
• Bob does the same, using his own secret number and Alice's public number.
• Both Alice and Bob will then arrive at the same generated number which can then be used for symmetric encryption.
• Eve who observed all the exchanges, could know the algorithm being used and can see all the numbers exchanged, including the g and p agreed upon, won't be able to compute the shared secret both Alice and Bob compute independently.

The following Javascript code snippets in nodejs demonstrates how the Diffie-Hellman scheme can be put to use.

const crypto = require('crypto')
const assert = require('assert');


// -- Select mod p group

// Alice is using the modp15 group
const groupA = crypto.getDiffieHellman('modp15')
// const groupA = crypto.createECDH('brainpoolP160t1')

// Bob is also using the modp15 group
const groupB = crypto.getDiffieHellman('modp15')
// const groupB = crypto.createECDH('brainpoolP160t1')

// The modp15 group just like others define same generator and prime. hence the prime and generator for groupA and groupB should be same
assert(groupA.getPrime().toString("hex") === groupB.getPrime().toString("hex"))
assert(groupA.getGenerator().toString("hex") === groupB.getGenerator().toString("hex"))


// -- Generate the keys

// With this call, the secret key k1 is generated
// The k1 is then use to generate the public key, which is = g^k1 mod p
// Note that new k1 is generated on each invocation of this function.
groupA.generateKeys()

// With this call, the secret key k2 is generated
// The k2 is then use to generate the public key, which is = g^k2 mod p
// Note that new k2 is generated on each invocation of this function.
groupB.generateKeys()


// At this point the public key can be shared with other parties which can then be used to arrive at the shared secret
// Not that the private key is also available via getPrivateKey call but there should be no need to retrieve this. Having
// this value known by a malicious actor will compromise the integrity of the protocol
console.log(`Public key of group A ${groupA.getPublicKey().toString("hex")}`)
console.log(`Public key of group B ${groupB.getPublicKey().toString("hex")}`)


// --- Now each party use the other public key to generate the shared secret

let sharedSecretA = groupA.computeSecret(groupB.getPublicKey())
let sharedSecretB = groupB.computeSecret(groupA.getPublicKey())


// Confirm that same value was arrived at

assert(sharedSecretA.toString("hex") === sharedSecretB.toString("hex"))

The Diffie-Hellman scheme is made up of components that can be varied. Depending on how these components are configured, the security guarantees of the scheme can be affected.

The main components that are configurable is the value of g (the generator), and p (the prime) to be used.

Thing is, anyone implementing Diffie-Hellman key exchange is free to pick whatever value they think is secured for these two components. For example this file contains some popular Diffie-Hellman configuration observed in the wild.

The only problem is, not all values lead to a secured scheme. For example in 2016, it was discovered that socat a networking utility, was using Diffie-Hellman configuration that makes it insecure. Hence as a developer, implementing or using any service that makes use of Diffie-Hellman key, to be aware what secured configurations are being used.

To help with this, there are recommendations stated in various RFCs. These recommendations are largely categorized into two: Recommended values to be used for multiplicative groups and those based on elliptic curves.

Recommendation for DH For the multiplicative groups, it is advised to use prime numbers whose values are not less than 2048 bits. This ensures that a minimum of 128 bit of security is provided. The recent RFC that contains up to date best practices and recommendations can be found in RFC 7919

Recommendation for ECDH For the Elliptic curve diffie-hellman, there are a couple of curves that most applications have standardized around. These include P-256, Curve25519, and Curve448. The curve P-256, and Curve25519 provides 128 bit of security, while Curve448 offers around 224 bit of security. When Curve25519 is used to implement the Diffie-Hellman key exchange, it is also referred to as X25519. ECDH is preferred over DH due to its smaller key size. To get a 128 bit of security DH requires nothing less than 2048 bits, but with ECDH, the requirement is nothing less than 256 bits.

RSA is a popular asymmetric crypto-system which can also be used for exchanging keys. RSA by itself is a distinct topic that should be covered separately but for completeness it is mentioned here as another way for two parties to exchange secret keys.

Summary

• Two parties can arrive at the same secret number by performing the Diffie Hellman key exchange. This helps solve the problem of how two parties securely share the secret key needed for encryption.
• Diffie Hellman requires numbers that form what is mathematically known as groups. Two popular groups are widely used: Multiplicative groups modulo a prime number and Elliptic curves
• When Multiplicative groups modulo a prime number is used, it is usually referred to as just Diffie-Hellman ie DH
• When Elliptic curves are used, it is usually referred to as Elliptic Curve Diffie Hellman ie ECDH
• The recommended standard to use for DH is found in RFC 7919
• The popular curves used in ECDH are P-256, Curve25519, and Curve448, with Curve25519 being quite popular.
• When Curve25519 is used to implement the Diffie-Hellman key exchange, it is also referred to as X25519.
• ECDH is preferred over DH due to its smaller key size. To get a 128 bit of security DH requires nothing less than 2048 bits, but with ECDH, the requirement is nothing less than 256 bits.

Mathematical Concepts

• Introduction to set theory
• Introduction to group theory
• Subgroup Generators
• Modular Arithmetic
• Discrete Log Problem
• Elliptic Curves
• Diffie-Hellman Explained

In Introduction to Symmetric Encryption for the Working Developer, I presented an overview of symmetric encryption. Which as explained in the post, is a process for establishing confidentiality during communication between two parties.

The only problem is that just encrypting data using symmetric encryption does not provide all the security requirements needed during secure communication.

The issue is that, it is still possible for an eavesdropper to tamper with the message even without being able to make sense of the ciphertexts, and the receiving party won't be made aware that the message they are decrypting has been tampered with and is not exactly the original one that was sent.

Hence even though encryption gives us confidentiality, done alone, it does not guarantee the other security requirements like authenticity/integrity that is often needed - i.e. it does not guarantee that a message came from the right party (authenticity) and it has not been modified in any way (integrity).

Encryption alone is not enough, we need authenticated encryption. This post will introduce the concept of authenticated encryption.

In Hash Function in Action: Message Authentication Codes a similar argument was made regarding hash functions. Where it was stated that hash functions are never enough and there is the need for message authentication to also ensure things like authenticity and integrity.

How then can authenticated encryption be achieved? Do we use a similar approach that was used in message authentication?

Well, an approach to achieving authenticated encryption is to take the output of encryption and then apply message authentication code to it.

For example, using HMAC to create an authentication tag (hash/digest of the cipher text)). Both the ciphertext and the authentication tag is then sent during communication.

In general such manual constructions are referred to as EtM (Encrypt-then-MAC) and a specific construction of it would be AES-CBC-HMAC. Which is a construction where you encrypt using AES-CBC and then create an authentication tag using HMAC.

The problem with this approach is that it is error prone. It is fickle and possible to make simple errors that lead to tremendous security issues.

For example one needs to ensure that the same key is not used for both encryption and generating the HMAC.

Also special care needs to be taken that the IV values used for the AES are truly random and never reused. Also because two separate cryptographic primitives need to be separately implemented, it does not make for the most efficient approach.

Because of these problems with manually constructed EtM schemes, it is not recommended to manually construct authenticated encryptions. Instead the safe advice is to make use of all-in-one constructions that simplify the process of having authenticated encryption.

This is what is discussed in the next section.

All-in-one authenticated encryptions can either be based on Advanced Encryption Scheme (AES) or using another algorithm apart from AES.

The all-in-one construction for achieving authenticated encryption that builds on AES does so by having an AES mode of operations that include authentication. For more about modes of operation, see Introduction to Symmetric Encryption for the Working Developer.

They do so in such a way as to hide the complexity and the various moving parts, presenting a much simpler interface to developers.

These all-in-one constructs also provide the ability to optionally attach some metadata with the ciphertext, if that is the case, then they are called Authenticated Encryption with Associated Data (AEAD).

An example of such a mode that supports both authentication with the ability to attach associated data is Galois/counter mode (GCM).

The associated data that is attached is unencrypted.

What then is the use you might ask?

The associated data is for scenarios where you have data that does not need to be hidden, but must go along with some other data that needs to be hidden.

An example of such a scenario is a network packet, where the body which contains the data should be encrypted, while the headers, which do not contain any sensitive data and are used by network equipment like routers, can remain in plaintext. But even though the headers are in plaintext, each header needs to be associated with the encrypted data it belongs with. Using AEAD, the header can be the associated data. This means that the same associated data used during encryption, needs to be provided on decryption. If the associated data has been tampered with, the decryption process will fail.

The associated data is a flexible component of AEAD, and can be used for anything. The general idea is that it provides some sort of encryption context. This encryption context can be whatever that is required for it to be.

There are other modes of operation that can be used with AES that are AEAD. This Wikipedia link provides an overview of some of this mode.

There is also ChaCha20-Poly1305 which is an AEAD. It is a separate encryption scheme and not just a mode of operation of AES.

Even though ChaCha20-Poly1305 is newer than AES and not a standard, it has grown to be a solid scheme and a popular alternative to AES.

The rest of this post looks at AES-GCM and ChaCha20-Poly1305.

AES-GCM is an AEAD. It is constructed from AES used with the Galois/Counter Mode. By it being an AEAD, it means it provides authenticated encryption out of the box (the AE part in AEAD), with the ability to add associated data (the ED part in AEAD).

The Web Crypto API supports AES-GCM, hence can be used within a browser or Deno. Using it is similar to the code snippet presented in Introduction to Symmetric Encryption for the Working Developer. The only difference is, instead of passing in "AES-CBC" and the algorithm identifier, the string "AES-GCM" is used instead. The updated code is reproduced below. It also makes use of associated data.

// We generate a key
// See https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/generateKey
let key = await crypto.subtle.generateKey({
   name: 'AES-GCM',
   length: 128
}, false, ['encrypt', 'decrypt']);

// We generate random 16 bytes to be used as initialization vector
let iv = new Uint8Array(16);
// and fill it with random numbers
// see https://developer.mozilla.org/en-US/docs/Web/API/Crypto/getRandomValues
await crypto.getRandomValues(iv);

// The web crypto requires data to be in ArrayBuffer
let plaintext = new TextEncoder().encode("hello world");

// The encryption and decryption requires specifying certain parameters:
// the algorithm and initialization vector, associated data etc
// see https://developer.mozilla.org/en-US/docs/Web/API/AesGcmParams
let associatedData = new TextEncoder("utf-8").encode("Greeting");

let param = {
   name: 'AES-GCM',
   iv,
   associatedData,
};

// We encrypt the plaintext to get the ciphertext
let ciphertext = await crypto.subtle.encrypt(param, key, plaintext);

// We decode and console logged the ciphertext
console.log(new TextDecoder("utf-8").decode(ciphertext))

// We decrypt the ciphertext back to plaintext
let deCryptedPlaintext = await window.crypto.subtle.decrypt(param, key, ciphertext);

// We tranform the plaintext from bytes to string and then print it out
console.log(new TextDecoder("utf-8").decode(deCryptedPlaintext));

// Causing decryption to fail by providing different associated data

// Create another associated data and param
let wrongAssociatedData = new TextEncoder("utf-8").encode("Payment");

let wrongParam = {
   name: 'AES-GCM',
   iv,
   wrongAssociatedData,
};


// Decryption would fail because the associated data used for decryption "Payment"
// is different from one used for encryption "Greeting"
await window.crypto.subtle.decrypt(wrongParam, key, ciphertext);

The code above makes use of AES-GCM as the AEAD. It provided an associated data. It demonstrates encryption and decryption. It also shows that the decryption will fail, if a different associated data, from the one used at encryption is attempted for decryption.

Unlike AES-GCM, (which is an AES), ChaCha20-Poly1305 is an entirely different encryption scheme designed by Daniel J. Bernstein. It is really a combination of a stream cipher (ChaCha20) and a MAC (Poly1305).

Even though ChaCha20-Poly1305 is not standardised by NIST, it has been specified as an RFC by Google in rfc7539. It has also seen massive adoption in the industry, being supported in protocols like OpenSSH, TLS etc. Hence it is a battle tested encryption scheme and can be safely used.

As of this writing, Web Crypto API does not support ChaCha20-Poly1305, so by extension is not available in a browser environment or Deno.

On the other hand, ChaCha20-Poly1305 can be used from Node since the Node crypto API depends on OpenSSL. So if you have the version of OpenSSL that supports ChaCha20-Poly1305, it can be used via Node.

The code snippet below shows how to use ChaCha20-Poly1305 in Node.

const { randomBytes, createDecipheriv, createCipheriv } = await import('node:crypto');

// chacha20-poly1305 requires 256 bit (32 byte) as key 
const key = randomBytes(32);
// Also a 96 bit iv/nonce
const iv = randomBytes(12);;
// create some associated data
let aad = Buffer.from("Greeting")

// create the cipher
let cipher = createCipheriv('chacha20-poly1305', key, iv, { authTagLength: 16 });

// Add the associated data to the cipher
cipher.setAAD(aad, {
  plaintextLength: Buffer.byteLength(plaintext)
})

// Data to encrypt
let plaintext = "Hello world";

// Encrypt and finalize encryption
const ciphertext = cipher.update(plaintext);
cipher.final()

// Console log the cipher text in hex
console.log("ciphertext:", ciphertext.toString('hex'))

// Decrytpion

// Create the decipher using exactly same parameters used to create the cipher
let decipher = createDecipheriv('chacha20-poly1305', key, iv, { authTagLength: 16 });
// Get the auth tag from the cipher and set it on the decipher
// This ensures the authentication done is done on decryption
decipher.setAuthTag(cipher.getAuthTag())
// We set the same associated data used in encryption
decipher.setAAD(aad);
// We start the decryption
let decrypted = decipher.update(ciphertext)
// We finalise the decryption and save the plaintext in a variable
decrypted = Buffer.concat([decrypted, decipher.final()])
// Print out the plaintext
console.log("plaintext:", decrypted.toString())

console.log("Encryption then Decryption Works", plaintext.toString() === decrypted.toString());

// Causing decryption to fail by providing different associated data

// Create the decipher using exactly same parameters used to create the cipher
let wrongDecipher = createDecipheriv('chacha20-poly1305', key, iv, { authTagLength: 16 });
// Get the auth tag from the cipher and set it on the decipher
// This ensures the authentication done is done on decryption
wrongDecipher.setAuthTag(cipher.getAuthTag())
// We set the another associated data, that differs from one used in encryption
let wrongAssocData = Buffer.from('Payment');

// We set the wrong associated data
wrongDecipher.setAAD(wrongAssocData, {
  plaintextLength: ciphertext.length
});

// We start the decryption
let wrongDecrypted = wrongDecipher.update(ciphertext)
// We finalise the decryption and save the plaintext in a variable
try {
  // This leads to exception
  wrongDecipher.final();
} catch (err) {
  throw new Error('Authentication failed!', { cause: err });
}

The code above makes use of chacha20-poly1305 as the AEAD. It provided an associated data. It demonstrates encryption and decryption. It also shows that the decryption will fail, if a different associated data, from the one used at encryption is attempted for decryption.

Encryption is all about establishing confidentiality. That is, making sure only authorised parties have access to specific data. It involves transforming data into a form that is inscrutable to unauthorised parties but with the ability for authorised parties to transform the data back into its legible form.

Think about two parties communicating without wanting other persons privy to the messages. This is achieved via the cryptographic primitive of Encryption.

In this post, I will provide a whirlwind tour of what encryption is for the working developer. The goal is to provide the basic information needed to be able to properly wield this cryptography tool without going into the inner workings of the algorithm. It is part of the Cryptography101 series of posts.

This post contains the following

• What Encryption is and its main components
• What Encryption is not
• Symmetric Encryption vs Asymmetric Encryption.
• Block Cipher vs Stream Cipher.
• Overview of encryption Standards
• Components of AES
  • Key
  • Padding
  • Mode

Encryption is the part of cryptography that deals with ensuring confidentiality of data. This usually involves taking data and transforming it into what looks like gibberish to unauthorised parties.

Encryption can be used as part of secured communication, where two parties communicate without any eavesdropper being able to make sense of the communication. It is also used to encrypt data at rest, for example the data on a harddisk; making sure it is only accessible by the owner or any other authorised parties. This can also be seen as securing communication, where the other party is the future self that needs to access the content of the harddisk.

It should be noted that encryption must also include the reversible component; that is decryption. This is the process of taking the gibberish output of the encryption process and transforming it back to the original data. Without being able to decrypt, encryption becomes useless.

Both the encryption and decryption process makes use of a critical component called the secret key. The secret key together with the encryption/decryption algorithm is what is used to transform a message into gibberish and also to recover the original message. As long as an eavesdropper does not know the secret key, there is no way to access the message. Hence the secrecy of the key is at the heart of the security of encryption.

The encryption, decryption and secret key generation algorithm is oftentimes together referred to as Encryption Scheme.

The data to encrypt is usually called plaintext, the algorithm that encrypts the data is usually called cipher and it makes use of a secret component called key. The encrypted data is usually called ciphertext.

Just as it is important to say what encryption is, it is also important to make clear what encryption is not.

Cryptographic Hashing is another cryptography primitive that is sometimes confused with encryption. A cryptographic hash algorithm takes data and produces a random bit string, often called digest, which serves as a form of fingerprints of the data.

Hashing is a one way operation, that is, with a secure hashing algorithm, it should be practically impossible to retrieve the original data from the digest. This is in contrast with what you want with encryption. With encryption you need to be able to retrieve the original message.

See Introduction to Cryptographic Hash Functions for the Working Developer for more about Cryptographic Hash Functions.

Encoding/Decoding schemes like base64 or even representing data in hexadecimal or binary string is not encryption even though they might also appear to be hiding data. They are not encryption because they do not make use of any secret key, hence they provide no confidentiality since anyone can perform the reverse decoding operation on the encoded data.

In infosec there is sometimes the need to hide the fact that communication between two parties is taking place. This usually involves hiding a secret message in something that is not secret. This is done via Steganography. This is different from Encryption which does not seek to hide that communication is taking place but only to hide the message that is being communicated.

With Steganography an eavesdropper won't be aware that communication is taking place.

With encryption an eavesdropper can observe communication taking place but won't be able to see what is being communicated.

So having touched on a couple of things that are not encryption, let us go on to see some different aspects of encryption that are often used to put them into different categories or types.

As mentioned earlier, the process of encryption depends on the usage of a secret key. You then have two flavours of encryption depending on how this secret key is used.

An encryption scheme where the same key is used for both encryption and decryption is called symmetric Encryption. While an encryption scheme where the key used for encryption is different from the key used for decryption is called asymmetric encryption (also known as asymmetric cryptography or public key cryptography).

This post is about symmetric encryption

Encryption schemes can also be categorised based on how the plaintext is passed on to the encryption algorithm.

Encryption schemes that require the plaintext to be encrypted a fixed size at a time are called Block ciphers.

This involves having the plaintext broken down to a certain fixed bit size, and each bit size encrypted. This is a hard requirement, hence if the plaintext or the last piece of the chopped up plaintext is not up to the required fixed size, it would have to be padded till it meets the required size.

Stream ciphers, on the other hand, can encrypt plaintexts at varied bits at a time. Since they do not require a fixed block to be passed in for encryption, stream ciphers hence do not have a need for padding.

This post is about symmetric encryption that are block ciphers.

As stated in Introduction to Cryptographic Hash Functions for the Working Developer there are different agencies that publish cryptographic related standards. In this post, we only limit to standards published by National Institute of Standards and Technology (NIST).

When it comes to symmetric encryption, the go to encryption scheme as standardised by NIST is Advanced Encryption Standard, usually referred to as just AES.

As with most standards by NIST, what later came to be the AES was as a result of an open competition that started in 1997. The competition was an open call to all interested cryptographers to come up with an encryption scheme to replace the then standard scheme: Data Encryption Standard usually referred to as DES.

In 2001, the competition was concluded and out of 15 different submissions, the Rijndael encryption scheme, designed by Vincent Rijmen and Joan Daemen emerged the winner and was named the AES.

AES has since then been the main encryption scheme. Replacing DES, which for all intents and purposes is broken and should not be used. Hence the remaining part of this post would be basically an overview of AES.

In this section, I will go over some of the key components of AES. This are key size, padding schemes, and mode of operation.

Like any symmetric encryption scheme, AES also requires a secret key. AES comes in 3 different versions depending on the size of the key.

We have:

• AES-128 takes a key of 128 bits (16 bytes)
• AES-192 takes a key of 192 bits (24 bytes)
• AES-256 takes a key of 256 bits (32 bytes)

The size of the key dictates the level of security. Larger keys provide higher security but for almost all use cases, AES-128 will suffice.

AES keys are random bits. That means any method that can generate true cryptographic random numbers of 128, 192, and 256 bits can be used to generate AES keys. As mentioned earlier on, the security of AES depends on the key, hence the methods used to generate the key needs to be truly random and unguessable.

For example, in JavaScript environment with Web API available, like in the browser or Deno, window.crypto.getRandomValues(new Uint8Array(16)) can be used to generate random 256 bits that can be used as a AES-256 key.

In Deno, this look like:

Deno 1.16.4
exit using ctrl+d or close()
> let counter = window.crypto.getRandomValues(new Uint8Array(16));
undefined
> console.log(counter)
Uint8Array(16) [
  189,  59,  14, 161,  84,
  117, 160, 119, 175,  96,
  130, 233, 121,  99, 176,
  240
]

AES is also a block cipher, hence it can only encrypt a fixed size at a time. AES block size is 128 bit hence it can only encrypt plaintext 128 bit at a time.

This means if your plaintext is larger than 128 bit, it would have to be broken up into blocks of 128 bit each. If the plaintext is less than 128 bit, or after breaking into 128 bits blocks, there remains a last piece that is less than 128 bit, then that piece needs to be padded somehow to become 128 bit.

Note that the block size of 128 bit is independent of the key size. Meaning that even if a key size of 192 bits or 256 bits is used, the encryption/decryption would still be done 128 bits at a time.

The padding scheme needs to be such that upon decryption the padding portion can be removed and discarded to reveal the original plaintext.

Hence AES needs a padding scheme. One of the popular standardized padding scheme, which is also used in AES implementation is PKCS#7. Simply put the PKCS#7 algorithm states the value of each padding byte should be the length of the padding required. So in the case where you have a plaintext of 12 bytes, this will require 4 bytes of padding to make it 16 bytes (that is to make it 128 bit that is required by AES). Hence the value of the 4 bytes should be 4.

To prevent ambiguity in the case where the plaintext is already a multiple of 128 bits, PKCS#7 stipulates that we add a full block of padding set to the value 16. This ensures that, to remove the padding, one can always look at the last byte, and the value found there, is the number of bytes to discard in order to get to the main data.

Most implementations of AES do not make the padding scheme configurable to the developer, hence you won't need to fiddle around with the padding scheme most of the time. Irrespective it is good to know, to have a better understanding of AES.

The consequence of AES being a block cipher, that is being able to only encrypt 128 bits at a time leads to two things: We need to provide a padding mechanism, which was mentioned in the previous section.

We also need a way to put the pieces of encrypted 128 bits together to get the final ciphertext. The way by which the individual encrypted 128 bits are assembled together is called Mode of operation and there are various ways to go about this.

There are various modes that have been defined for block ciphers (i.e. not limited only to AES), but to have a better understanding of what they are actually supposed to help with, we look at two modes: ECB mode and CBC mode.

A naive way to piece back the encrypted blocks, is to assemble them back exactly the way the plaintext was broken up. This mode of encryption is called Electronic Code Book (ECB).

The ECB mode has the drawback that it leaks information about the plaintext.

If the plaintexts are broken apart, encrypted and assembled back the same way, patterns about the data can still be carried over into the encrypted version, because similar bits in the plaintext would encrypt to similar cipher text. This allows some information to still be learnt about the plaintext, which defeats the purpose of encryption. A good encryption scheme should not reveal any useful information about the encrypted message, none at all.

A very good example of illustrating this weakness of the ECB mode is the ECB penguin. An image of a penguin is encrypted using the ECB mode, but the shape of the penguin can still be seen in the encrypted image.

Hence for all intents and purposes, ECB mode is usually not used.

The Cipher Block Chaining mode is another mode of operation that can be used with AES that does not suffer from the weakness of ECB.

The problem with the ECB mode is that identical portions of the plaintext would encrypt to identical cipher text. This is what enables patterns to still be carried over to the ciphertext.

The CBC mode avoids this problem by including a procedure in its algorithm that first randomises the plaintext before encryption. This process of randomisation ensures that identical portions of the plaintext will never encrypt to the same ciphertext.

Simply put, CBC achieves this randomisation by taking a previous encrypted block and using this to randomise (by performing an XOR operation) the next block to be encrypted. This then provides the output that is then fed as the next block to the encryption. This chaining goes on until the plaintext is fully encrypted into the ciphertext.

You might ask, how will the first block be randomised before encryption since there is no previous cipher text that can be used for the XOR operation? The answer is that CBC used with AES requires an additional 16 bytes value called the Initialisation vector (IV) to kickstart the encryption process. This is the value that is used to randomise the first block to kickstart the encryption process. It should be noted that the IV also needs to be random and unpredictable in order to have a secured encryption scheme.

So far we have seen what encryption is, what it is not, the difference between symmetric encryption and asymmetric encryption and also the difference between block ciphers and stream ciphers. We also saw how block ciphers lead to needing padding and mode of operation. The weakness of ECB mode and how CBC avoids that.

Next we will see how encrypting data with CBC looks like in code. To do this, we would again be using Deno.

The following code shows how to encrypt and deprect a piece of text using AES with the CBC mode.

// We generate a key
// See https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/generateKey
let key = await crypto.subtle.generateKey({
   name: 'AES-CBC',
   length: 128
}, false, ['encrypt', 'decrypt']);

// We generate random 16 bytes to be used as initialization vector
let iv = new Uint8Array(16);
// and fill it with random numbers
// see https://developer.mozilla.org/en-US/docs/Web/API/Crypto/getRandomValues
await crypto.getRandomValues(iv);

// The web crypto requires data to be in ArrayBuffer
let plaintext = new TextEncoder().encode("hello world");

// The encryption and decryption requires specifying
// the algorithm and initialization vector
let param = {
   name: 'AES-CBC',
   iv: iv
};

// We encrypt the plaintext to get the ciphertext
let ciphertext = await crypto.subtle.encrypt(param, key, plaintext);

// We decode and console logged the ciphertext
console.log(new TextDecoder("utf-8").decode(ciphertext))

// We decrypt the ciphertext back to plaintext
let deCryptedPlaintext = await window.crypto.subtle.decrypt(param, key, ciphertext);

// We tranform the plaintext from bytes to string and then print it out
console.log(new TextDecoder("utf-8").decode(deCryptedPlaintext));

If you copy this code into a filename enc_dec_aec_cbc.ts and then execute deno run enc_dec_aec_cbc.ts, you should see two lines printed out. One looks like gibberish binary blob, while the second line should be "hello world"

As the comment in the code shows, we are generating a key, and IV, and then used to encrypt the text "hello world". Then print the cipher text and then decrypt it back to the plain text "hello world".

In Introduction to Cryptographic Hash Functions for the Working Developer, I presented a straight to the point, overview of some essential things a developer should know about cryptographic hash functions. 

This post continues in the theme around hash functions, by taking a look at another cryptographic construction hash functions make possible, that is: Message Authentication Codes (MACs).

It is worth quoting Bruce schneier again:

Much more than encryption algorithms, one-way hash functions are the workhouses of modern cryptography

Because Message Authentication Code based on hash functions is a perfect demonstration of how crucial hash functions are.

This post contains the following sections:

• Why Hash Functions alone are not enough
• What is Message Authentication Code
• What is a Hash Based Authentication Code (HMAC)
• What is a Keccak Based Authentication Code (KMAC)
• Some real world applications of Message Authentication Code

A message authentication code, is extra data or information that is used to confirm that a piece of data came from the stated sender (confirming authenticity) and has not been changed (providing integrity). This piece of extra data/information is often referred to as the authentication tag. 

In general authentication tag can be broadly classified into 4 categories: unconditionally secure, hash function-based, stream cipher-based and block cipher-based. In this post, I would only be touching on two of the hash function-based: HMAC (Keyed-hash message authentication code) and KMAC (KECCAK Message Authentication Code).

In summary, the hash function based MACs can be seen as a mix of a hash function and a secret. Let's look first at HMACs.

HMAC is a MAC created from using a cryptographic hash function and a secret cryptographic key. 

The cryptographic hash function can be any secure hash function, such as SHA-2 or SHA-3. This is reflected in the name of the given HMAC. For example, HMAC-SHA256 and HMAC-SHA3-512 is created using a secret key with SHA-2 (256) and SHA-3 (512) respectively.

As noted in Introduction to Cryptographic Hash Functions for the Working Developer naively using SHA-2 to hash concatenated secrets with a message is insecure due to the weakness of SHA-2 to the length extension attack. This is why HMAC is needed when the requirement is to use SHA-2 with a key in other to create an authenticated tag. 

Although it is beyond the scope of this post to dig into how HMAC works, it is worth pointing out that HMAC construction is not a naive concatenation of secret keys and data. HMAC uses a nested construction which avoids the length extension attack pitfall, Hence it is a special one that takes care of combining secret keys and data together in such a way to avoid being susceptible to the length extension attack.

SHA-3 on the other hand is not susceptible to the length extension attack and it is relatively safe to create an authentication tag manually using SHA-3-512(𝐾𝐸𝑌‖𝑚𝑒𝑠𝑠𝑎𝑔𝑒), but since the HMAC construction is available, it is also possible to use the construction with SHA-3 instead, hence HMAC-SHA3-512.

Using HMAC-SHA3-512 is not advised though, as it is not efficient. As stated in NIST SP 800-185

KMAC is a keyed hash function or pseudo-random function (PRF) that can be used, e.g., to compute a message authentication code (MAC) or to derive a session key from a master key. It is more efficient than HMAC by removing the need for HMAC's nested construction 

Hence when there is the need to use SHA-3 hash functions to create authentication tag, it is preferable to do so using KMAC, which we look at next.

The KECCAK Message Authentication Code (KMAC) algorithm is also a keyed hash function but based on KECCAK. 

It provides variable-length output, and unlike SHAKE and cSHAKE, altering the requested output length also generates a new, unrelated output. 

KMAC has two variants, KMAC128 and KMAC256, built from cSHAKE128 and cSHAKE256, respectively. 

Much more than encryption algorithms, one-way hash functions are the workhouses of modern cryptography - Bruce schneier

1. Cryptographic hash function: A definition
2. Properties of cryptographic hash functions.
3. Types of hash functions
1. Fixed length hash Functions
2. Extendable Output Functions (XOF)
3. Password hashing functions 
4. Some Hashing Hygiene

T206 Honus Wagner baseball card sold for $6.606 million just this August 2021, Pikachu Illustrator, a pokeman card sold for $195,000 in 2019. You can get different Micheal Jordan's basketball card on e-bay ranging from couple of thousands to couple of millions. Sentimental and collectible items have always been part of the human experience. NFTs are not to blame.

Why should pieces of cards with pictures of athletes (or game characters) be worth that much? Why should JPEGs/PNGs be worth that much? Maybe because humans also value things for sentimental reasons just as they do for utilitarian/sustenance reasons.

NFTs only made the act of ownership that was so easy to establish in the physical realm, also possible in the digital realm. It's only natural it will be used for digital arts/collectible culture.

The mistake to avoid, is thinking NFTs are only about digital arts and collectibles. No! they are more than this. The ability to create unique, non fungible digital items that can be provable owned, can be, and is being deployed for other use cases.

Non-equity funding campaigns are always based on promises: "Back us now, and we promise you early access to the product when we launch, or we promise you a discount, digital shout-out or swags". I am currently running an Indiegogo campaign where the promise will also include an NFT: Non Fungible Token, which is a digital artifact that can be used to prove things like ownership. This, as far as I know, is the first time NFTs would be used in this capacity.

In this post, I will shed some light on what NFTs are and share my thoughts on why they are a  perfect perk for a non-equity crowdfunding campaign. In doing so, I will touch on my thoughts on the continual march towards the digitalization of the human experience. How digitalization and technologies like virtual/augmented reality, blockchain technology, etc are making it possible to further digitize aspects of society and human interaction that were previously rife with problems or just plain impossible to digitize.

By the way, my campaign is to kickstart the development of DishAfrik: a recipe app to discover, enjoy and share African recipes. Think of it as Yummly or Tasty.co but focused on African cuisine. One of the perks of the campaign is going to be 54 uniquely generated collages made up of 54 recipes from 54 countries. 54 because currently Africa is home to 54 fully recognised sovereign states.

• Ever since the third industrial revolution, there has been a march towards digitalization of the physical world. 
• Blockchain-related technologies, like NFTs, provided the possibility to digitize aspects of the physical world that relied on scarcity, uniqueness, and proof of ownership, aspects that were previously hard to digitize. 
• DishAfrik's Indiegogo campaign will be making use of NFTs as a perk, using it as a way to capture participation, while also creating digital art in the form of a recipe collage that can also be owned.

This post will be the conlcuding post in the Introduction to Advanced Types in TypeScript series. It looks at some of the utility Types in TypeScript, explain how it works while pointing out the features within the TypeScript type system that makes it possible.

TypeScript comes with a powerful and expressive type system. Its expressiveness makes it a joy to work with while its powerful features makes it possible to build, and scale large codebases by providing type safety.

One of the ways TypeScript brings about type safety is by providing developers the tool to manipulate types in ways that encode constraints within the type system. This then ensures that code that could lead to runtime exception do not compile, making it possible to catch errors during development time and not in production. One of such tools TypeScript provides for this are Utility Types.

Utility Types are a set of Generic Types that come natively within TypeScript that makes it possible to transform one type into another. This sort of type transformation is useful because it makes it possible to take exiting types, and apply modifications to it which would ensure the enforcement of certain constraints.

In this post, we would look at 2 of such Utility type and how they could be used to provide more type safety. After that, we would take a step back to understand some of the other TypeScript features that come together to make Utility Types possible. Armed with this knowledge, we will then demystify Utility types by taking a peek under the hood of the 2 Utility types in focus to see how they are implemented.

The two utility types that will be exmaned are Required<T> and  Exclude<T>. This post assumes basic knowledge of TypeScript and what Generic Types are. Other features of TypeScript type systems would be explained.

We first look at how to make use Required<T> and  Exclude<T>. Starting with Required<T>.

Required<T> is an utility type that creates a new type based on an existing one by marking all the properties in the new type required.

To illustrates let’s imagine we have a Post type. A post can also have a score value which is calculated dynamically or might not even be calculated yet, hence the score property on the Post type should be optional. The definition of such a post type could look like this:

type Post = {
 title: string
 publishedDate: number
 score?: number
}

When the score on a post has been calculated we want this to be obvious and explicit. Having score as optional does not give use explicitness. One way to go about this, is to create another type with score not optional. For example:

type ScoredPost = {
 title: string
 publishedDate: number
 score: number
}

And we can then have a function that scores a post by taking a value of Post and converting it to a value of ScoredPost

function scorePost(post: Post): ScoredPost {
   return {
      title: post.title,
      publishedDate: post.publishedDate,
      score: post.publishedDate * Math.random()
     }
}

But having both Post and ScoredPost is a bit verbose as it is clearly obvious that ScoredPost is just a Post with the score property now required.

Having Post and ScoredPost is not only verbose it also adds to maintenance burden as any update to either of the types need to be manually reflected on the other.

Since ScoredPost is clearly a transformation of Post with score now required, we can easily apply the Required<T> utility type here. To do that, the scorePost function would be updated as follows:

function scorePost(post: Post): Required<Post> {
    return {
      title: post.title,
      publishedDate: post.publishedDate,
      score: post.publishedDate * Math.random()
     }
}

This shows the power of a utility type such as Required<T>: easily create a new type from an existing type while enforcing the required property constraint.

Exclude<T, U> is also another utility type that creates a new type based on an existing one. It does this by excluding certain properties from one type to create a new type.

To illustrates let’s imagine we have a AdminRights type and a UserRights type. The UserRights could either be read or write, while AdminRights is all the permissions presents in UserRights with the execute permission added. These two types could be defined as follows:

type  AdminRights = "read" | "write" | "execute"
type  UserRights = "read" | "write"

But having both AdminRights and UserRights can be seen as being verbose since UserRights is AdminRights without the execute option.

It also does not make it explicit that UserRights is all the AdminRights with the execute excluded. To make this clearer we can use the Exclude<T, U> utility type.

The definition would then look as follows:

type AdminRights = "read" | "write" | "execute" 
type UserRights = Exclude<AdminRights, "execute">

Here, we are creating a new type UserRights, from AdminRights by excluding the "execute" property. Making it explicit how UserRights is related to AdminRights.

Now we have seen two examples of Utility types in TypeScript. We would now take a quick look at some of the features of TypeScript that makes these utility types possible.

The KeyOf type operator is an operator that can be used to get all of the properties of a type as a union type. For example, given the type Post:

type Post = {
 title: string
 publishedDate: number
 score: number
}

All the properties of Post, that is title, publishedDate and score can be retrieved as union type using the keyOf operator:

type Props = keyof Post
// Props => "title" | "publishedDate" | "score"

Check Introduction to Index Types in TypeScript for a more detailed introduction to the keyof operator.

Conditional types can be thought of as a mechanism by which we can perform ternary operations on types. For conditional types, the operation that determines which type is returned is an assignability check, unlike a boolean check as is the case with normal ternary operation that works on values. A conditional types look as follows:

type  Post = {
   title: string
   publishedDate: number
   score: number
}
// the conditional type
type  RatedPost<T> = "score"  extends  keyof  T ? T : never

In the above code snippet, RatedPost\<T> is defined as a conditional type, which makes use of generics. It returns the type T as long as it has the score property, if not it returns never. never is a type that has no value ever, hence if we have a compile error if the conditional type ever evaluates in such a way that the never is returned.

See Conditional Types in Typescript for a more indepth exploration of Conditional Types. See any, unknown and never types in Typescript for a more detailed introduction of the never type

To see this in use, we can define a function: resetScore that only takes a post that already has a score. Such definition would look like this:

function resetScore(post: RatedPost<Post>) {
  // do stuff
}

This can be called as follows:

resetScore({
 title: "hello post", 
 publishedDate: 1621034427002, 
 score: 2
})

but passing an object that has no score property would lead to a compile error:

resetScore({
  title: "hello post", 
  publishedDate: 1621034427002
})

A mapped type is a type that is created by using the keyof operator to access the properties of another type and modifying the extracted properties in some way to create a different type. For example, a Mapped type that takes a Post type and convert all its properties to string would be defined as follows:

// initial type
type  Post = {
  title: string
  publishedDate: number
  score: number
}

// mapped type
type  StringPost<T> = {
   [K  in  keyof  T]: string
}

// usage of mapped type
let stringPost: StringPost<Post> = {
   title: "How to do the wiggle",
   publishedDate: "15/05/2021",
   score: "2"
}

Taking a closer look at the mapped type definition:

type  StringPost<T> = {
   [K  in  keyof  T]: string
}

We can break down the code listed above as :

• keyof T gets all properties of T
• [K in keyof T] loop through all the properties of T
• and finally [K in keyof T]: string, loop through all the properties of T and assign string as their type

See Mapped Types in Typescript for a more indepth exploration of Mapped Types.

Now that we are armed with the knowledge of these TypeScript features, let's now see how they are applied to create Utility types.

Utility types are created from the application of some or all of the TypeScript features highlighted in previous sections. We will now take a look at the source code of the two utility types described above: Required<T> and Exclude<T, U> and see how they are defined.

The source code can be found in es5.d.ts

The Required<T> is defined as follows:

/**
* Make all properties in T required
*/
type Required<T> = {
    [P in keyof T]-?: T[P];
};

source

Let us break it apart:

• This is a Mapped Type
• keyof T gets all properties of T
• [P in keyof T] loop through all the properties of T
• the -? in [P in keyof T]-? means remove the character ? from the definition of the properties. This is how the optionality of the properties are removed and made required
• The T[P] in the type definition means, take as type whatever property P is as part of type T. This way the original type of P is used in the mapped type.
• Finally [P in keyof T]-?: T[P]; can be read as loop through all the properties of T remove the optional modifier ?, and keep original type.

That is it. Note that in the case where modifiers are to be added, the - can be dropped which defaults to adding, or +, can be used.

For example, instead of making all properties required, if we want to make them option, we would have the definition as [P in keyof T]+?: T[P]; or [P in keyof T]?: T[P];. In fact there is utility type for this and it is called Partial<T>.

The Exclude<T, U> is defined as follows:

/**
* Exclude from T those types that are assignable to U
*/
type Exclude<T, U> = T extends U ? never : T;

source

Let us break it apart:

• This is a conditional type
• Exclude<T, U> defines a generic with two variables: T and U
• T extends U is the conditional check of the conditional type.
• If T extends U then never is returned
• If T extends U is false, then T is returned.

The T extends U is checking if T is a subtype of U. That is, if all properties of U can be found in T. If this not the case, then T is returned which would then be all the properties ofU with the properties found in T excluded.

This is how the exclusion in Exclude<T, U> is implemented.

As can be seen, even though utility types are powerful, there is nothing mysterious about them and they are understandable. They are created by using other features of TypeScript.

TypeScript comes with more utility types defined which can be found here, and hopefully this post now helps in appreciating what they are and how they are implemented to do what they do.

PlanetTypeScript.com is a content aggregator for all things TypeScript. It regularly polls RSS feeds for TypeScript related content, which it then list on the site, send out as a tweet via @planetypescript, and also as a weekly digest to subscribers.

Back when I used to write a lot of Java-related content on this blog, I found out that my blog ended up on www.topjavablogs.com/, which is a site that aggregates Java contents. So when I decided to play around with Deno, I thought a nice little idea would be to build something similar to topjavablogs.com, but for TypeScript. And that is how PlanetTypescript.com was born.

In this post, I will share some of my first impressions with Deno. The things I enjoyed and the bits I found inconvenient. Let's start with the cool stuff.

I find it super convenient that I could just write TypeScript without worrying about setting up a build chain that would take care of the transpilation to JavaScript. This is really nice and it makes TypeScript feel more like a standalone language in its own rights. This has to be one of the top-selling points of Deno for me.

The deno CLI comes with a whole lot of useful utilities and just comes in handy. For example deno info to check the dependencies. deno fmt, a code formatter,  deno bundle for bundling modules and dependencies into a single file, deno compile for creating a self-contained executable, etc.

I personally find dealing with build pipelines within the JavaScript ecosystem to be a tiring chore. Having something like deno bundle or deno compile, built-in is super useful, as it takes care of most of the heavy lifting when it comes to deployment.

Deno seeks to provide implementations of Web API, and this I found to be really handy. For PlanetTypeScript.com, I need to make HTTP requests and also run background tasks. I could use fetch and web workers for this. API's I am already familiar with coming from the Web, and no need to install third-party libraries. This is super convenient.

In the course of building PlanetTypeScript.com, I needed to have logging, base64 encoding, use hash functions, and generating UUIDs. I was able to find modules within the Deno standard library that I could use for all of these. The Deno standard library is quite decent and has a battery-included feel to it. You can peruse the available modules here.

I primarily use Jetbrains IDEs. When not within a full-blown IDE environment, I use Microsoft Visual Studio Code. Both of these tools have plugins that implement support for Deno and they work very well. I was able to use the visual debugger in IntelliJ, which was quite handy.

So these are some of the cool things that made a good first impression on me with Deno. Below are some of the things that were a bit of a hassle.

Deno does not have any well-known central registry from which a package manager manages dependencies. Instead, Deno works with plain URLs. This setup leads to certain consequences which I believe hamper the developer experience. 

For example, there is no central file to look into, to see all the dependencies of a project. To work around this, the convention in the Deno community is to re-import all dependencies from a file called deps.ts - basically recreating some sort of package.json.

Also, there is no easy way to quickly see if there are new versions of any of my project's dependencies. Or even better still, no command I could run from the command line that updates dependencies. 

I later learned about Trex and dmm which could be used to achieve some of these tasks. I can't comment on them now as I have not gotten around to try them out yet.

From the perspective of the developer, I am yet to see the advantage the Deno model brings to the table. Instead, it feels to me more like a philosophical design decision, rather than a pragmatic one.

The inability to find needed third-party libraries proved to be one of the main inconveniences.

I had issues finding a third-party library to deal with OAuth. I tried Dashport, but it has become incompatible with recent versions of Deno and Oak (the HTTP middleware library I used). I tried using Firebase-auth, but there is no support for Deno.

I use MailChimp for sending the Weekly digests. Since the content of the emails would be different weekly, I needed to build the message dynamically. This involves interacting with MailChimp via its API. Unfortunately, MailChimp does not have an API client library for Deno, so I ended up having to directly call the needed HTTP endpoints.

Deno does have a compatibility layer that makes it possible to use Node/npm modules from within Deno. There are also CDN's like Skypack that make it possible to import npm modules for use within Deno. What I quickly found out, is that these workarounds, work best for the simplest of libraries. It does not work for anything slightly complicated. For example, I could not use any of these methods to get the Node library for Firebase-auth nor Mailchimp client library to work.

This nascency is strictly not a structural issue with Deno and something to be expected, given how new Deno is. This is also the sort of thing that would improve with time.

I enjoyed using Deno. Having native TypeScript support, various CLI utilities, and a battery included-standard library, makes it easy to get productive real quick. The main inconveniences I encountered can be chucked up to Deno still being relatively new, but even with that, it is still a hell of a lot usable/productive for a 3 years old project.

TypeScript type's system is structural and hence, it allows for the easy creation of types based on the shape of data. These types can be created and used without having a name. For example:

let john: {name: string, age: number} = { 
  name: "John Doe", 
  age: 30
}

Where {name: string, age: number} is the type defined.

But there is a problem with using the type definition this way. If we want to create another person and annotate another variable with the type, we would need to repeat ourselves:

let john: {name: string, age: number} = {
  name: "John Doe", 
  age: 30
}

let jane: {name: string, age: number} = {
   name: "Jane Doe", 
   age: 45
}

This is not dry.

It is not dry because we have to repeat the types. A more efficient approach would be to define the type once, give it a name and then use the name when applying the type annotation. Fortunately, we can do exactly this.

TypeScript provides mechanisms for giving a name to a type, which can then be used when applying type annotations. We are going to look into the 3 different mechanisms through which this can be done. Namely: via classes, type aliases and interfaces.

A class can be seen as a blueprint for the creation of objects. For example given the following class definition:

class Person {
    name: string;
    age: number;

    constructor(name: string, age: number) {
        this.name = name;
        this.age = age;
      }

}

We can use it to create various instance objects:

// class definition
class Person {
    name: string;
    age: number;

    constructor(name: string, age: number) {
        this.name = name;
        this.age = age;
      }

}

// creating instances
let john = new Person("John Doe", 30)
let jane = new Person("Jane Doe", 45)

The interesting thing with classes is they are also a mechanism for defining and naming types. They not only serve as a blueprint for object creation, but they also create a type, which is named.

The Person class we defined also simultaneously creates a Person type, which we can use as a type annotation:

// class definition
class Person {
    name: string;
    age: number;

    constructor(name: string, age: number) {
        this.name = name;
        this.age = age;
      }

}

// annotating instances
let john: Person = new Person("John Doe", 30)
let jane: Person = new Person("Jane Doe", 45)

Type aliases are another mechanism for naming types in TypeScript. It is a feature that makes it possible to give a name (or alias) to another type.

type Person = {name: string, age: number}
type MyString = string
// An alias can be given for an existing alias
type OhMyString = MyString

Hence it is best to see type aliases as a way to give a name to an already existing anonymous type, primitive type, or even another type alias.

Interfaces are another mechanism for creating types and naming them in TypeScript. It takes the following form:

interface Person {
    name: string,
    age: number
}

As can be seen, even though interface looks similar to using type aliases, they are not the same. Unlike type aliases, interfaces actually create a new type, and gives it a name, while type aliases only create a name for an existing type.

The 3 mechanisms shown thus far (classes, type aliases, and interfaces) provide us with 3 ways for doing the same thing: the naming of types. Even though they allow us to achieve similar things, there are some differences. In the next section, we take a look at some of these differences.

Classes differ from type aliases and interfaces in that it also provides a constructor that can be used to create well form instances of values for the type being defined. This is achieved by using the familiar new keyword:

// class definition
class Person {
    name: string;
    age: number;

    constructor(name: string, age: number) {
        this.name = name;
        this.age = age;
      }

}

// creating instances using constructor
let john: Person = new Person("John Doe", 30)

Constructors are not created as part of using type aliases nor interfaces.

Classes, type aliases, and interfaces all provide a mechanism for type extension, which is the ability to create a new type based on an already existing type. The only difference is the syntax used.

Extending types via classes takes the following form:

//type definition 
class Person {
    name: string;
    age: number;

    constructor(name: string, age: number) {
        this.name = name;
        this.age = age;
    }
      
}

// type extension
class Employee extends Person {
       isManager: boolean;
       
       constructor(name: string, age: number, isManager: boolean) {
           super(name, age)
           this.isManager = isManager
       }

}

doing the same via type aliases look like the following:

// type definition
type Person = {
    name: string
    age: number
}

// type extension
type Employee = Person & {isManager: boolean}

and finally with interface we have:

interface Person {
    name: string
    age: number
}

interface Employee extends Person {
    isManager: boolean
}

The syntax for extension with classes and interfaces looks similar while it is different for type aliases. Extending type aliases makes use of intersection types, which is a topic that is explored in details in Union and Intersection Types in TypeScript

Interfaces differ from the other mechanisms in that, the type being named can be spread across multiple definitions.

//first definition 
interface Employee {
    name: string
}

// second definition
interface Employee {
    age: number
}

// third definition
interface Employee {
    isManager: boolean
}

// Merged Employee type being used
let john: Employee = {
    name: "John Doe",
    age: 30,
    isManager: true
}

TypeScript will take all the definitions of the interface Employee, and merges them into one. This is not possible with type aliases nor interfaces as their definition needs to be unique.

Most popular mainstream languages have a sleep functionality that allows for the pausing of code execution for a specified amount of time. Not surprising this functionality is often implemented via a function usually named sleep. For example:

// ruby
sleep(time_secs)
// python
import time
time.sleep(time_secs)
// Java
Thread.sleep(time_milli_secs)
// Perl
sleep(time_secs)

JavaScript does not have a similar sleep function, although it has the setTimeout function which can be used to achieve a similar purpose. The only problem is, using setTimeout is not as convenient as one would like. This is due to the fact that setTimeout is an asynchronous function, hence one would have to continue the execution of the code, in the callback passed to it, after the elapsed time.

console.log("Executing code..."); 
setTimeout(() => { 
console.log("continue after pause"); 
}, 3000)

This is not convenient.

A more convenient option would be to have something similar to:

console.log("Executing code..."); 
// sleep for some time
// sleep()
console.log("continue after pause");

Unfortunately, this does not exist natively in JavaScript. But this is not to say we can't implement one!

We can implement such a sleep function by combining the capabilities of Promises and Async/Await in JavaScript. For a quick recap of the syntax involved, see Callback, Promise and Async/Await by Example in JavaScript.

The implementation of such a sleep function, involves "Promisifying" the setTimeout function, and then using the Async/Await syntax to provide a synchronous feel to a rather asynchronous function. It looks this:

function sleep(milliseconds) {
    return new Promise(resolve => {
        setTimeout(() => {
            resolve({})
        }, milliseconds);
    })
}

(milliseconds: number) => new Promise(resolve => setTimeout(resolve, milliseconds))

Usage would have to be within a function marked as async:

// definition
function sleep(milliseconds) {
    return new Promise(resolve => {
        setTimeout(() => {
            resolve({})
        }, milliseconds);
    })
}

//usage
async function doStuff() {
    console.log("start")
    await sleep(3000)
    console.log("continue")
}

doStuff()

The requirement to put the code in an async function is because, as of now, browsers and NodeJS do not support top-level async functions. Attempting to have the code used in the browser, outside of an async function, would lead to an error similar to this:

Uncaught SyntaxError: await is only valid in async functions and the top level bodies of modules

Attempting to run it in Node, would also lead to an error:

SyntaxError: await is only valid in async functions and the top level bodies of modules

Deno on the other hand, already supports top-level async functions, hence with Deno, the code can be used right away:

// definition
function sleep(milliseconds) {
    return new Promise(resolve => {
        setTimeout(() => {
            resolve({})
        }, milliseconds);
    })
}

//usage
console.log("start")
await sleep(3000)
console.log("continue")

Assuming the above code is in a file named script.js it can be executed using Deno as follows:

$> deno run script.js 
start
// pause for 3 seconds
continue

This post is going to show,  by way of code examples, how to take a callback based API, modify it to use Promises and then use the Async/Await syntax. This post won't go into a detailed explanation of callbacks, promises or the Async/Await syntax. For such a detailed explanation of these concepts please check Asynchronous JavaScript, which is a section of MDN Web Docs, that explains asynchronicity and how callbacks, promises, and the Async/Await syntax helps with working with asynchronous JavaScript.

This post is meant for the developer who has somewhat of an understanding of asynchronicity in JavaScript, but requires a straight to the point code example to serve as a quick syntax reference for how to take a callback based API, update it to use promises and finally use the Async/Await with it.

For demonstration purposes, we are going to use the fs.readFile, which is a callback based API from reading files. We will have a file test.txt that would contain some text, we will then have a file script.js that would open the file, read the contents, and print it to the terminal.

The code will first be implemented using callbacks, it would then be updated to use Promises, and finally, instead of using Promise directly, it will be updated to use Async/Await.

Let's get started.

We first create a directory where we are going to work from, create also the file that will contain our code, and the two files that we would be reading from.

We first create the two files with contents.

$ mkdir ~/code
$ touch ~/code/script.js
$ echo "Beam me up, Scotty" > ~/code/test.txt
$ cd ~/code/

Next in the script.js file, we have the following code:

const fs = require("fs")

function readFileCallBack() {

fs.readFile("./test.txt", 'utf8',  (err, data) => {
  if (err) {
    console.error(err)
    return
  }
  
  console.log(data.trim() + " [callback]")
 })

}

readFileCallBack()

Executing the script by running node script.js should get "Beam me up, Scotty" printed to the terminal:

$ node script.js
Beam me up, Scotty [callback]

With the callback style, the result of the asynchronous operation is handled by a function passed in to the function performing the asynchronous operation.

Update script.js and add a version of readFileCallback that uses promises. It looks like this:

function readFilePromise() {
  return new Promise((resolve, reject) => {
     fs.readFile("./test.txt", 'utf8',  (err, data) => {
     if (err) {
       reject(err)
       return
     }

      resolve(data.trim())
    })
  });
}


readFilePromise()
 .then(data => console.log(data  + " [promise]"))
 .catch(err => console.log(err))

Execute the script by running node script.js:

$ node script.js
Beam me up, Scotty [callback]
Beam me up, Scotty [promise]

With promises, the result of the asynchronous operation is handled by a function passed to the then function exposed by the promise object.

Update script.js and add a third version that uses the Async/Await syntax. Since Async/Await is a syntax that makes using promises easier, the Async/Await implementation would make use of the readFilePromise() function. It looks like this:

async function readFileAsync() {
  try {
    const data = await readFilePromise()
    console.log(data.trim() + " [async-await]")
  } catch (err) {
    console.log(err)
  }
}

readFileAsync()

Executing the script by running node script.js will print something similar to this, to the terminal:

Beam me up, Scotty [callback]
Beam me up, Scotty [promise]
Beam me up, Scotty [async-await]

With async/await, the result of the asynchronous operation is handled as if it was a synchronous operation. The await is responsible for this, while the function in which it is used has to be preceeded with async keyword.

The complete file with the 3 implementations is presented below:

const fs = require("fs")

// callback
function readFileCallBack() {

fs.readFile("./test.txt", 'utf8',  (err, data) => {
  if (err) {
    console.error(err)
    return
  }
  console.log(data.trim() + " [callback]")
  
 })

}

readFileCallBack()

// promise
function readFilePromise() {
  return new Promise((resolve, reject) => {
     fs.readFile("./test.txt", 'utf8',  (err, data) => {
     if (err) {
       reject(err)
       return
     }

      resolve(data.trim())
    })
  });
}


readFilePromise()
 .then(data => console.log(data  + " [promise]"))
 .catch(err => console.log(err))


// async/await
async function readFileAsync() {
  try {
    const data = await readFilePromise()
    console.log(data.trim() + " [async-await]")
  } catch (err) {
    console.log(err)
  }
}

readFileAsync()

To illustrate that the error handling in the 3 implementation work as expected, rename the test.txt file and rerun the script:

$ mv test.txt test.txt.backup
$ node script.js
[Error: ENOENT: no such file or directory, open './test.txt'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: './test.txt'
}
[Error: ENOENT: no such file or directory, open './test.txt'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: './test.txt'
}
[Error: ENOENT: no such file or directory, open './test.txt'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: './test.txt'
}

Showing that the error handling code, which is to just print the error to the console, works as expected in the 3 implementations.

ip-num is A TypeScript/JavaScript library for working with ASN, IPv4, and IPv6 numbers. It provides representations of these internet protocol numbers with the ability to perform various IP related operations like parsing, validating etc, on them.

A new version of ip-num, version 1.3.2 is now available.

This release contains one new feature and a bug fix.

In previous versions of ip-num, it was possible to take IP ranges with prefix less than /32 and split them into two. For example:

import {IPv4CidrRange} from "ip-num/IPRange";
import {IPv4Prefix} from "ip-num/Prefix";

let ipv4CidrRange = IPv4CidrRange.fromCidr("192.168.208.0/24");
let splitRanges: Array<IPv4CidrRange> = ipv4CidrRange.split();

// console logs:
// 192.168.208.0/25
// 192.168.208.128/25

splitRanges.forEach(range => console.log(range.toCidrString()))

But what if one needs to take an IP range and instead of just splitting into two, there is the requirement to split into a number of ranges with a specified prefix? Well, that is now possible with the addition of splitInto method:

import {IPv4CidrRange} from "ip-num/IPRange";
import {IPv4Prefix} from "ip-num/Prefix";

let ipv4CidrRange = IPv4CidrRange.fromCidr("192.168.208.0/24");
let splitRanges: Array<IPv4CidrRange> = 
ipv4CidrRange.splitInto(IPv4Prefix.fromNumber(26));

// console logs:
// 192.168.208.0/26
// 192.168.208.64/26
// 192.168.208.128/26
// 192.168.208.192/26

splitRanges.forEach(range => console.log(range.toCidrString()))

Previous versions of ip-num failed to properly report that a string representing a single IP can be represented in the CIDR notation. This has now been fixed:

import {RangedSet} from "ip-num/IPRange";

let rangedSet= RangedSet.fromRangeString("1.2.3.4-1.2.3.4");

// now returns true
console.log(rangedSet.isCidrAble())

As always, ip-num is just an npm install or npm upgrade away.

Feel free to open an issue to discuss a feature or to report a bug.

type Employee = {
name: string
employedOn: Date
age: string
role: {
  title: string
  salary: number
  isManagement: boolean
 }
}

type Properties = keyof Employee

var value: Properties = "name" // compiles
var value: Properties = "employedOn" // compiles
var value: Properties = "age" // compiles
var value: Properties = "role" // compiles
var value: Properties = "gender" // does not compiles

type Person = {
    age: string | number
    height: string | number
}

type ToStringProps<T> = {
    [P in keyof T]: string
}

type StringPerson = ToStringProps<Person>

var person: StringPerson = {
    age: "10",
    height: "6"
}

type Immutable<T> = {
    +readonly [P in keyof T]: T[P]
}

type Mmutable<T> = {
    -readonly [P in keyof T]: T[P]
}

type Immutable<T> = {
    readonly [P in keyof T]: T[P]
}

type Optional<T> = {
    [P in keyof T]+? : T[P]
}

type NonOptional<T> = {
    [P in keyof T]-? : T[P]
}

type Optional<T> = {
   [P in keyof T]? : T[P]
}

type UnderlineProp<T> = {
    [P in keyof T as `_${string & P}`]: T[P]
}

type Person = {
    name: string
    age: number
}

type _Person = UnderlineProp<Person>

var _person: _Person = {
    _name: "John",
    _age: 20
}