https://blogger.com/feeds/5936111837781935054/posts/default

Here is a short blog entry just to explain the newly added multiple monitor support to View engine.

Screen faces

Each connected monitor gets associated with a screen face in system/view/screens list. You can check that all monitors have been detected correctly using (two monitors in this case):

    >> length? system/view/screens
    == 2

Each screen's /offset indicates the screen position using virtual screen coordinates. The main screen gets a (0, 0) offset. All other screens are positioned relatively to that, in the same virtual space. Screen sizes are also expressed in virtual coordinates. Screen's scaling factor is exposed in the /data facet as a float value.

Monitors properties changes and adding/removing detection are also supported, updating the screens list. Windows present on removed monitors are automatically closed by the OS.

A simple display-geometry.red script is provided to show how screen faces reflect the monitors topology in that virtual space:

That script will also output the screen details in the console:

    >> do %tests/displays-geometry.red
    1 - offset: (0, 0) size: (3840, 2160) scaling: 150%
    2 - offset: (1165, 2160) size: (1480, 320) scaling: 100%

NB: I am using a little 11.9inch screen below my main display for developing this specific feature, as it takes very little extra space on my desk.

Red apps on multiple screens

You can now launch Red consoles and apps on any display, the app will open on the screen it was launched from, using the specific scaling value of that screen.

A get-current-screen function has been provided to return the current screen (where the mouse cursor is currently located).

When displays have different scaling factor or DPI, View will adjust the window and its content to fit that scaling factor. This also works when dragging a View window between displays with different DPI, the window and its content will resize accordingly.

On Windows, the API View requires, are not supported on pre-Windows8 platforms. So a new  Windows7 compilation target has been added to fallback on an older API that is less accurate and may produce sizing issues.

Known limitations

- Screen offsets are using virtual coordinates, but window offsets are in scaled coordinates (using scaling factor). This is a current implementation constraint and a semantics problem, as we express windows content in a DPI-independent coordinate system, which also affects windows geometry.

- Opening a child window from one screen to another one is currently not supported. It could be achieved manually by forcing a window offset to another virtual space, though the scaled coordinates would make the target offset calculation more complex and will create sizing issues (unless you write a routine that will recalculate all face sizes on the fly).

This new milestone brings many low-level improvements to Red's memory management and garbage collecting. Most of those are long-planned additions needed to complete the internal memory model and make it robust enough for the future stable Red v1.0.

First, here is a simplified overview of the Red memory model (existing parts in green color, new parts in orange, non-Red parts in blue):

All Red values are stored in series. Some Red values require one or more buffers to hold their content. The values can never reference a buffer directly, but only through a node reference, to enable relocation when expanding the series buffer or when moving it around during compaction by the GC.

Now let's dive into the hairy details!

External resources GC

The Red/View engine backends rely on external resources provided by the OS. Among those resources, some are linked to face! or font! object and require special care when those objects are not reachable anymore. So far, our GC (Garbage Collector) was not able to release such resources (images bitmap buffers and fonts handles), as unreachable Red aggregate values are seeing as simple series during the sweeping GC stage. In order to improve that, we have added an external resources manager, that will track and free unused resources, allowing now unrestricted images and fonts usage!

Accurate GC

The Red GC relies on allocated memory walking and native stack scanning to identify live Red values. Scanning the native stack can be challenging. The scanner used so far a conservative approach, which is simpler, but can lead to corruptions or crashes in rare cases (e.g. a floating point number being mistaken for a series or node pointer). Moreover, such approach precluded from having a nodes frame GC, as there was no way to accurately identify node pointers on the stack. This is now solved. The plan was always to make it precise when getting closer to a Red v1.0 and that's what we did in this release.

In order to achieve that, several key additions were made:

➤ Frame records hints: the R/S compiler now generates a map of hints for arguments and local variables which are series/nodes pointers, using bit arrays stored in the .data segment and retrieved during scanning. In order to match a call frame on stack to the right bitmap array, an offset is now pushed on stack by each function call as part of their prolog sequence. Only the stack slots corresponding to 1's in the bit array are analyzed further to identify their origin series/node frame, then marked and stored in an list for the collector to later update them if needed. The bit arrays are compressed using our CRUSH algorithm implementation, so that, e.g. for the GUI console executable, all bit arrays add only about 3472 extra bytes to the final executable.

➤ Variadic hints (typed vs untyped): for variadic functions, the bit array is dynamically created. If the typed mode is used, an accurate bitmap is produced. If the generic untyped variadic mode is used, all the arguments stack slots will be marked for processing. This could, in theory, create false positives, but in practice, in Red's runtime code, all such cases are safe, referencing only Red values.

➤ Optimized pointers identification performance: each extracted pointer from the stack needs to be confirmed to be a valid series or a node pointer. Such checking is now achieved using cached sorted lists and binary search, ensuring vastly faster operation.

➤ Optimized frame walking by skipping non-Red frames: the stack scanning is done by jumping between call frames, relying on the saved frame pointer in each frame to chain the frames. However, when R/S callback functions are invoked by external (mostly OS) code, those external frames should be skipped to avoid false positives and for sake of performances. Now the scanner identifies which call frames are part of Red's code segment and skips the rest. However, one last hurdle remains, the dreadful compilation option in C compilers where the frame pointer is omitted in call frames (e.g. -fomit-frame-pointer in gcc). In such cases, walking the stack by dereferencing frame pointers is not an option anymore. The workaround is to save an extra "last known Red frame" pointer before calling any external code, which is then used by the scanner to jump over external code directly into the parent Red frame.

Node frames compaction

The GC is now capable of reclaiming node frames where the number of used slots is very low. It was, until now, a cause of memory leaking for long-running apps with bursts of high number of series allocations, as new node frames were allocated, but unused (non-empty) ones were never released.

This is now taken care of through a special GC pass that runs when specific conditions are fulfilled, moving live nodes from emptier frames to fuller frames, then freeing the entirely empty frames. The GC is then updating all references to relocated nodes during its marking and stack scanning stages.

In addition to that, the internal structure of node frames was improved. The free slots tracking method was changed from a stack-oriented model, to free slots linked-lists, resulting in doubling the node frames capacity, while keeping constant-time allocation/freeing performance.

External Red values reference management

Red values can sometimes be referenced by external non-Red code. The View engine relies on that and was storing copies of face object values inside external OS structures in order to be able to retrieve them on OS-generated events that would trigger Red callbacks. Such practice is not reliable and not compatible with the new node GC, as some node pointers could be stored away from the GC reach. So a new external values management system was introduced to only export a reference (an array index used as ID) to external code and keep all values inside Red series. The View backends were modified accordingly to rely on those references instead of copying the face object values.

That sub-system could in the future also be used for libRed external values management, to replace the ring buffer used there, which is functionally almost identical but now redundant.

Low-level allocations tracking

The Red runtime code sometimes has a need for allocating memory regions which last until the end of the Red process or need to be kept away for the GC. For that purpose, Red relies on malloc for such use-cases, just importing it from C library. Instead of a direct mapping, it now uses a thin layer on top of it in order to track all allocations providing extra features:

➤ Freeing of all system allocated memory regions on Red exit. This is not strickly needed for Red runtime, but allows to track possible leaks (rare case as most of such allocations are permanent).

➤ Ability to gather stats about such allocations (reported in show/info in "allocated on heap" part).

➤ Buffer overflow detection in debug mode using guard barriers at the tail of allocated buffers and checked on freeing for possible overflows.

➤ This layer is part of the R/S runtime, so available to R/S code too.

Other Changes

➤ stats native improvements: /info has been extended to contain also total allocated from OS and allocated memory from heap (see above). /show refinement has been implemented to pretty-print all those infos.

➤ Lowered memory allocations in Red runtime at start (about 1MB gained in total).

➤ Memory frames integrity auto-testing in debug mode (only node frames for now).

➤ Handle! values now hold a sub-type, revealed by mold/all (for debugging purposes):

    view [b: button "Hi!" [print mold/all b/state/1]]

    #[handle! 030A063Eh window] 

➤ Now the final buffer is preallocated internally for insert and append calls with /dup refinement, resulting in much lower memory usage.

➤ Using zero?, with a point3D value was always returning false due to an incomplete copy/paste change. Fixed now.

➤ Out-of-range integer math operations now promote results to float values.

➤ Updated GPIO definitions for RaspberryPi devices. Pi 4 should be supported, but untested yet. Pi 5 not yet supported (should be updated soon).

➤ Camera widget: now viewport aspect ratio is honored regardless of the container face size. Viewport is now centered and black bars are added if needed to fill the container face. A %camera-resize.red script is provided for testing.
➤ Toolchain: --view <engine> compilation option added to force a different View engine than the default one for the target.
➤ 47 tickets closed with a fix since 0.6.5.
➤ Stable releases are back!

Red/System changes

➤ Added system/lib-image to support libRedRT image properties.

➤ Improved native stack trace reports (frame address, stack records chaining support).
➤ On IA-32 backend, the passive casting mode (no conversion, all bits preserved) between integer! and float32! (as [integer!|float32!]keep)now returns the correct results.

What's next?
Next release (should be 0.7.0) will feature the full async IO support we are all waiting for! There are also other major features we're working on which probably will be released after 0.7.0.
In the meantime, enjoy this release!

Last year, qtxie worked on a toy text backend project and submitted a PR for that. After some extra additions and testing recently, it has now been merged even if it is still incomplete, it is usable enough. So, in addition to the shiny GUI backends in Red/View, now we have an old-school text-based user interfaces (TUI) backend for the View engine!

The new TUI backend has currently a subset of the GUI backends features. Here is an overview:

• View styles: base, panel, button, check, radio, field, text, progress, rich-text, image and text-list.
• Draw commands: text, line, box, triangle, circle, ellipse (block-based for now).
• Rich-text supported in Draw.
• Keyboard handling: key-down and key events (which are the same event).
• Mouse handling: disabled by default. Use system/view/platform/mouse-event?: yes to enable it.
• Images support Truecolor (24-bit RGB) for image rendering if the terminal supports it, otherwise it falls back to 256 colors.
• Timers supported through /rate facet.
• Facets supported: /offset, /size, /text, /image, /color, /data, /enabled?, /visible?, /selected, /flags, /options, /pane, /rate, /para and /draw.
• Flags supported: password and all-over.
• Frames drawing using squared or rounded corners (
• Limited ANSI escape codes support in /text facet, only Colors / Graphics Mode codes.
• Uses 256 colors for text. It should works fine on most of the terminals.
• Works on the big-3 platforms (Linux, macOS and Windows10/11).

The pre-built CLI console binaries on our Download page now have View/VID included by default along with the TUI backend. You can use them to test and play with the TUI code examples here and in the TUI folder.

To use the TUI backend in your own compiled code, you need to add the two following options in the Red header block:

    Needs:  'View
    Config: [GUI-engine: 'terminal]

Here are a few examples, starting with a HelloWorld!:

    view [text "Hello TUI World!"]

    Hello TUI World!

When view is invoked, an event loop is launched. In order to return back to the console prompt, press the Escape key.

Here is an animated example using a progress bar:

    view [
        bar: progress 30% rate 10 on-time [
            face/data: remainder (face/data + 10%) 100%
            info/text: form face/data
            info/font/color: random white
        ]
        info: text 4 font-color white "30%"
    ]

Spinners are also fun to watch:

Here is a rich-text example. Press TAB key to switch focus between the buttons. Press Enter key to push the button.

    btn-quit: rtd-layout [i/red ["Q"] "uit"]

    view compose/deep [
        rich-text 40x3 transparent data [
            yellow "Hello"  white red " Red "  green "World!^/"
            u "Underline" /u " " s "Strike" /s i " Italic" /i
        ] return
        button "button 1"
        button 4x2 draw [text 0x0 (btn-quit)] [unview]
    ]

An example of mouse support (not all the terminals have mouse support):

    system/view/platform/mouse-event?: yes

    view [panel 80x20 [base 11x1 center "drag me😀" loose]]

Simulating old-style text interfaces:

    view [
        panel navy 40x15 draw [
            pen off fill-pen black box 5x4 36x10
            fill-pen pewter pen black box 4x3 35x9
            pen red text 15x3 "Hello Red"
        ][
            origin 5x5
            rich-text 30x1 pewter data [
                green "Welcome" yellow " to" red " Red "
                u "TUI" /u blue " World!"
            ]
            return
            pad 12x1 button 4 "<OK>" [print "Hi!"]
        ]
    ]

File and folder requesters are also available in TUI, navigation is done using TAB and arrow keys, selection using Enter key:

Images support:

    url: https://upload.wikimedia.org/wikipedia/en/e/e9/Red_Language_Tower_Logo.png
    view [image url 32x15]

Here is the same image displayed in TUI next to the GUI version:

Some examples of 2D vector graphics using the Draw dialect (currently using only block graphics, braille-based graphics in the future):

    view [
        base 80x40 transparent draw [
            pen orange
            triangle 3x2 18x5 5x15
            fill-pen blue
            circle 30x8 5
            pen off
            fill-pen green
            ellipse 50x2 15x15
            pen brick fill-pen brick
            box 3x20 15x30
            pen gold
            line 20x30 28x20 40x28 44x24
        ]
    ]

A special mention to group-box widget. It has a couple of new options for the frame style:

    border-corners: round | #(none)
    border-color: #(tuple!)

Here is a usage example:

    view [
        group-box font-color green " Folders " 26x8 options [
            border-corners: round
            border-color: 255.0.0
        ]
        group-box " Files " 26x8
    ]

This TUI implementation is still not on par with our GUI backends. If some of you are motivated to extend and improve it, contributions are welcome! For example, we did not yet implement menus support. If someone is up to the task, please follow the GUI View menu dialect.

What's next?

We are finishing the work on some significant improvements to the Red and R/S memory management sub-systems and garbage collector that will bring them to the level required for a Red v1.0. Those changes will be released in a bumped 0.6.6 version. Those memory improvements are also needed for completing the work on the async IO branch.

Another version bump will follow with the deprecation of the high-level Red compiler and the addition of a new powerful layer to our Red tower of languages. All those changes are pre-requirements to start our work on 64-bit support.

In the meantime, enjoy this new toy!

We're often asked what Red can be used for, or what apps have been written in Red. Red can be used to write almost anything, but the sparse ecosystem and some missing pieces limit certain use cases. It's used a lot for in-house data processing, custom DSLs, simple GUI apps, and more. We also used it to build Redlake's DiaGrammar product.
When we heard that someone had written a commercial app in Red, we thought that was great news, and we're here to tell you a little about it. Your first question is likely "What is it?" and the second "Where can I get it?". It's an XML processor, and you can find it here. The video on their site goes into detail about use cases and features, so we won't cover that here.


We asked the author to talk about why they wrote SmartXML why those chose Red for the implementation. Here's what they had to say:

---

Once I encountered the need to parse multiple XML files. I always thought that parsing tasks were very simple and that I wouldn't encounter any difficulties because there are things like XPath and XSD that, as I was told, solve all possible problems. However, I quickly realized that this was not the case, and some tools/standards only complicate life and are of little use for real-world usage. Thus, my XML parser project was born, which would allow working with real data rather than synthetic examples where XPath and XSD are truly effective.

I chose Red because I was tired of the complexity of 90% of modern languages and the constant breaking changes in many of them. If you were to ask me what language I would choose to start a project with, looking back, I would still choose Red or perhaps try to use Hare (even considering that it's not yet completed) simply because I want to be sure that my solution will work in 10 or even 20 years. Initially, I thought I could finish within half a year, but the project took me many years. Nevertheless, I brought the project to completion.

The main idea behind SmartXML was:

1. To make the parsing process as visual and simple as possible.
2. To abandon the use of XSD schemas, which create more problems than they solve.
3. To rethink XPath by creating a replacement that could work with complex structures.
4. To implement the ability to generate SQL from XML.
5. To implement batch processing of files.

---

And their advice on development in general:

---

During the writing process, I had a lot of time to think about what constitutes good code and a good product. Here are some of my thoughts that I realized while working on SmartXML:

1. Standard tools have standard problems. And people very often become hostages of such solutions. Most people prefer the shortest, not the most correct path.

2. The time spent on design is directly proportional to the lifetime of the software product. If you spend 10 years thinking about a problem from different angles, you are more likely to come up with an architecturally beautiful and cohesive solution, but maybe not on the first try. The easiest way to make your application 100-200 times heavier than necessary is not to try to think ahead, but to solve problems as they come.

3. You need to have the strength to admit mistakes. Even if they are design mistakes that require rewriting everything from scratch. Sometimes it is wiser to throw everything away and start over than to continue day by day moving into a dead end.

4. Sometimes it's better to do reengineering of the old instead of inventing something new.

5. A language that allows you to write code quickly solves tactical, not strategic tasks. A huge amount of code written in C 30 years ago will still be relevant in another 30 years. Rebol was designed for 20 years, so most of the code on it will still work with minimal modifications in 30 years.

6. Fighting complexity should take as much time as optimizing algorithms. Simple things are always obvious in hindsight. Writing complex code is always easier than writing simple code. Simple code will always be easier to extend and maintain, and it will always have fewer errors.

7. If you can sacrifice 10% of functionality at the cost of removing 90% of code, you should do it.

---

This application is a great fit for Red, whose `parse` function makes processing data easy (as much as XML processing can be) and clearly defined. The latter aspect is important, and often ignored. Can you write code to get that job done, maybe with regexes in this case? Yes. But can you maintain and extend that code? This is where dialects add enormous value. In a use case like this, being able to represent the data in Red format internally, for processing, also makes your life easier.
We thank the author of SmartXML for taking the time to talk about SmartXML with us, and we're excited to see what others do with it. Tell us about your project!
Happy Reducing!

Bumping up the version number was motivated by the breaking syntax change done recently. We do not offer specific builds for a given version number anymore since we provide automatic builds (with builds history) on each new master commit. Though, the changelog for new version number changes will still be provided...and this one is pretty big as it covers about 5000 commits! Hope this will help users who might have missed some changes to catch up.

613 PRs were merged, 2415 fix commits were pushed, among which 902 are closing issues tracked on Github. The most notable new features and changes are listed below with eventual links to docs or previous blog posts describing or mentioning them:

Main new features

• New datatypes: money!, ref!, point2D!, point3D!.
• New codecs: Redbin, JSON, CSV
• New high-performance run-time lexer with instrumentation support.(blog)(doc)
• Interpreter instrumentation support (debugger, tracer, profiler).(doc)
• New powerful APPLY native, with deep interpreter support.(blog)
• Dynamic refinements support.(blog)
• Adds compress and uncompress natives with gzib, zlib and deflate algorithms support.
• Adds gpio:// port with GPIO dialect for RaspberryPi.(blog)
• Adds TAB navigation support to View. (blog)
• Adds raw strings syntax support.(doc)
• Swaps map! and construction syntax. (blog)
• Hashtables are now used for fast lookups in contexts.
• Custom dtoa library implementation to load and form float values.
• Standard library and garbage collector stability vastly improved.

Finished or almost finished features in branches::

• Full IO ports with async support, including new IPv6! datatype.(branch)
• TextUI backend to View.(PR)
• XML codec.(PR)

Other general new features or changes

• New natives: TRANSCODE, SCAN, AS-MONEY, ENHEX.
• New functions: SINGLE?, LAST?, DT, TRANSCODE-TRACE, TRACE, CLOCK, NO-REACT, DO-NO-SYNC
• New routines: SET-TRACE, TRACING?
• Extends EMPTY? to support map! values.
• Allows NONE as value in map!.
• Adds REMOVE/KEY support for removing map! entries.
• Adds FOREACH support for map!.
• Add new-lines automatically when converting maps to blocks.
• Supports issue!, money! and refinement! as key in map
• Allows any-string! series to be matched against binary input in PARSE.
• Adds Accept-Charset and User-agent to HTTP header.
• Update libcrypto version requirement on Linux platforms.
• Adds support for native port actors.
• Extends ROUND/TO to support time!.
• Adds a cross-platform threading library to Red runtime library.
• Adds a FIFO MPMC queue to Red runtime library.
• Adds history support to Red console.
• Extends GET native to accept any-word! as argument.
• FIND is now returning FALSE instead of NONE when used on bitsets.
• Deprecates FIND on objects (redundant with IN).
• Adds ABOUT function to console.
• Preprocessor: fetch-next now supports set-words/paths, get/lit-args, object/series paths.
• COMPLEMENT is now allowed on tuples.
• Adds RANDOM/SECURE.
• SORT support of /skip & /all & /compare integer extended to strings, vectors and binaries.
• Extends FIND/LAST to support hash!.
• Allows FINDing by datatype and typeset on hash! series.
• Removes percent-encoding from files, use double-quotes when needed instead.
• Adds SYSTEM/CATALOG/ACCESSORS.
• Allows pair! for COPY/PART argument on series.
• Allows LOOP and REPEAT to take a float argument as counter.
• Extends REDUCE/INTO to support hash! destination.
• Extends BODY-OF to support action! and native! values.
• REPLACE reimplemented.
• Adds support for REVERSE/SKIP.
• Extends VALUE? native supports any-word! argument.
• Enable image! in red/core on Linux.
• Adds GC support to libRed.
• Extends DISTANCE? to support pair! arguments.
• Adds /KEEP refinement to TRY.
• Implements RENAME action for FILE! datatype.
• Adds routine arguments type-checking support to compiler.
• Accelerates the output speed of LIST-DIR.
• Allows error! values to be modified by users.
• Adds [trace] and [no-trace] function attributs.
• Faster/simpler EMPTY? function implementation.
• Forces the inclusion of Git information in the runtime.
• Optimization for appending values into hashs.
• Extends CHANGE to support image! argument.
• PICK on pair!, date! and time! values now support named accessors as index argument.
• Adds PICK action to event! datatype.
• Makes RECYCLE returns the allocated memory total by default.
• Cleaner implementation of deep reactive paths support.
• Internalizes SYSTEM/STATE/NEAR value.
• Compound scalar datatypes (pair!, date!, time! and tuple!) will now emit ON-CHANGE events when one of their component is changed using an access path (both in compiled and interpreted code).
• Adds reactivity support to bitset! values.
• Adds support for REFLECT action on bitset! values.
• Reports a proper path in compiled path errors.
• Adds memory usage before/after a GC cycle in debug output.
• Measures GC time in debug mode.
• [ARM] Adds "division by zero" and "division overflow" checks in debug mode.

Parse

• KEEP PICK on paren expressions now merges list of values to collected block.
• Optimizes Parse's KEEP memory usage on strings/binaries.
• Removes the end checking in iteration rules.
• Speed optimizations for `TO <token>` rules.
• Adds a fast path for `TO/THRU end` rules.
• New set of optimizations for looping on a char! value in PARSE.

VID

• Smarter merging of actors in a style with custom actors in the instance.
• Added password flag for hidden input
• Adds scrollable flag.
• defines VID origin and spacing values per backend.
• adds next and prev as default options to VID-made faces.

View

• Adds a calendar widget.
• Adds support for tri-state checkboxes.
• Significant GTK backend improvements and upgrades to match other backends.
• Better handling of DPI changes.
• Handles pause and scroll-lock keys.
• Now EVENT/PICKED is a float wheel delta-value in on-wheel event.
• Improves user experience when closing window that contains a large number of faces.
• Scale the font size with Ctrl + mouse wheel in GUI console.
• Adds stop-events function to easily exit a View events loop.
• Adds resize-ns and resize-ew mouse cursors.
• Adds bounds option to /options facet for restricted dragging area.
• Adds new /sync refinement to VIEW function. 
• Adds /color facet info to DUMP-FACE output.
• Improves memory usage when changing /draw facet content.
• Adds support for semi-transparent no-border top-level windows (Windows).
• Minimal dark mode support on GUI console.

Draw

• Switch command parameters to point2D! and float! for subpixel precision.(blog)
• New Direct2D backend for Windows.
• Add line-pattern command for drawing dashed lines.
• Extends scale command to support percent! values.
• Supports image mapping to arbitrary quadrilateral.
• Removes matrix-order command in DRAW.

Red/System

• Adds subroutines to R/S functions.(blog)
• Implements system/io/* instrinsics for CPU I/O read/write instructions.
• Adds system/stack/push-all and system/stack/pop-all intrinsics.(blog)
• Support for atomic operations using system/atomic/* intrinsics.(blog)
• Adds #inline directive to R/S for including assembled binary code.
• Implements support for integer hardware division (ARMv7+).
• Generates optimized code for divisions by a power of 2 literal (ARM).
• Vastly improved loop counter handling robustness.
• Full support for special float values (-0.0, 1.#NaN, 1.#INF, -1.#INF).
• Drops support for % and // operators on float types.
• Adds support for function pointers in literal arrays.
• Switches to 16-bytes stack alignment on Linux.
• Adds log-2 function to standard library.
• Allows cross-referenced aliased fields in structs defined in same context.
• Support multiple variable assignments.
• Allows grouping arguments and local variables type specification.
• Relax lexical format of hexadecimal literals.
• Allows get-paths pointers on function nested in contexts.
• Adds support for simple pointer to pointer type.
• Allows function! type to be specified for local variables.

Toolchain

• Optimizes critical section in linker, twice faster linking time now on average.
• Adds --show-func-map compilation option.
• Various minor improvements to PE format support.
• Adds working Linux-musl target.
• Expands libRedRT to support View backend.
• Adds --no-view option for Red binaries.
• Shows the global words used by the toolchain in the compilation report.
• Adds new Pico compilation target.

Sometimes deep changes take a huge amount of code. Sometimes they take a lot of detailed explanation and consideration, leading to long discussions and people taking sides. Rarely does an important syntactic change to a language happen quickly, with universal agreement, simple implementation, and tools to help update scripts in the wild. Today is one of those rare days.

Admittedly, this idea has been discussed for a long time. It would surface, people nodded their virtual heads, and it would submerge again. Today it's ready to deploy. Not only that, but Rebol3 is making the same change, so the two languages will still be compatible in this regard.

What is the change?

It's easy to describe. Today, map! values use this syntax: #(...) and construction syntax (sometimes called serialized form or loadable form) looks like this: #[...]. Going forward, those syntactic forms will be swapped. Why? The answer is easy. In Redbol langs, blocks do not evaluate by default, you have to do or reduce them. Parens, on the other hand, do evaluate by default. Today, maps use paren-like syntax, but they do not evaluate, while construction syntax uses block-like syntax, but does evaluate. This is a carryover from Rebol, so the major concession here is that Red and Rebol3 will no longer be compatible with Rebol2's construction syntax.

If you've never heard of construction syntax, there's a nice explanation of it here. Red only supports a few values via construction syntax today, all datatype literals, true, false, none, and unset; but eventually it will support much more. If you look at the help for mold, you'll see that /all is TBD (very partially implemented for now), and that's how you create loadable, serialized, data that can safely and easily contain any value (like redbin but readable by humans). It helps avoid cases where none or true/false may load as words. This is also why construct evaluates those specific words (including also on/off/yes/no), but not others. When loading untrusted data, we have to strike a balance between ease of use and safety.

What do I have to do?

Not much. There are two tools available, which will convert your scripts automatically. The first is small and simple, showing just how powerful Red is, and leveraging its lexer instrumentation. You can find it here (once merged, that branch may go away and the tool will be in the main branch). The second is a more advanced and standalone tool written by @hiiamboris. You can find that here.

For the simple script it's necessary that you run it under a current version of Red's lexer. Once the change is in place, running it under the new lexer will make the exact opposite change. Of course, you can compile it into a standalone EXE, or use Boris' app, which is already available.

To run the simple script from a Red console, you can just:

  do https://raw.githubusercontent.com/red/red/master/utils/migration/map-conv.red

then you can use help map-conv to see all the available options. By default it runs in preview mode, making no changes, and showing you all the instances it found, which will be changed if you use the /save refinement. A copy of each changed file is created with a .saved extension in the same folder. If you don't want them, you can use the /no-copy refinement.

A word of warning, if you run the conversion tools a second time on the same files, they will convert the data back, because they can't know what you're thinking. On the bright side, this is an effective "undo" feature. Still, it's wise to back up your data before running any tools against them.

Thanks to the power of Red, and these tools, there are already PRs pending for updates to docs and community scripts. But the other thing you can do to help is to let us know when you find things that need to be updated for this change, and especially if you run into any issues when converting your own code.

Conclusion

We know changes like this can be hard, but better now than when there is even more code in the wild that would be affected. If we had been any other language, this long-view improvement might not have happened. Only because Red (and Redbol langs in general) can consume their own code as data and have powerful parse and lexing features, was this change so easy and safe. It's still a code-porting process, and if you run multiple versions of Red you may need to maintain separate versions for a while. The other hard part is retraining your hands and eyes to the new syntax.

Happy Reducing!

We finally got tab navigation implemented! You might think it should have been an easy feature to add, but achieving a consistent and controllable behavior across our different native GUI backends is not that straightforward. So we opted for a mixed implementation with a general high-level navigation layer in Red and left spatial navigation handling to each backend, in order to preserve the native behavior as much as possible.

Automatic navigation

By default, pressing TAB key will allow you to navigate to all the GUI widgets in a window, capable of acquiring the focus. Once the last widget is reached, the next TAB press will circle back to the first focusable widget. Conversely, back-navigation can be achieved using Shift-TAB key combination, circling from first face to last one. Here is a simple example:

    view [
        text  "Name"     field focus return
        text  "Surname"  field return
        below
        check "Single"
        check "Employed"
        button "Send"
    ]

Note: check-boxes selection/unselection is done using the Space key (default on Windows).

It is possible to make a face "TAB-transparent", so that TAB navigation will skip it in both directions. This is achieved by removing the focusable flag from a navigable face. For example, in the following code, clicking on the "Click me!" button will toggle the button's focusable flag on and off (using set-flag/toggle):

    view [
        text "Name"     field focus return
        text "Surname"  field return
        below 
        check "Single"     
        check "Employed"   
        button "Send"
        button "Click me!" 100 [
            face/text: pick ["TAB ignore" "TAB stop"] to-logic face/flags          
            set-flag/toggle face 'focusable
        ]
    ]

In case of area face, the default behavior for TAB navigation means that tab characters cannot be input in the area. In such cases, the alternative Ctrl-TAB key combination can be used to input tab characters. In case the focusable flag is removed from an area face, then TAB key will directly produce tab characters. Here is an example:

    view [
        text "Name"     field focus return
        text "Surname"  field return
        below 
       	text "Comments" com: area 
        button "Send"
        button "Toggle Area" [set-flag/toggle com 'focusable]
    ]

Note: when the focusable flag is on, Ctrl-TAB is used to input tab characters, when it's off, it's just using TAB key.

Manual override

In some cases, the user can decide to set a different path for keyboard navigation. For each navigable face (the ones with a focusable  flag), it is possible to manually define the next and/or previous one when tabbing forth and/or back. In order to do so, next and prev options can be set to define how tabbing will navigate to the next or previous face.

Here is a simple example where the default navigation is changed to jump into fields marked as invalid or empty (using pink background) after a typical form submission:

    view [
        group-box 2 [
            style error: field pink
            text "Name"     field "John"
            text "Surname"  field "Smith"
            text "Age"      error "abc" focus
            text "Address"  error "-"
            text "Zip code" field "12345"
            text "City"     error
            text "Country"  error
        ] return
        btn-send: button "Send"
        do [
            list: collect [foreach-face self [if face/color = pink [keep face]]]
            forall list [list/1/options/next: list/2]
            btn-send/options/next: list/1
        ]
    ]

Notes:

• For the sake of simplicity in this example, only forward navigation is restricted, backward navigation will visit all focusable faces.
• In the do block, self refers to the window face, as do denotes a global section, not widget-related.
• When list/1 refers to the last element, list/2 returns none, so it does not point to any specific face. In such case, the default tab navigation will automatically (and conveniently) select the next face, which is the "Send" button.
• The last line is there to connect that last face (the button) to the first face in our restricted list.

Other notable changes

An important change concerns the insert-event-func function specification, it now requires a name as argument:

    >> ? insert-event-func
    USAGE:
         INSERT-EVENT-FUNC name fun

    DESCRIPTION: 
         Adds a function to monitor global events. Returns the function. 
         INSERT-EVENT-FUNC is a function! value.

    ARGUMENTS:
         name         [word!] 
         fun          [block! function!] "A function or a function body block."

The name is an arbitrary word that only needs to be unique, so it becomes easier to check if a given global handler has been installed or not. It also makes it easier to remove it, as it can be referred by name in remove-event-func. Existing handler names can be checked using:

    >> extract system/view/handlers 2
    [tab field-sync reactors radio enter debug dragging]

Please update your code if you have been using those functions.

Enjoy!

Maybe you didn't notice, but Red/View, our GUI engine, has subpixel precision from the beginning! Unfortunately, that level of precision was not directly accessible to end users, until now. 

Actually, it would be more accurate to say that we had subpixel resolution only so far. The guilty part is the pair! datatype being limited to integer components only, while subpixel precison requires decimal numbers. So we have recently introduced new datatypes to cope with that.

What urged us to make those changes now was a very peculiar visual glitch caused by that dissonance. That glitch happens during face dragging operations. Here is an example using our View test script:

As you can see, on some positions, the face starts shaking while the mouse cursor remains still. This affects any type of face. The shaking is about ±2 pixels. It is caused by the difference in precision between the /offset facet expressed in integer numbers and the backend API, which only deals with floating point numbers. The accumulated error when converting integer->float->integer gives a 2 pixels difference. Such error happens on displays where the scaling factor is different from 100%. With the rise of 2K, 3K and 4K displays, a scaling factor > 100% has become the norm, making this glitch more frequent. You might think that this is not a big issue until you start building custom scrollbars and see your entire scrolled content shaking massively...

New point datatypes

In order to provide decimal positions and sizes for View faces, extending the existing pair! datatype was considered, though, the pair syntax can hardly scale up for such needs:

    2343.122x54239.44
    2343.122x54239.44x6309.332
    2343.122x54239.44x6309.332x442.3321
    2.33487e9x54239.44
    2.33487e9x54239.44x9.83242e17
    2.33487e9x54239.44x9.83242e17x5223.112
    1.#infx1.#infx1.#inf

As you can notice there, it quickly becomes difficult to read and identify the individual components. So we opted for adding a new literal form (hence a new datatype) that matches how coordinates for two or more dimensions are commonly represented:

    (2343.122, 54239.44)
    (2343.122, 54239.44, 6309.332)
    (2343.122, 54239.44, 6309.332, 442.3321)
    (2.33487e9, 54239.44)
    (2.33487e9, 54239.44, 9.83242e17)
    (2.33487e9, 54239.44, 9.83242e17, 5223.112)
    (1.#inf, 1.#inf, 1.#inf)

Such literal forms requires the comma character to be a delimiter, so that it cannot be used anymore as a decimal separator. That was, unfortunately, a necessary decision in order to unlock such literal forms. The gains should be bigger than the loss.

So, two new datatypes have been added:

• point2D!: a two-dimensional coordinate or size.
• point3D!: a three-dimensional coordinate or size. 

Their canonical lexical forms are:

    (<x>, <y>)
    (<x>, <y>, <z>)

    where <x>, <y> and <z> are integer or float numbers.

Optional spaces are allowed anywhere inside the point literals on input, they will be removed on loading.

    >> (1,2)
    == (1, 2)
    >> (  1.35 ,  2.4  )
    == (1.35, 2.4)

Both for 2D and 3D points, their components are internally stored as 32-bit floating point numbers, so that their precision is limited to 7 digits. This should be far enough for their use-cases though.

When one of the components has a fractional part equal to zero, it is displayed without the .0 part for easier reading. Similarly, integers are accepted as input for any component and are internally converted to a 32-bit float.

    >> (0.3, 0.5) + (0.7, 0.5)
    == (1, 1)
    >> (2.0, 3.0)
    == (2, 3) 

Creation

Besides literal points, it is possible to create them dynamically, the same way as pairs, using make, to or one of the as-* native functions:

    >> make point2D! [2 4.5]
    == (2, 4.5)
    >> to-point2D 1x2
    == (1, 2)
    >> as-point3D 1 (3 / 2) 7 * 0.5
    == (1, 1.5, 3.5)

Accessors

Point components can be individually accessed using ordinal numbers or component names using action accessors or path syntax:

    >> pick (2, 4.5) 1
    == 2.0
    >> pick (2, 4.5) 'y
    == 4.5
    >> p: (2, 4.5)
    == (2, 4.5)
    >> p/x
    == 2.0
    >> p/y: 3.14159
    == 3.14159
    >> p
    == (2, 3.14159)

Math operations

Basic math operations are supported as well:

    >> (1, 1) + (2, 3.5)
    == (3, 4.5)
    >> (1, 1) - (2, 3.5)
    == (-1, -2.5)
    >> (2, 3) * (10, 3.5)
    == (20, 10.5)
    >> (20, 30) / (10, 3)
    == (2, 10)

Notice that mixing pairs with point2D in math expressions is allowed. The pair value will be promoted to a point2D in such case (as integers with floats):

    >> 1x1 + (2, 3.5)
    == (3, 4.5)

Other actions/natives

    >> round (2.78, 3.34)
    == (3, 3)
    >> round/down (2.78, 3.34)
    == (2, 3)
    >> random (100, 100)
    == (53, 81)
    >> zero? (0, 0)
    == true
    >> min (10, 100) (24, 35)
    == (10, 35)
    >> max 10x100 (24, 35)
    == (24, 100)    

Notice that pairs will be promoted to point2D in mixed use cases with min/max.
  View and VID adjustments

The main changes are in face! object:

• /offset: now only accepts point2D! values.
• /size: accepts both pair! and point2D! values.

In VID, both pair and point2D values can be used to denote positions and sizes, so that VID is backward compatible. All previous VID code should work without any change. VID will convert all positions to point2D values. Sizes by default in VID, keep using pairs, unless a point2D is provided by the user.

All Draw commands that were accepting pairs now also accept point2D values for higher precision.

The related documentation will get updated soon to reflect those changes.

In order to illustrate the difference in using pairs and point2D positions, here is a (not so) simple animation comparison showing the subpixel positioning difference (correctness of animation in this case is privileged over simplicity of code):

    view/no-wait [
        size 800x200 space 0x0
        b1: box 2x40 red return
        b2: box 2x40 blue
    ]
    x: b1/offset/x
    until [
        do-events/no-wait                   ; processes GUI events in queue
        wait 0.1                            ; slows down the animation
        do-no-sync [                        ; switches to manual faces redrawing
            b1/offset/x: b1/offset/x + 0.1
            b2/offset/x: to-integer x: x + 0.1
            if all [b1/state b2/state][show [b1 b2]] ; redraws both faces
        ]
        any [b1/offset/x > 700 none? b1/state none? b2/state]
    ]

Here's the zoomed capture of the result (on a display with 200% scaling factor):

The red bar uses the newly enabled subpixel precision, while the blue bar simulates the old pair positioning precision (so only allowing integer positions). What you can see is that the red bar makes two smaller steps while the blue bar makes a single one, looking more "jumpy".

This means that now animations on displays with a scaling factor > 100% can be smoother as they benefit from more accurate positioning.

Note: the animation code above is far from being simple or elegant, we'll be working on improving that. The animation code could have been quite simpler by using a rate option in VID and putting the animation code in a on-time handler. Though, timer events firing (especially on Windows) are not very reliable, so unrolling a custom event loop lowers that risk when the timing is critical (like for fast game loops).

As a conclusion, here is an old-school style starfield code demo using 2D and 3D points (moving mouse left/right changes the stars speed):

Let us hear your feedback about those changes on our Gitter (now Matrix) channel.

Enjoy!

It's Time to Apply Yourself to Red
Red has never had an apply function, though we knew it would come someday. In the meantime, some of us rolled our own. Gregg's was simple, neither flexible nor efficient, and just a couple lines of code. Boris made a much more capable version, but it could still only be so fast as a mezzanine. R2 had a mezz version, which suffered the same problem. All that changes now. Apply is dead! Long live Apply! It required deep work, and a lot of design effort, but we think you'll like the results, whether you're a high-level Reducer, or anxious to see how much leverage you can, um, apply, from a functional perspective. Everybody wins.
If you don't know what apply is, in terms of functional languages, take a moment and read up. If you can get through the introduction there without getting dizzy, great. If your head is spinning, feel free to stop after the first section of this article and ignore the deep dive. You still get 90% of the value for most high level use cases. Gregg got so dizzy that he fell down, but was still able to help with this article.
Function application is largely about composition. How you can combine functions in a concise way for maximum leverage and minimum code. The problem with its design in many languages is that it makes things harder to understand. Rather than concrete functions names, there is indirection and abstraction. It can be tricky to get right, especially in a flexible language like Red, while also maintaining as much safety as possible. You can drive fast, but still wear your seat belt.
Dynamic Refinements
This subtle feature is likely to see wide use, because it will reduce code and let people build more flexible functions. It's also easy to explain. Here's an example. First, how you would write it today:

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /only "Appends a block value as a block"
][
    either only [
        append/only series reduce :value
    ][
        append series reduce :value
    ]
]

With dynamic refinements, it can be done like this.

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /only "Appends a block value as a block"
][
    append/:only series reduce :value
]

In case you missed the subtlety, it's :only being a get-word in the path. That's right, it's evaluated, rather than being treated literally, just like you use in selector paths. The value for the dynamic refinement is taken from its context, which can be a refinement in the function, or a local value. It can be any truthy value to use the refinement, and it is only retrieved not evaluated. That means you can't use a computed value directly, which makes it safer. Other than that, paths work just as they always have. If a refinement is used, fixed (literal) or dynamic, which can be mixed however you want, any arguments it requires will be fetched and used. Otherwise they are silently ignored, so you don't have to clutter your code or worry about what a dynamic path expression will consume.

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /only "Appends a block value as a block"
    /dup  "Duplicate the value"
        count [integer!]
][
    append/:only/:dup series reduce :value count
]

>> only: false  dup: false  repend/:only/:dup [] [1] 3
== [1]
>> only: true  dup: false  repend/:only/:dup [] [1] 3
== [[1]]
>> only: false  dup: true  repend/:only/:dup [] [1] 3
== [1 1 1]
>> only: true  dup: true  repend/:only/:dup [] [1] 3
== [[1] [1] [1]]

This is an incredibly exciting and powerful feature. It's a shame there isn't more to explain. ;^)
Function Application
Functional Programming has never yet become mainstream, though it has periodic rises in popularity and a devoted following in many language camps. Even Red. Yes, Red is a functional language. It's not a pure functional language, because functions can have side effects, but functions are first class values and can be passed around like any other. It lets you do things like this:

>> do-math-op: func [
    fn    [any-function!]
    arg-1 [number!]
    arg-2 [number!]
][
    fn arg-1 arg-2
]

== func [fn [any-function!] arg-1 [number!] arg-2 [number!]][fn arg-1 arg-2]
>> do-math-op :add 1 2
== 3
>> do-math-op :subtract 1 2
== -1

That's called a Higher Order Function, or HOF. It also means you can return a function as a result.

>> make-multiplier: func [
    arg-1 [number!]
][
    ;; We're making a function and returning it here.
    func [n [number!]] compose [(arg-1) * n]
]

== func [arg-1 [number!]][func [n [number!]] compose [(arg-1) * n]]
>> fn-m: make-multiplier 4
== func [n [number!]][4 * n]
>> fn-m 3
== 12

That's all well and good, but what if you want to call different functions that take a different number or type of arguments? Now it gets tricky, and inefficient. Because Red uses free-ranging evaluation (function args are not contained in a paren or marked as ending in any way at the call site), how do you handle different arities (number of arguments)? Here's a very simple apply mezzanine:

apply: func [
    "Apply a function to a block of arguments." 
    fn [any-function!] "Function to apply" 
    args [block!] "Arguments for function" 
    /only "Use arg values as-is, do not reduce the block"
][
    args: either only [copy args] [reduce args] 
    do head insert args :fn
]

So easy! The power of Red. But there is a cost. It's a mezzanine function, so it's slower than a native function, the args are copied or reduced, and then do is used to evaluate it. We can live with this kind of overhead for a great deal of Red code, but apply is a building block, and may be used in deep code where performance is important. You may also have noticed that the fn argument is any-function!, that means two things: 1) If you want to know the name of the function, the word that refers to it, too bad. You'd have to pass another arg for that. 2) Refinements. You can't use them with this model. And that limitation is a killer. For example, you could pass :append but not :append/:only. And there's no way you could have an /only refinement in your function and just pass that along. Until now. 
The Real Apply

Here's the new apply native that is now available in Red:

USAGE:
    APPLY func args

DESCRIPTION: 
    Apply a function to a reduced block of arguments. 
    APPLY is a native! value.

ARGUMENTS:
    func    [word! path! any-function!] "Function to apply, with eventual refinements."
    args    [block!] "Block of args, reduced first."

REFINEMENTS:
    /all    => Provides a continuous list of arguments, tail-completed with false/none.
    /safer  => Forces single refinement arguments, skip them when inactive instead of evaluating.

Notice that the func arg can now be a word! or path!, so you can use the name, or a path including refinements. That's right, the Dynamic Refinements feature explained above works with apply too. And having access to the name being used to call the function is enormously valuable when it comes to tracing and debugging. It's a huge win. 
One big difference is that all the arguments for apply are in a single block. Another is that you MUST include the on/off values for each dynamic refinement in the arg block, they DO NOT come from the environment (context). Compare this version to the one in the Dynamic Refinements section. Really, paste them into an editor and look at them side by side. 

>> only: false  dup: false  apply 'append/:only/:dup [[] [1] only dup 3] 
== [1]
>> only: true  dup: false  apply 'append/:only/:dup [[] [1] only dup 3] 
== [[1]]
>> only: false  dup: true  apply 'append/:only/:dup [[] [1] only dup 3] 
== [1 1 1]
>> only: true  dup: true  apply 'append/:only/:dup [[] [1] only dup 3] 
== [[1] [1] [1]]

; Refinement names in the arg block don't have to match the spec.
; You can use other names, or literal values. For example:

apply 'append/:only/:dup [[] [1] false false 3] 

a: b: false  apply 'append/:only/:dup [[] [1] a b 3]

It means you have to be more careful in lining things up with the function spec, in a different way than you're used to, but here's where you could use a computed refinement value, which may be useful for generative scenarios like testing. You can also see that both refinements are dynamic, so both need an associated on/off value in the args block.

>> only: does [random true]
== func [][random true]
>> blk: copy [] dup: no  loop 10 [apply 'append/:only/:dup [blk [1] only dup none]]
== [1 [1] 1 [1] [1] 1 [1] [1] 1 1]

But if you use a fixed refinement, it does not need the extra on/off value. In this example, /dup is always used, so there is no on/off value for it in the args, block, but its associated count arg has to be there, and is type-checked normally.

>> only: does [random true]
== func [][random true]
>> blk: copy [] dup: no  loop 10 [apply 'append/:only/dup [blk [1] only 2]]
== [[1] [1] 1 1 1 1 [1] [1] [1] [1] 1 1 [1] [1] [1] [1] 1 1 [1] [1]]
>> blk: copy [] dup: no  loop 10 [apply 'append/:only/dup [blk [1] only none]]
*** Script Error: append does not allow none! for its count argument
*** Where: append
*** Near : apply 'append/:only/dup [blk [1] only none]
*** Stack: 

But those aren't your only options. During the design of apply there was a lot of discussion about the interface(s) to it, and different use cases benefit from different models. For example, programmatically constructed calls means you need to build the path, and keep the args in sync. It may be easier to build a single block with everything in it, which you can do.

>> only: does [random true]
== func [][random true]
>> blk: copy []  loop 5 [apply :append [blk [1] /only only /dup no none]]
== [1 1 [1] [1] 1]
>> blk: copy []  loop 5 [apply :append [blk [1] /only only /dup yes 2]]
== [[1] [1] 1 1 1 1 [1] [1] 1 1]
>> blk: copy []  loop 5 [apply 'append [blk [1] /only only /dup no none]]
== [1 1 [1] [1] 1]
>> blk: copy []  loop 5 [apply 'append [blk [1] /only only /dup yes 2]]
== [[1] [1] [1] [1] 1 1 1 1 [1] [1]]

This interface is used if the first argument to apply is a function or lit-word, and /all is not used.
Apply/all

This is the most direct model, and what the others map to internally. In those models, you get friendly refinements, which may be optional, and those may have their own optional args. It's great for humans, and one of the best things about Redbol languages. But look at it from the view of a function spec.

>> print mold spec-of :append
[
    {Inserts value(s) at series tail; returns series head} 
    series [series! bitset! port!] 
    value [any-type!] 
    /part "Limit the number of values inserted" 
    length [number! series!] 
    /only {Insert block types as single values (overrides /part)} 
    /dup "Duplicate the inserted values" 
    count [integer!] 
    return: [series! port! bitset!]
]

The doc string isn't used when calling functions, so we can ignore that for this discussion. We can also ignore return: here. What's left is all the parameters (we often interchange arg and parameter, but there's a technical difference. Parameters are the named slots in a function spec, and arguments are the actual values passed in those slots). There are seven of those. Some are required args, some are refinements, and some are optional args. But there are seven slots, and when a function is invoked it expects there to be seven values on the stack that it could use if needed, or ignored if not.

When you use /all, you're telling apply that you are going to provide all those values in a single block, in the order of the function spec, like a stack frame (don't worry about the terminology too much if it's unfamiliar). Apply/all calls look like this:

1.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none false true 2]]
== [1 1 1 1 1 1 1 1 1 1]

2.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none true true 2]]
== [[1] [1] [1] [1] [1] [1] [1] [1] [1] [1]]

3.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none true false 2]]
== [[1] [1] [1] [1] [1]]

4.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none only false none]]
== [1 [1] 1 1 [1]]

5.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none 1 false none]]
*** Script Error: append does not allow integer! for its only argument
*** Where: append
*** Near : apply/all 'append [blk [1] false none 1 ]
*** Stack:  

6.
>> blk: copy []  loop 5 [apply/all 'append [blk [1] false none true true none]]
*** Script Error: append does not allow none! for its count argument
*** Where: append
*** Near : apply/all 'append [blk [1] false none true ]
*** Stack:  

7.
>> blk: copy []  loop 5 [apply/all 'append [blk [1]]]
== [1 1 1 1 1]

You can see that the refinement slots are now anonymous logic values in examples 1-3, but 4 uses only, our func from earlier examples, which randomly returns true or false. You can use anything that evaluates to logic! for a refinement slot. 5 shows that it has to be logic!, not just truthy, because types are checked (and logic refinement values then stand out against none argument values). And 6 shows that if you use /dup (second from the last arg), the count arg is also type checked, where 4 didn't complain because /dup was false. Confused yet? Look at 7. How can that work? I thought we had to fill all the slots! Yes and No. Apply "tail completes" the block with false/none values for you, if you don't provide enough arguments. Think of find, which has 16 slots. You only have to include enough args up to the last one you need. That may help, but if you need to use /match, the last refinement in find, you will have to provide all 16 args. Before you think this is unacceptable, consider our first repend example:

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /only "Appends a block value as a block"
][
    append/:only series reduce :value
]

It would look like this:

repend: func [
    {Appends a reduced value to a series and returns the series head} 
    series [series!] 
    value 
    /as-one "Appends a block value as a block"
][
    apply/all 'append [series reduce :value false none :as-one]
]

The point here is not to show that apply/all is longer, but that we can use a different name, where the first example must use :only in the path (make your own version to try it). Not all refinements will propagate using the same name. With /all it cares only about the logic value in the slot.

Apply/safer

/Safer is a form of short-circuit logic. Its purpose is to avoid the evaluation of unused args. Without it, everything in the args block is evaluated, but may be discarded if an associated refinement isn't active. The easiest way to explain this is with an example. 

; This is the function we're going to apply
applied: func [i [integer!] b /ref c1 /ref2 /ref3 c3 c4][
    reduce ['i i 'b b 'ref ref 'c1 c1 'ref2 ref2 'ref3 ref3 'c3 c3 'c4 c4]
]
; And some passive and active arg values
c: 0
bar40: does [4 * 2]
baz40: does [c: c + 1 456]

; Refinement args are evaluated
apply       'applied [10 "hi" /ref on bar40  /ref3 on  baz40            "ok"]

; No /safer difference because all refinement args are single values.
apply       'applied [10 "hi" /ref no bar40  /ref3 no  baz40            "ok"]
apply       'applied [10 "hi" /ref no bar40  /ref3 no  (c: c + 1 4 * 2) "ok"]
apply/safer 'applied [10 "hi" /ref no bar40  /ref3 no  (c: c + 1 4 * 2) "ok"]


apply       'applied [10 "hi" /ref no 4 * 2  /ref3 no  baz40            "ok"]
apply       'applied [10 "hi" /ref no bar40  /ref3 no  c: c + 1 4 * 2   "ok"]
apply/safer 'applied [10 "hi" /ref no 4 * 2  /ref3 no  baz40            "ok"]
apply/safer 'applied [10 "hi" /ref no bar40  /ref3 no  c: c + 1 4 * 2   "ok"]

In Real Life

Here are some examples of these new features being applied in the Red code base. The parse-trace example is jaw-dropping, not because it turns 9 lines into 1 (though, wow!), but because it makes the intent so much clearer and eliminates so much redundant code and the errors they can lead to. Not only that, it adds a capability! Before now you couldn't use both refinements together, i.e. parse-trace/case/part, but now you can.

Things we left out

The design of apply took many turns, with long and vigorous discussion and analysis. Many views and preferences were expressed, and which ultimately led to dynamic refinements as what we called straight sugar. That is, syntactic sugar at its sweetest. We knew /all had to be there, as that's what the others build on, but it was originally the default. We all eventually agreed that it shouldn't be, as it's the lowest level and likely the least directly used, though still vital for some use cases. Our problem was striking a balance between what would be most useful, with minimal overlap in use cases, and making the rules too complex to remember and get right. So a couple models didn't make the cut.

If Dynamic Refinements are straight sugar, the candy-wrapper version might be something like this, where you can still use a path, but only the args are in the block.

apply-args: function [
    "Make straight sugar call from semi-sweet model."
    fn [word! path!] "Get-word refinements come from context."
    args [block!]	 "Args only, no refinement values."
][
    ; Use temp result so we don't return any extra args
    do compose [res: (fn) (args)]
    res
]

>> apply-args 'append [[] [a]]
== [a]
>> apply-args 'append/only [[] [a]]
== [[a]]
>> only: false
== false
>> apply-arg 'append/:only [[] [a]]
== [a]
>> only: true
== true
>> apply-args 'append/:only [[] [a]]
== [[a]]

It's easy, but has quite a bit of overhead, because of compose and do. Remember, this amount of overhead only matters in loops running thousands of times at the very least, or in a real-time interactive interface. When in doubt, clock it. Write for humans to understand, and only optimize as needed (and after you know what's slow)

Another model is name-value args. That is, you provide a structure of arg names and their values, which is applied. It can make some code much clearer, but you also have to make sure the names match, so any refactoring of names will break code, which doesn't happen if args are positional. This is a bit involved, but it shows the power of Red. We'll use objects in our example, for a particular reason. That reason is values-of. The idea being that apply/all wants all the args, in order, and every slot filled. If your object matches the function spec, it's a perfect match. But making objects manually that way is error prone. So we'll use reflection for an object tailor-made for a given function.

Step 1: Find all the words in the spec. Remember it could have doc strings and arg types as well.

func-spec-words: function [
    "Get all the word-type values from a func spec."
    fn [any-function!]
    /as type [datatype!] "Cast results to this type."
][
    arg-types: make typeset! [word! lit-word! get-word! refinement!]
    parse spec-of :fn [
        ; If we want an apply-specific version of objects, we could
        ; denote refinements with a sigil for added clarity.
        collect [
            any [set w arg-types keep (either type [to type w][w]) | skip]
        ]
    ]
]

Step 2:  Make an object from that

func-spec-to-obj-proto: function [
    "Returns an object whose words match a function's spec."
    fn [any-function!]
    ; The idea here is that you can both preset values that are in the spec,
    ; and also extend the object with extra words, which will be ignored.
    /with args [block!] "APPLY/ALL ignores extra values."
][
    obj: construct/with any [args []]
    construct append func-spec-words/as :fn set-word! none
    ; Refinement values in APPLY/ALL calls MUST be logic!, not none!.
    foreach word func-spec-words :fn [
        if refinement? word [set in obj to word! word false]
    ]
    obj
]

Step Aside: Here's another approach, which combines steps 1 and 2, and lets you use a path for the function arg.

; Alt approach to func-spec-to-obj-proto, does NOT allow extending the spec.
make-apply-obj-proto: function [
    "Returns an object whose words match a function's spec."
    fn [any-function! word! path!]
    /with args [block!] "TBD: If APPLY doesn't ignore extra values, keys must be in spec."
][
    if path? :fn [refs: next fn   fn: first fn]        ; split path
    if word? :fn [
        name: fn                                       ; hold on to this for error report
        set/any 'fn get/any fn                         ; get func value
        if not any-function? :fn [
            do make error! rejoin ["FN argument (" name ") does not refer to a function."]
        ]
    ]                                                  ; get func value
    obj: construct append func-spec-words/as :fn set-word! none	; make object
    ; Refinement values in apply calls MUST be logic!, not none!.
    foreach word func-spec-words :fn [
        if refinement? word [set in obj to word! word false]
    ]
    if refs [foreach ref refs [obj/:ref: true]]        ; set refinements
    ; can't use obj/:key: if key is a set-word!
    if args [foreach [key val] args [set in obj key :val]]  ; set arg values
    obj
]

o-1: make-apply-obj-proto/with 'find/case/part/tail/skip/with [wild: "*?+" length: 10 size: 2]

Step 3. Using the object with APPLY

; We finally get here, and it's anticlimactic.
apply-object: func [
    "Call APPLY using an object's values as args."
    fn  [any-function!]
    obj [object!]
][
    apply/all :fn values-of obj
]

And an example call:

>> o-fctm: make-apply-obj-proto/with 'find/case/tail/match [series: [a b c] value: 'a]
== make object! [
    series: [a b c]
    value: 'a
    part: false
    length: none
    only: false
    case: true
    same: fal...
>> apply-object :find o-fctm
== [b c]

But you may see that this is verbose and inefficient, making a whole object just for a call like this. And you'd be right. It's just an example.

You don't want to recreate objects like this, especially in a loop. But you don't have to. You can reuse the object and just change the important values in it. This blog is getting long already, so we'll leave that as an exercise for the reader, or a question in community chat. And if you reuse the same object, the overhead is minimal.

We even talked about an idea whose time has not come, and is not guaranteed to work in the future. Here's the idea:

dyna-ref: func [p [path!]][
    res: make path! collect [
        keep p/1
        foreach val next p [
            case [
                get-word? val [if get :val [keep to word! val]]
                all [paren? val get-word? val/1] [if do next val [keep to word! val/1]]
                paren? val [do make error! "Sorry, has to be (get-word expr) for use with dyna-path."]
                'else [keep val]
            ]
        ]
    ]
    res
]

c: true
print mold dyna-ref 'a/b/:c/(:d true)
print mold dyna-ref 'a/b/:c/(:d false)
c: false
print mold dyna-ref 'a/b/:c/(:d true)
print mold dyna-ref 'a/b/:c/(:d false)

That's right, it's a dialected path! that builds a dynamic path. Crazy, right? You may know that while paths can currently contain parens, for Rebol compatibility, that feature may go away. It has deep design implications, but is also very handy at times. And while this isn't part of Red, it's another example of how Red lets us think off the beaten path.

Interpreter improvements

Dynamic refinements and function application are supported at interpreter level, for maximum efficiency and code reuse (mostly arguments fetching and type-checking). In addition to that, long standing issues and needed simplifications have been made in the interpreter code. Here is the changelog:

Function arguments cache entirely reimplemented:

• Massively reduces the amount of code needed to manage the caches.
• Simpler and faster cache design, O(1) lookup time for refinements in paths.
• Adds a context! to native!, action! and routine!, to speed up word lookups.
• Fixes long standing issue #4854 (CRASH & CORRUPTION in "dynamic" function calls with refinements)

Simpler reimplementation of op! datatype and its evaluation:

• red-op! structure redesigned: it is now a shallow copy of the underlying function (nodes are not copied), the sub-type is stored in the op! header.
• is infix function has been deprecated and replaced by a prefix version: relate.
• Massive code reduction and simplification compared to previous version. Now op! maximally reuses the interpreter code for evaluating other functions.

Those changes lead to a general interpreter speed-up of 3-5% and up to 20% in some cases.

Additional language changes:

• The in native now accepts any-function! as its first argument and refinements as the second argument. Refinements, if matched, will be converted to word values. This makes in a fast way to check if a symbol is part of an object or function spec and return a word bound to that context.

Conclusion

The most important thing you should do now is try it. It's a new design, and we want to hear what people like, see what they try, and where it falls short.

Happy Reducing!
-Red Team

Since many years, we are offering pre-built binaries for the Red toolchain, as a more convenient way to use Red, even if it is not strictly needed, as Red can be run from its sources, the toolchain being run by a Rebol2 interpreter. As the Red REPL and toolchain are not run by the same engine, the console (REPL) used to be compiled on first run of the `red` executable (when no arguments was provided or a Red script was passed). This resulted in a significant delay on the first use of the console (both for the GUI and CLI versions). 
We have now decided to change that by providing separate pre-built binaries for the consoles and toolchain. This is a temporary split until Red gets self-hosted, at which point we can recombine everything into a single binary.
Another change is the temporary dropping of the semantic versioning until version 1.0 and related "stable" releases, as it seems to be too confusing to some users (Red being still in alpha stage). This also will remove a tendency from some users to care more about version increments than feature availability and work being done overall. We will now be proposing only pre-built binaries for latest commit, though older binaries will still be available if that can be of any help to anyone.
So the pre-built binaries now are:

• Red GUI : Red interpreter + View + GUI console
• Red CLI : Red interpreter + CLI console
• Red Toolchain : Encapper for Red + Red/System compiler

We are also considering ways to merge the GUI and CLI consoles into a single binary which can work even if no GUI API is available, falling back on CLI mode. We will also have the console(s) act as a front-end for the toolchain, even downloading it for you in the background when needed. Though for that we need a proper asynchronous `call` function implementation. More news about this soon.
In the meantime, enjoy running Red consoles almost instantly from just a click on the Download page!

You cannot have missed that in the last months (and even last years), our overall progress has slowed down drastically. One of the main reasons is that we have spread our limited resources chasing different objectives while making little progress on the core language. That is not satisfying at all and would bring us most likely to a dead-end as we exhaust our funding. We have spent the last weeks discussing about how to change that. This is our updated action plan.

From now on, our only focus will be to finish the core language and bring it to the much-awaited version 1.0. We need to reach that point in order to kickstart a broader adoption and provide us and our users a stable and robust foundation upon which we can build commercial products and services necessary for sustainability.

Given the complexities involved in completing the language and bringing an implementation that can run on modern 64-bit platforms, we have devised a two-stage plan.

Upgrade the current 32-bit Red implementation
👉 Language specification

It is now time to do so in order to clean-up some semantic rules and address all possible edge cases which will help fulfill our goals of implementation robustness and stability. The process of writing down the complete language specs will result in dropping some features that we currently have that end up being problematic or inconsistent. OTOH, we might add some new features that will need to be implemented for 1.0.

👉 Modules

We need a proper module system in order to be scalable. We also need to have a proper package management system which will be tied to a central repo where we can gather third-party libraries. That would also enable modular/incremental compilation (or encapping) which will be most probably supported in the self-hosted toolchain.

👉 Concurrency

We need a proper model for concurrent execution in order to leverage multicore architectures. We will define one and make a prototype implementation in the 32-bit version.

👉 Toolchain

Before starting to work on the new toolchain, we will make some changes to the existing version in order to prepare for the transition. The biggest change is the dropping of the Red compiler, which will only act as a (smart) encapper. Routines and #system directives will still be supported, but probably with some restrictions. The Red preprocessor might also see some changes. This change means that Red will only have one execution model instead of the two it has currently. The Red compiler has become more of a burden than a help. The speed gains are not that significant in real code (even if they can be in some micro-benchmarks), but the impossibility for the compiler to support the exact same semantics as the interpreter is a bigger problem. This move not only will bring more stability by eliminating some edge case issues but also will reduce the toolchain by almost 25% in size, which will help reduce the number of features to support in the new toolchain.

👉 Runtime library

Some improvements are long overdue in the Red runtime library. Among them:

• unified Red evaluation stack.
• unified node! management.
• improved processing of path calls with refinements.
• improved object! semantics.

All those changes are meant to simplify, reduce the runtime library code and address some systemic issues (e.g. stack management issues and GC node leaks).

👉 Documentation

We need proper, exhaustive, user-oriented documentation for the Red core language. This is one of the mandatory tasks that needs to be completed and done well for wider adoption.

Self-hosted Red for 64-bit version
👉 Toolchain

In order to go 64-bit, we have to drop entirely our current toolchain code based on Rebol2 and rewrite it with a newer architecture in Red itself. The current toolchain code was disposable anyway, it was not meant to live this long, so this was a move we had to do for 1.0 anyway.

So the new toolchain will feature:

• a new compilation pipeline with a plugin model.
• an IR layer.
• one or more optimizing layers.
• modular/incremental compilation support.
• x64, AArch64 and WASM backends.
• linker support for 64-bit executable file formats for the big 3 OS.
• support for linking third-party static libraries.

32-bit backends will not be supported in 1.0, though, they could be added back in the future.

👉 Runtime library

The current Red runtime library written in R/S will be kept and some adjustments will be needed in order to be fully compatible with a 64-bit environment (like updating all imported OS API to their 64-bit versions). 
View engine will not be part of that upgrade for 1.0, but will be done in a 1.1 version, priority is given to Red/Core for the 1.0.

Roadmap

Here are the main milestones:

• v0.7   : Full I/O with async support.
• v1.0b : (beta) completed self-hosted Red with 64-bit support.
• v1.0r  : (release) first official stable and complete Red/Core language release.
• v1.1   : View 64-bit release.
• v1.2   : Android backend and toolchain release.
• v1.3   : Red/C3 release.
• v1.4   : Web backend for View release.
• v2.0   : Red JIT-compiler release.
• v3.0   : Red/...

The 0.7 should be the last version for the 32-bit Red version and current toolchain and we will be working on that first.

For reaching the 1.0-beta milestone, we target 12 months of intensive work, so that will bring us to Q3 2023. That's an ambitious goal but necessary to reach for the sake of Red's future.

The currently planned beta period for 1.0 is 2-3 months. We want a polished, rock-solid, production-ready 1.0 release.

For the 1.1, we will probably make some (needed) improvements to View engine architecture and backends.

For Red/C3, as the Ethereum network is transitioning to 2.0 and a new EVM, we need the WASM backend in order to support it.

Version 1.4 will bring a proper web runtime environment to the WASM backend, including GUI support.

The 2.0 will be focused on bringing a proper JIT-compiler to Red runtime, that should radically improve code execution of critical parts without having to drop to R/S.

Version 3.0 is already planned, but I will announce that once 1.0 will be released. ;-)

One major platform is missing from the above plan, that is iOS. Given how closed that platform is, we will need to come up with a specific plan on how to support it, as it won't be able to cross-compile for it (you would need a Mac computer), nor probably generate iOS apps without relying on Xcode at some point (not even mentioning dynamic code restrictions on the AppStore), which are layers of complexity that Red is trying to fight against in the first place... So for now, that platform is not among our priorities.

To finish, let me borrow some words from someone who succeeded more than anyone else in our industry:

Expect me to say "no" even more so from now on, as we get laser-focused on our primary goal.

Cheers and let's go!

Another quarter, another blog post. Seems almost rushed after the previous drought. 

To set the stage, I'll start with a bit of a rant about complexity. If you just want the meat of what's happening in the Red world, feel free to skip the introduction. 

Complexity Considerations: Part 1

I liked what the InfoWorld article, Complexity is Killing Software Developers said, which we all know, about difficult domains (voice and image recognition, etc.) being available as APIs. This lets us tackle things we couldn't in some cases. Though I imagine @dockimbel or others also used Dragon Dictate's libraries back in the 90s. What we have now is massive data to train systems like that. Those work well, allowing us to add features we otherwise couldn't with a small team.

The problem I see is that the trend has become for everything to be outsourced, including simple features like logging, and those libraries have exploded. There must be graphs available to show the change. Moderately complex domains, UIs for example, have risen in number and lead to what @hiiamboris says about Brownian Movement. It's a random collection of things, not designed to work together, without a coherent vision. A quote from the above article says it this way:

"Complexity is less the issue than inconsistency in an environment."

It used to be that you could take a FORTRAN, COBOL, Lisp, VB, Pascal/Delphi, Access/PowerBuilder, dBase/Clipper/Paradox, or even a Java developer, drop them into a project, and they could work from a solid core, learning the team's custom bits and any commercial tools as they went. With JS leading the way, but not alone in this, a programmer can only rely on a much smaller core, relative to how many libraries are used.

Because those libraries, and the choices to use a particular combination of them were not designed to work together, there is no guarantee (or perhaps hope) of consistency to leverage. It's worse if you came from a history of other tools that were based on different principles or priorities, because you have to unlearn, breaking the patterns in your mind. Or you convince people to use what you did before, even if there is overlap with tools already in use.

Things are changing now, and will even more. New service-based companies are coming, and a drive to APIs rather than libraries. So we not only have risks like LeftPad, but also companies going out of business under you. The modern trend means it's no longer dependent on an author or team committed to a project long term, but to what investors want, and what changes are made to gain adoption at all costs. As a service-based company you can't hold dearly to design principles if the investors tell you to pivot. Because it's no longer about your vision, but their return. If it is a solo FOSS author or small team, what is their incentive to maintain a project for free, while others profit from it? Success can be your worst enemy, and we need a more equitable solution than what we have now. The software business model has changed dramatically, and will likely continue to do so.

Here is what I personally see as the crux of the problem: the goal of scaling. FOSS projects and companies are only considered successful if they have millions (or, indirectly, billions) of users. Companies that want to be sustainable, providing long term, moderate profits don't make headlines, but they make the world go 'round. They are not the next big social media disruption where end users are the product, to be bought and sold. It is a popular business model and profit is the goal. It's nothing personal.

This has led us to the thinking that every project needs to be designed for millions of users at the very least. Sub-second telemetry for all the data collected, another explosion, giving rise to data analytics for everyone; not just Business Intelligence (BI) for large companies. I won't argue against having data. I love data and learning from it. But I do believe there is a point of diminishing returns which is often ignored. Rather, in this case, there is a cost of entry that small projects wouldn't otherwise need to pay.

What do you do, as an "architect" (see the previous blog post about my thoughts on software architecture) or developer on a team? Your small team (we all know small teams are best, plenty of research and history there) simply can't design and build every piece to support these scaling demands, while the sword of Damocles hangs over you in the form of potential pivots (dramatic changes in goals).

As an industry, we are being inexorably forced to make these choices. Either you're a leader and make your own Faustian bargain, or you're in the general mass of developers being whipped and driven to the gates of Hell.

Only you, dear reader, can decide the turns this tragic story will take, and what you forgive in this telling perchance I should exaggerate.

Complexity Considerations: Part 2

Complexity doesn't come only in the form McCabe is famous for, the decision points in a piece of code, but in how many pieces there are and how often they change either by choice or necessity. Temporal Complexity if you will. This concept is unrelated to algorithmic time complexity. Rebol2 for any faults we can point out, still works to this day (except in cases where the world changed out from under it, e.g., in protocols). It was self-contained, and relied only on what the OS (Operating System) provided. As long as OSs don't break a core set of functionality that tools rely on, things keep working. R2 had a full GUI system (non-native, which insulated it from changes there), and I can only smile when I run code that is 20 years old and it works flawlessly. If that sounds silly, remember that technology, in most cases, is not the goal. It is a means to an end. A lot of very old code is still in production, keeping businesses running.

We talk about needing to keep up with changes, but some things don't change very much, if at all. Other things change rapidly, but for no good reason, and without being an improvement. If a change is just a lateral move there is no value in it, unless it is to align us on a different, and better, path in the future. I started programming with QuickBASIC, but also used other tools as I quickly learned my tool of choice came with a stigma attached, and I wanted to be a serious, "real" programmer. What became clear was that QB was a great tool, with a few companies providing terrific ASM libraries, and had a wonderful IDE to boot. It was simpler, not only as a language, but because every 12-18 months (the release cycle way back when) my new C compiler would break something in my code. But QB, and later BASIC/PDS and then VB very rarely broke working code. Temporal complexity.

Even then there were more complex options. The cool kids used Zortech C++ and there were various cross-platform GUI toolkits. But those advanced tools were often misapplied to simple projects. We still do that today. Much of that is human nature, and the nature of programmers. If it's easy we are no longer special. We may not mean to, but we make things harder than they need to be. Some of us are even elitist about what we do, to our own detriment. If you don't need to be cross platform, why do you have multiple machines or VMs each with a different compiler setup? If you need a GUI, why are you using a language that was not designed with them in mind? If you need easy deployment, which is simpler: a single EXE with no dependencies, or a containerization approach with all that entails? How many technologies do you need in your web stack? Are you the victim of peer pressure, where you feel your site has to be shiny and "responsive", or use the latest framework?

A big argument for using other's work is performance. They've taken time, and may be experts, to optimize Thing X far beyond what you could ever do. That JIT compiler, an incredible virtual DOM, such clever CSS tricks, the key-value DB with no limits, and yet...and yet our software is slower and more bloated than ever. How can that be? Is it possible we're overbuilding? Is software sprawl just something we accept now?

Earlier I mentioned that a hodge-podge assembly of parts that have no standards, norms, or even aesthetic sense applied does not make our lives easier. Lego blocks, the originals anyway, are limited, but consistent in how they can be used. We misapply that analogy, because the things we build are far from consistent or designed to interact. Even in the realm of UX and A/B testing on subsets of users that companies apply today. I love the idea of data-driven HCI to guide us to a more evidence-oriented approach. This includes languages. But when a site or service moves fast and changes their interface based on their own A/B testing, they don't account for the others doing the same. Temporal complexity.

As a user, every app or site I access may change out from under me in the flash of refresh or automatic update I didn't ask for. Maybe it's better, an actual improvement, if you only use that one site. But if all your tools constantly change out from under you, it's like someone sneaking into your office and rearranging it every night while you sleep. Maybe this is the developer's revenge, for the pain we inflict on ourselves by constantly changing our own tools. If we suffer, why shouldn't our users? For those who truly have empathy for their users and don't want to drive them mad, or away, perhaps the lesson is to have empathy for ourselves, for our own tribe. I don't want to see my friends and colleagues burn out, when it was probably the enjoyment and passion that solving problems with software can bring which led them here to begin with.

Every moving part in your system is a potential point of failure. Reduce the moving parts and reliability increases. Whether it's the OS you run on (we now have more of those than ever, between Linux distros and mobile platforms always trying to outdo each other), extra packages or commercial tools, FOSS libraries, environments, [?]aaS, or platform components like containers and cluster management, every single piece is a point of failure. And if any of them break your code, or your system, even in the name of improvements or bug fixes, you may find yourself running just to stay in the same place. Many of those pieces are touted as the solution to reliability problems, but a lot of them just push problems around, or target problems you don't have. Don't solve problems you don't have. That adds complexity, and now you really have a problem.

Less Philosophy, More Red
Interpreter Events
Having a debugger in Red has been a request of many users for a long time, even since the Rebol era. We have tackled this feature from a larger perspective, considering general instrumentation of the interpreter (note: not the compiler), extending it with an event system and user-provided event handlers, similar to how parse and lexer tracing operate today. This approach allows us to build more than just a debugger, though it was a lot of work to design and we expect it will be refined once people start using it in earnest. It's a brave new world, with a lot of tooling possibilities.
It's important to note that this is not magic. Because it operates as the interpreter evaluates values and expressions, including functions, it can't see into the future. In order to get a complete trace, you have to evaluate everything. That means we'll see tools which silently collect data, like a profiler does, which can later be viewed and analyzed, perhaps up to the point where an error occurred. This is an important aspect, and plays once again into the power of Red as data. Your event handlers can easily collect data into any structure or model you like. And because event handlers can filter events, you can tailor them for specific needs. It should even be possible to build interpreter level DTrace-like tools in the future. We also hope to build higher level observability and monitoring tools, based on eventing systems, in the future, but those are long term projects.
Event generation is not active by default, it is enabled using do/trace and by providing an event handler function. For example, here's a simple logging function:

  logger: function [
      event  [word!]                      ;-- Event name
      code   [any-block! none!]           ;-- Currently evaluated block
      offset [integer!]                   ;-- Offset in evaluated block
      value  [any-type!]                  ;-- Value currently processed
      ref    [any-type!]                  ;-- Reference of current call
      frame  [pair!]                      ;-- Stack frame start/top positions
  ][
      print [
          pad uppercase form event 8
          mold/part/flat either any-function? :value [:ref][:value] 20
      ]
  ]

Given this code:

  do/trace [print 1 + 2] :logger

It will output:

  INIT    none                    ;-- Initializing tracing mode
  ENTER   none                    ;-- Entering block to evaluate
  FETCH   print                   ;-- Fetching and evaluating `print` value
  OPEN    print                   ;-- Results in opening a new call stack frame
  FETCH   +                       ;-- Fetching and evaluating `+` infix operator
  OPEN    +                       ;-- Results in opening a new call stack frame
  FETCH   1                       ;-- Fetching left operand `1`
  PUSH    1                       ;-- Pushing integer! value `1` on stack
  FETCH   2                       ;-- Fetching and evaluating right operand
  PUSH    2                       ;-- Pushing integer! value `2`
  CALL    +                       ;-- Calling `+` operator
  RETURN  3                       ;-- Returning the resulting value
  CALL    print                   ;-- Calling `print`
  3                               ;-- Outputting 3
  RETURN  unset                   ;-- Returning the resulting value
  EXIT    none                    ;-- Exiting evaluated block
  END     none                    ;-- Ending tracing mode

Several tools are now provided in the Red runtime library, built on top of this event system:

• An interactive debugger console, with many capabilities (step by step evaluation, a flexible breakpoint system, and call stack visualisation).
• A simple profiler that we will improve over time (especially on the accuracy aspects).
• A simple tracer. The current evaluation steps are quite low-level, but @hiiamboris has already built an extended version, operating at the expression level that will soon be integrated into the master branch.

Full docs are here.

Format
Boy, I really thought this was was going to be easy, or at least not too hard. I couldn't have been more wrong. When I did my format experiments, I imagined at least some of the code would be useful, requiring polish and more work of course, providing a foundation to work from. It turns out that I missed a key aspect, and my approach was just one of many possible. @hiiamboris and @giesse both weighed in, and we chatted about specific parts. Then it sat idle for a while, and I asked Boris to take it over to get it into production. He identified the key missing piece, which would have limited its usefulness until we eventually had to address it. Better now than later. He also made a strong case for a different approach to the core masked-number and I told him to run with it. That led to a lot of design chat about one aspect, which is as yet undecided. It's not a fight to the death, but there has definitely been some sparring. :^)
The missing piece I've alluded to is Localization (L10N). As an American who has never had to develop software requiring Internationalization (I18N), I've been blissfully ignorant of all the aspects that come into play when Globalization (G11N) becomes part of the process. We have talked about how to implement L10N in Red, and have system/locale for a months, weekdays, and currency codes. The first two we inherited from Rebol's design, the latter was added when @9214 designed the currency! datatype. Thinking of locale data in a system catalog of some kind is easy enough, but how to actually apply it (and not apply it when necessary) is a different story entirely. And I mean entirely. Format forced us to start down this path, and is a guinea pig feature that will guide future plans for all future L10N work. But keep my complexity rant in mind. While we want to make it as easy as possible for Reducers to write globally aware apps, if you don't need it, don't do it. We don't yet know if we can make it so magical that you can write your app ignoring that for the most part, and then flip a switch, or simply include local data, and have it work. Don't get your hopes up. There's a lot that can go wrong with that approach.
We agreed to start with masked numbers but, in order to do that, L10N R&D had to be done. This led to broad and deep dives into unicode.org and other resources. While they cover far more than we need, and is overly complex in many cases (or just doesn't match our aesthetic sense for Red), the data they have there is enormously valuable, and we deeply appreciate it being available. We just draw the line around a smaller scope than they do, and no committees are involved where people fight to get their own bits included. Well, we do that too, to some extent. What Boris managed to do was identify the key elements needed for our work, and then wrote tools (using Red of course) to extract and reformat the data for use in Red. I can't stress how much work this was. Truly a heroic and mostly thankless effort most people will never know about.
In order to test masked number formatting, and give others an easy way to play, Boris created a Playground App and I can't tell you how important that was. You see, a particular piece of behavior came up while I was playing with it and got unexpected results. Unexpected to me, but Boris confirmed it was by design. I will just say here that it's about a significant digits mode, and let you play with the app from there. Named formats will be available, but everything will likely boil down to wrappers around masks, which should cover almost any need.
Next up is date formatting. This time I knew locales would play a role because some IETF RFCs specify that date elements be in English. So you may have localized dates for some things, but if you use RFC2822 dates or HTTP cookie dates, they must not be affected by any locale settings. Dates will use masks at the core, like numbers, because masks are an easy to understand WYSIWYG format. Well, easy if the masks make sense. If you look at printf and some other mask syntax, it can be quite obscure. By trying to cram things into a limited syntax, people end up using whatever low ASCII letters might be left over for some elements. We hope to avoid that. 
Our main choices are what Boris termed the stuttering format. e.g. MMDDYYYY/HHMMSS. Think in terms of "progressing in a hesitant or irregular way." rather than stuttering in terms of human speech. I prefer to call this a symbolic format, where the letters map to date elements. This, of course, isn't perfect. e.g. is MM month or minute? Context is required. We don't want to be case sensitive, or use other letters randomly to avoid that conflict. So there's an alternate approach; a literal mask. e.g. 1-Jan-2022. We're not the first to consider it, and it is in use elsewhere, but it's not a perfect solution either. Do masks have to be written in English terms, or can they use any locale? How do you disambiguate numbers (does 01-01 mean MM-DD or DD-MM, and how do you write that without the separator to get MMDD?) Does it make code more or less readable, because Red already has a literal date form, and it would add what look like literal dates as strings in code.
Play with the app, give us feedback, and stay tuned. We think this will be a crucial feature for a lot of users, and we want to make it the best it can be.
Split
Like format, split seems a relatively simple subject at a glance. And if you limit it to basic functionality, it is. That's what other languages do, though some add a few extra features. See this table for examples. Wolfram appears quite broad in scope, because there are multiple variants for each named function. Something else common to all other languages is that they split only strings and sometimes byte arrays. In Red we have blocks, and while `parse` is great for string parsing, where it really shines is when applied to blocks to build dialects. We knew split should be block aware for more leverage. I (Gregg) helped design the version in R3, and used DiaGrammar to design a new dialected interface that aimed to extend the functionality. Wanting to do more evidence based language design, I also prototyped a small practice/playground app, thinking we'd put it out and see what kind of feedback we could get. 
Toomas stepped up and suggested an alternative, refinement-based, interface. He did a number of versions of that, and then we had to decide what to do next. There was a great deal of design discussion, still going on, about behavior details. Once you start adding options, it's easy for things to become confusing for the user. We need to strike a balance between ease of use and flexibility. Split is meant to handle the most common cases, and those with the most leverage, not every case. And while a refinement-based interface seems natural for Reducers, we also know how readable parse, draw, and VID dialects are. There are pros and cons to each, but we don't want dual interfaces, which will be confusing. If a function is dialected, any refinements should work in support of that dialect. So the test app was reimagined by @GalenIvanov to compare the two approaches.
Here's a screenshot of the test app, which we'll release to the community in January.

We learned by doing this that it's hard to compare them side by side, without having the user write full calls directly. That defeats some of the purpose, and the DRY principle, so we'll put this one out, then revise it based on feedback.
Markup Codec
Who knew that parsing HTML and XML would be the easy part? Well, many Reducers would. What they, and we on the team, might not have guessed, is just how hard it is to decide on a data format for the output. Red gives us many options, and XML gives us many headaches. The two formats, while closely related, also have some critical differences. Fortunately, once @rebolek set things up so we could play, and made the emitter modular, we could look at real examples and dive even deeper. What we discovered is that there is no perfect solution. No elegant model to fit all uses and cases. Key to many insights was @dander's input, as he works with XML a lot. Turns out, an infinitely extensible format is infinitely challenging to nail down.
Should we emphasize path access? Being data driven, people probably shouldn't hardcode their field names, but working with known data makes it a clear access model. Should attributes come before or after the text/content for a tag? As we learned, attributes aren't always small, so the locality argument isn't won either way there. Is it better to provide an interface to the structure and tell people to always use that, or to create a bland and obvious data structure that is possible to access in many ways? Will these things all complicate HOF access, which we know we want to leverage? How much do we need to care about efficiency? We don't want to be wasteful without purpose, but if we're too miserly, users may pay the price because it's harder to use. If we make more things implicit, do we paint ourselves into a corner somehow?
What we settled on was a modular approach, so there will be more than one standard emitter. What is yet undecided is how other emitters might be supported. They will likely be quite custom, as the standard versions will cover most needs. But is it worth making the system extensible? Once you have a result, it's easy to post-process into your preferred format. For now that's our recommended approach.
CLI Module
If you don't follow our channels on Gitter, you may not know about Boris' CLI module. It's very slick, very Reddish, and will become a standard part of Red in the near future. You won't believe how easy it is to create rich command line interfaces for your Red apps with this feature. Huge thanks to @hiiamboris for all his innovation and work on it.
IPv6 Datatype
It hasn't been merged to the mainline yet, but it's fully operational. You can see the code here, and some lexer tests here. You may be impressed that it's only a couple hundred lines of code, not counting the lexer changes, and think it was easy. It wasn't. As usual, there was a lot of design chat and compromise involved. For example, the name is not 100% finalized because, technically, the datatype itself is more generally applicable, being simply a vector of numbers internally. You can think of it like a tuple! on steroids. Less slots (8 vs 12), but each slot can hold a larger value (tuple! slots are limited to byte values).
Just as tuple! is a general name, used both for IPv4 addresses and colors, but also useful for other things, IPv6! could be used for things like GUIDs or extended time values. But the lexical form for GUID/UUID values is quite different, even ignoring the shortcut forms in the IPv6 specification. As you probably know, lexical space is tight in Red, and the colon is an important character in other places, and URL lexical forms were impacted, so this is a deep change and commitment, in that regard. Why do it then?
Because IPv6 networking support was already in place in Red, and IPv6 is the future. How often people will write literal URLs like http://[FEDC:BA98:7654:3210:FEDC:BA98:7654:3210]:80/index.html we can't say. But we do know that addresses often end up in config files as data and that modern, dynamic systems generate addresses dynamically. They will appear in log files, messages, and more. As with the value of other lexical forms in Red, it's an important one that is part of our modern networking vocabulary.
Getting Near
@dockimbel created a new branch here, which will interest almost every Reducer. It's not ready yet, but expect it to be available in January. For those who used R2, you may recall that errors gave you a Near field, to hint at where the error occurred. Red will get this feature when the new branch is merged. e.g., in Red today you get this:
    >> 1 / 0     *** Math Error: attempt to divide by zero     *** Where: /     *** Stack:  
Where in R2 you got this:
    >> 1 / 0     ** Math Error: Attempt to divide by zero     ** Near: 1 / 0
A little extra information goes a long way. We're anxious to see all the virtual smiles this features brings.
The Daily Grind
We closed roughly 120 tickets in 2021, that's 10 per month. We also merged almost 50 PRs. These numbers don't sound large, but when you consider how much time and effort may go into the deep ones, along with all the other work done, it's steady progress. We'd love for both tickets and pending PRs to be at zero, but that's not practical for a project like Red. The deep core team must have uninterrupted time for design and bigger, more complex tasks.
Roadmap
Q4 2021 (retrospective)

• We hoped to have `format` and `split` deployed, but they will push back to Jan-2022.
• `CLI` module approved, needs to be merged, then refined as necessary.
• `Markup Codec` took longer than expected due to extensive design chat on formats.
• Interpreter instrumentation, with PoC debugger and profiler. Took longer than expected, but are out now.
• Async I/O, out but some extra bits didn't make it in. One unplanned addition was `IPv6!` as a datatype. It's experimental, and subject to change.
• @galenivanov did some great work on his animation dialect, but @toomasv's `diagram` dialect took a back seat and will move to Q1 2022.
• Audio has 3 working back ends and a basic port implementation. Next up is higher level design, device and format enumeration, and device control. A `port!` may not be the way to go for all this, but it was step one.
• Animation has more great examples all the time. Like this and this. @GalenIvanov is doing great work, and we are planning to make his dialect a standard addition to Red.

2022
I'm not going to list items in any particular order, because our plans often change. This way you have things to look forward to, but still with an element of surprise.

• `Table` module, `node!` datatype and other REP reviews
• Full HTTP/S protocol and basic web server framework
• New DiaGrammar release
• Animation dialect
• New release process
• New web sites updated and live
• Red/C3 (Including ETH 2.0 client protocol)
• Red Language Specification (Principles, Core Language, Evaluation Rules, Datatype Specs (including literal forms), Action/Native specs, Modules spec.
• 64-bit support (LLVM was a possibility, but we learned from Zig that LLVM breaking changes can be quite painful for small teams to keep up with. We may be better off continuing to roll our own, though it's a big task.)
• Android update
• Red Spaces cross-platform GUI
• Module and package system design
• RAPIDE (Rapid API Development Environment)

RAPIDE, from Redlake Technologies
If you've used Postman or Insomnia, you know what the most popular tools in the API IDE space look like today. If you haven't used them, but use APIs, they're worth a look. For all that those tools do, and there are a few other players in the space, there is a lot they don't do. We think we can add a lot of value in the API arena, thanks to Red's superpowers and how important data-centric thinking is. For example, testing a group or series of APIs together seems like it could be greatly improved. Also, how APIs are found, and collaboration possibilities.
While we haven't set a release date, the plan is to start work on RAPIDE in Q2 2022, after we wrap up some infrastructure pieces it will rely on. 

In conclusion
Happy New Year to all, and may 2022 see us all healthy, happy, and writing more Red. :^)

 It's been almost a year since our last blog post. Sorry about that. It's one of those things that falls off our radar without a person dedicated to it, and we run lean so don't have anyone filling that role right now. We know it's important, even if we have many other channels where people can get information. So here we are.

Last year was a tough year all around, even for us. We were already a remote-only team, but the effect the pandemic had on the world, particularly travel, hit us too. We had some team changes, and also split our focus into product development alongside core Red Language development. This is necessary for sustainability, because people don't pay for programming languages, and they don't pay for Open Source software. There's no need to comment on the exceptions to these cases, because they are exceptions. The commercial goal, starting out, is to focus on our core strengths and knowledge, building developer-centric tools. Our first product, DiaGrammar for Windows, was released in December 2020, and we've issued a number of updates to it since then. Our thanks to Toomasv for his ingenuity and dedication in creating DiaGrammar. We are a team, but he really accepted ownership of the project and took it from an idea to a great product. Truly, there is nothing else like it on the market. 

We learned a lot from the process of creating a product, and will apply that experience moving forward. An important lesson is that the product itself is only half the work. As technologists, we're used to writing the code and maybe writing some docs to go with it. We don't think about outreach, marketing, payments, support, upgrade processes for users, web site issues, announcements, and more. The first time you do something is the hardest, and we're excited to improve and learn more as we update DiaGrammar and work on our next product. We'll probably announce what it will be in Q4. One thing we can say right now is that the work on DiaGrammar led to a huge amount of work on a more general diagramming subsystem for Red. It's really exciting, and we'll talk more about that in a future blog post.

So what have we been doing?

Since our last blog post we've logged over 400 fixes and 100 features into Red itself. Some of these are small, but important, others are headline-worthy; some are deep voodoo and some visible to every Reducer (what we call Red users). For example, most people use the console (the REPL), so the fixes and improvements there are easy to see. A prime feature being that the GUI console, but not the CLI console, didn't show output if the UI couldn't process events. This could happen if you printed output in a tight loop. The results would only show up at the termination of the loop, when the system could breathe again. That's been addressed, but wasn't easy and still isn't perfect. Red is still single threaded, so there's no separate UI thread (pros and cons there). We make these tradeoffs every day, and need feedback from users and real world scenarios to help find the right balance. Less obvious are things like improvements to parse, which not everyone uses. Or how fmod works across platforms, and edge cases for lexical forms (e.g. is -1.#NaN valid?). The latter is particularly important, because Red is a data language first.

JSON is widely used, but people may not notice that the JSON decoder is 20x faster now, unless they're dealing with extremely large JSON datasets. JSON is so widely used that we felt the time spent, and the tradeoffs made, were worth it. It also nicely shows one of Red's strengths. Profiling showed that the codec spent a lot of time in its unescape function. @hiiamboris rewrote that as a Red/System routine, tweaked it, and got a massive speedup. No external compiler needed, no need to use C, and the code is inlined so it's all in context. Should your JSON be malformed, you'll also get nicer error information now. As always, Red gives you options. Use high level Red as much as possible, for the most concise and flexible code, but drop into Red/System when it makes sense.

Some features cross the boundary of what's visible. A huge amount of work went into D2D support on Windows. D2D is Direct2D, the hardware-accelerated backend for vector graphics on Windows. For users, nothing should change as all the details are hidden. But the rendering behavior is not exactly the same. We try to work around that, but sometimes users have to make adjustments as well; we know because DiaGrammar is written in Red and uses the draw dialect heavily. It's an important step forward, but comes at a cost. GDI+ is now a legacy graphics back end, and won't see regular updates. Time marches on and we need to look forward. As if @qtxie wasn't busy enough with that, he and @dockimbel also pushed Full I/O forward in a big way. It hasn't been merged into the main branch yet, but we expect that to happen soon. @rebolek has been banging on it, and has a full working HTTP protocol ready to go, which is great. TLS/SSL support gets an A+ rating, which is also a testament to the design and implementation. It's important to note that the new I/O system is a low level interface. The higher level API is still being designed. At the highest level, these details will all be hidden from users. You'll continue to use read, write, save, load exactly as you do today, unless you need async I/O. 

Another big "feature" came from @vazub: NetBSD support. The core team has to focus on what stands to help the project overall, with regard to users and visibility. Community support for lesser known platforms is key. If you're on one of those platforms, be (or find) a champion. We'll help all we can, but that's what Open Source is for. Thanks for this contribution @vazub!

We also have some new Python primers up, thanks to @GalenIvanov. Start at Coming-to-Red-from-Python. Information like this is enormously important. Red is quite different from other languages, and learning any new language can be hard. We're used to a set of functionality and behaviors, which sometimes makes the syntax the easiest part to learn. Just knowing what things are called is a learning curve. Red doesn't use the same names, because we (and Carl when he designed Rebol) took a more holistic view. That's a hard sell though. We feel the pain. A user who found Red posted a video as they tried to do some basic things. We learned a lot from watching it. Where other languages required you to import a networking library, it's already built into Red. When they were looking for request or http.get, and expecting strings to be used for URLs, they couldn't find answers. In Red you just read http://.... It's obvious to us, but not to the rest of the world. So these new primers are very exciting. We have reference docs, and Red by Example, but still haven't written a User's Guide for Red. We'll get there though. 

Why do things take so long?

Even with that many fixes and features logged, and huge amounts of R&D, it can still feel like progress is slow. The world moves fast, and software projects are often judged by their velocity. We even judge ourselves that way, and have to be reminded to stay the course, our course, rather than imitating others. Red's flexibility also comes into play. Where other languages may limit how you can express solutions, we don't. It's so flexible that people can do crazy things or perform advanced tricks which end up being logged as bugs and wishes. Sometimes we say No (a lot of times in fact), but we also try to keep an open mind. We have to ask "Should that be allowed?", "Why would you want to do that (even though I never have)?", and "What are the long term consequences?" We have to acknowledge that Red is a data format first, and we never want to break that. It has to evolve, but not breaking the format is fundamental. And while code is expected to change, once people depend on a function or library it causes them pain if we break compatibility. We don't want to do that, though sometimes we will for the greater good and the long view. There are technical bandages we can patch over things, but it's a big issue that doesn't have a single solution. Not just for us, but for all software development. We'll talk more about this in the future as well.

I'll note some internal projects related to our "slow and steady" process:

• Composite is a simple function that does for strings what compose does for blocks. It's a basic interpolator. But the design has taken many turns. Not just in the possible notations, but whether it should be a mezzanine function, a macro, or both. Each has pros and cons (Side note: we don't often think about "cons" being an abbreviation for "consequences"). This simple design and discussion is stalled again, because another option would be a new literal form for interpolated strings. That's what other languages do, but is it a good fit for Red? We belabor the point of how tight lexical space is already, so have to weigh that against the value of a concise notation.
• Non-native GUI. Red's native GUI system was chosen in response to Rebol's choice to go non-native. Unfortunately it's another case of needing both. Being cross platform is great for Red users, but Hell for us. Throw in mobile and it's even worse. Don't even talk about running in the browser. But every platform has native widget limitations. Once you move beyond static text, editable fields, buttons, and simple lists, you're in the realm of "never the twain shall meet". How do you define and interact with grids and tables or collapsible trees? Red already has its own rich-text widget, so you don't have to embed (even if you could) an entire web browser and then write in HTML and CSS. To address all this, with much research and extensive use case outlines, @hiiamboris has spent a lot of time and effort on Red Spaces. Show me native widgets that can do editable spiral text, put any layout inside a rotator, or define recursive UIs. I didn't think so. Oh, and the wiggling you see in the GIFs there are not mistakes or artifacts, they are tests to show that any piece of the UI can be animated.
• Other projects include format, split, HOFs, and modules, each with a great deal of design work and thought put into them. As an example, look at Boris' HOF analysis. They are large and important pieces, based on historical and contemporary research, but not something we will just drop into Red, though we could. A simple map function is a no-brainer, and could have been there day one. But that's not how we work. It's not a contest to see how many features we can add, or how fast; but how we can move software forward, make things easier, and push the state of the art. Not just in technical features (the engineering part), but in the design of a language and its ecosystem.

Not Everyone Has These ProblemsAn important aspect of Red is being self-contained. We talk about this a lot. Yes, we're considering LLVM as a target, but that has a big cost, not just benefits. Using our own compiler for everything also has costs, like slowing the move to 64-bit which is an issue for Mac users now. Workarounds like VMs and Docker containers are just that. We want things to be easy for you, but that doesn't mean they're easy for us. Here's an example.
Boris found a bug related to printing time values in Red/System. @dockimbel finally tracked it down, and posted this investigative report:

@hiiamboris It was a (R/S) compiler issue afterall. ;-)size? a was the guilty part. The compiler was wrongly generating code for loading a even though size? is statically evaluated by the compiler and replaced by a static integer value. Given that a was a float type, its value was pushed onto the x87 FPU stack, but never popped. That stack has a 7 slots limit. Running the loop 5 times was enough to leave only 2 slots free. When the big float expression is encountered in dtoa library, it requires 3 free slots on the FPU stack, which fails and results in producing a NaN value, which wreaks havoc in the rest of the code.

The fix in the compiler was trivial (fetch-expression/final vs fetch-expression) but getting to that point was not. Understanding machine architectures at the lowest levels isn't for everyone, but even though our compiler code will be rewritten in the future, it's small and maintainable today. If we rely on GCC, Clang, or other compilers, hitting a bug may mean hitting a wall. So while there are costs to using our own compiler, there are also costs to depending on others. Robert Heinlein popularized  TANSTAAFL, but the concept is not science fiction. As a side note, just as we moved from GDI+ to D2D,  x87 for float support was an early choice meant to support older platforms and we are planning to switch to SSE.If compilers are your thing, or you like system level programming, join our community and get to know Red/System. See how our toolchain works, and consider joining us.
The Big PictureI just read 101 things I learned in architecture school (which I heard about via Kevlin Henney, (though it may not have been that specific talk) and what struck me the most about it is how we've commandeered the word "architecture" for software but completely removed the human aspect. An architect does so much more than we do. Software architects are really structural engineers. If a single developer builds a complete app, they have to do the UI. They engineer a building and then slap on whatever sheathing is at hand, cutting doors and windows without concern for their location. And the app is viewed in isolation, as if it's the only thing a user has on their system, without consideration for its site, context, or relationships. What makes real architecture hard (and why the author notes that many architects hit their stride when they are older), is that you have to know so much. So many considerations, disciplines, and constraints are involved, and you have to unify them. It's both creative and scientific. What makes great architecture great is that it makes your experience better. Maybe even wonderful. If we only think about the mechanical aspects, our software will never be beautiful.
We haven't articulated this view for what we do, I think because we didn't realize it. At least I didn't. We talk about the whole being greater than the sum of its parts, and not just making everything libraries so it feels more natural and less mechanical. How a REPL and single exe make it easier to get started, and not having to use many tools is better. But we haven't explicitly said "Here's how it's laid out, and why. Here's how it's put together; these are the critical elements. Here's what it looks like from a distance, and when you enter its space." Implicitly we do that every day, through the work, but we don't talk about it. Or only once a year.

In the past months, many new features were added to Red/System, the low-level dialect embedded in Red. Here is a sum up if you missed them.

Subroutines

During the work on the low-level parts of the new Red lexer, the need arised for intra-function factorization abilities to keep the lexer code as DRY as possible. Subroutines were introduced to solve that. They act as the GOSUB directive from Basic language. They are defined as a separate block of code inside a function's body and are called like regular functions (but without any arguments). So they are much lighter and faster than real function calls and require just one slot of stack space to store the return address. 

The declaration syntax is straightforward:

    <name>: [<body>]

    <name> : subroutine's name (local variable).
    <body> : subroutine's code (regular R/S code).

To define a subroutine, you need to declare a local variable with the subroutine! datatype, then set that variable to a block of code. You can then invoke the subroutine by calling its name from anywhere in the function body (but after the subroutine own definition).

Here is a first example of a fictive function processing I/O events:

    process: func [buf [byte-ptr!] event [integer!] return: [integer!]
        /local log do-error [subroutine!]
    ][
        log: [print-line [">>" tab e "<<"]]
        do-error: [print-line ["** Error:" e] return 1]
    
        switch event [
            EVT_OPEN  [e: "OPEN"  log unless connect buf [do-error]]
            EVT_READ  [e: "READ"  log unless receive buf [do-error]]
            EVT_WRITE [e: "WRITE" log unless send buf    [do-error]]
            EVT_CLOSE [e: "CLOSE" log unless close buf   [do-error]]
            default   [e: "<unknown>" do-error]
        ]
        0
    ]

This second example is more complete. It shows how subroutines can be combined and how values can be returned from a subroutine:

    #enum modes! [
    	CONV_UPPER
    	CONV_LOWER
    	CONV_INVERT
    ]

    convert: func [mode [modes!] text [c-string!] return: [c-string!]
        /local
            lower? upper? alpha? do-conv [subroutine!]
            delta [integer!]
            s     [c-string!]
            c     [byte!]
    ][
        lower?:  [all [#"a" <= c c <= #"z"]]
        upper?:  [all [#"A" <= c c <= #"Z"]]
        alpha?:  [any [lower? upper?]]
        do-conv: [s/1: s/1 + delta]
        delta:   0
        s:       text

        while [s/1 <> null-byte][
            c: s/1
            if alpha? [
                switch mode [
                    CONV_UPPER  [if lower? [delta: -32 do-conv]]
                    CONV_LOWER  [if upper? [delta: 32 do-conv]]
                    CONV_INVERT [delta: either upper? [32][-32] do-conv]
                    default     [assert false]
                ]
            ]
            s: s + 1
        ]
        text
    ]
    
    probe convert CONV_UPPER "Hello World!"
    probe convert CONV_LOWER "There ARE 123 Dogs."
    probe convert CONV_INVERT "This SHOULD be INVERTED!"

will output:

    HELLO WORLD!
    there are 123 dogs.
    tHIS should BE inverted!

Support for getting a subroutine address and dispatching dynamically on it is planned to be added in the future (something akin computed GOTO). More examples of subroutines can be found in the new lexer code, like in the load-date function.

New system intrinsics
Several new extensions to the system path have been added.
Lock-free atomic intrinsics
A simple low-level OS threads wrapper API has been added internally to the Red runtime as preliminary work on supporting parts of IO concurrency and parallel processing in the future. In order to complement it, a set of atomic intrinsics were added to enable the implementation of lock-free and wait-free algorithms in a multithreaded execution context.
The new atomic intrinsics are all documented here. Here is a quick overview:

• system/atomic/fence: generates a read/write data memory barrier.
• system/atomic/load: thread-safe atomic read from a given memory location.
• system/atomic/store: thread-safe atomic write to a given memory location.
• system/atomic/cas: thread-safe atomic compare&swap to a given memory location.
• system/atomic/<math-op>: thread-safe atomic math or bitwise operation to a given memory location (add, sub, or, xor, and).

Other new intrinsics

• system/stack/allocate/zero: allocates a storage space on stack and zero-fill it.
• system/stack/push-all: saves all registers to stack.
• system/stack/pop-all: restores all registers from stack.
• system/fpu/status: retrieves the FPU exception bits status as a 32-bit integer.

Improved literal arrays

The main change is the removal of the hidden size inside the /0 index slot. The size of a literal array can now only be retrieved using the size? keyword, which is resolved at compile time (rather than run-time for /0 index access).

A notable addition is the support for binary arrays. Those arrays can be used to store byte-oriented tables or embed arbitray binary data into the source code. For example:

    table: #{0042FA0100CAFE00AA}
    probe size? table                      ;-- outputs 9
    probe table/2                          ;-- outputs "B"
    probe as integer! table/2              ;-- outputs 66

The new Red lexer code uses them extensively.
Variables and arguments grouping

It is now possible to group the type declaration for local variables and function arguments. For example:

    foo: func [
        src dst    [byte-ptr!]
        mode delta [integer!]
        return:    [integer!]
        /local
            p q buf  [byte-ptr!]
            s1 s2 s3 [c-string!]
    ]
 

Note that the compiler supports those features through code expansion at compile time, so that error reports could show each argument or variable having its own type declaration.

Integer division handling

Integer division handling at low-level has notorious shortcomings with different handling for each edge case depending on the hardware platform. Intel IA-32 architecture tends to handle those cases in  a slightly safer way, while ARM architecture produces erroneous results silently typically for the following two cases:

• division by zero
• division overflow (-2147483648 / -1)

IA-32 CPU will generate an exception, while ARM ones will return invalid results (respectively 0 and -2147483648). This makes it difficult to produce code that will behave the same on both architectures when integer divisions are used. In order to reduce this gap, R/S compiler will now generate extra code to detect those cases for ARM targets and raise a runtime exception. Such extra checkings for ARM are produced only in debug compilation mode. In release mode, priority is given to performance, no runtime exception will occur in such cases on ARM (as the overhead is significant). So, be sure to check your code on ARM platform thoroughly in debug mode before releasing it. This is not a perfect solution, but at least, it makes it possible to detect those cases through testing in debug mode.

Others

Here is a list of other changes and fixes in no particular order:

• Cross-referenced aliased fields in structs defined in same context are now allowed. Example:

      a!: alias struct! [next [b!] prev [b!]]
      b!: alias struct! [a [a!] c [integer!]]
  

• -0.0 special float literal is now supported.
• +1.#INF is also now supported as valid literal in addition to 1.#INF for positive infinite.
• Context-aware get-words resolution.
• New #inline directive to inline assembled binary code.
• Dropped support for % and // operators on float types, as they were relying on FPU's relative support, the results were not reliable across platforms. Use fmod function instead from now on.
• Added --show-func-map compilation option: when used, it will output a map of R/S function addresses/names, to ease low-level debugging.
• FIX: issue #4102: ASSERT false doesn't work.
• FIX: issue #4038: cast integer to float32 in math expression gives wrong result.
• FIX: byte! to integer! conversion not happening in some cases. Example: i: as-integer (p/1 - #"0")
• FIX: compiler state not fully cleaned up after premature termination. This affects multiple compilation jobs done in the same Rebol2 session, resulting in weird compilation errors.
• FIX: issue #4414: round-trip pointer casting returns an incorrect result in some cases.
• FIX: literal arrays containing true/false words could corrupt the array. Example: a: ["hello" true "world" false]
• FIX: improved error report on bad declare argument.

A programming language lexer is the part in charge of converting textual code representation into a structured memory representation. In Red, it is accomplished by the load function, which calls the lower-level transcode native. Until now, Red was relying on a lexer entirely written using the Parse dialect. Though, the parsing rules were constructed to be easily maintained and not for performance. Rewriting those rules to speed them up could have been possible, but rewriting the lexer entirely in Red/System would give the ultimate performance. It might not matter for most user scripts, but given that Red is also a data format, we need a solution for fast (near-instant) loading of huge quantities of Red values stored in files or transferred through the network.
The new lexer main features are:

• High performance, typically 50 to 200 times faster than the older one.
• New scanning features: identify values and their datatypes without loading them.
• Instrumentation: customize the lexer's behavior at will using an event-oriented API.

The reference documentation is available there. This new lexer is available in Red's auto-builds since June.

Performance
Vastly increased performance is the main driver for this new lexer. Here is a little benchmark to let you appreciate how far it gets.
The benchmarking tasks are:

• 100 x compiler.r: loads 100 times compiler.r source file from memory (~126KB, so about ~12MB in total).
• 1M short integers: loads a string of 1 million `1` separated by a space.
• 1M long integers: loads a string of 1 million `123456789` separated by a space.
• 1M dates: loads a string of 1 million `26/12/2019/10:18:25` separated by a space.
• 1M characters: loads a string of 1 million `#"A"` separated by a space.
• 1M escaped characters: loads a string of 1 million `#"^(1234)"` separated by a space.
• 1M words: loads a string of 1 million `random "abcdefghijk"` separated by a space.
• 100K words: loads a string of 100 thousands `random "abcdefghijk"` separated by a space.

And the results are (on a Core i7-4790K):

    Loading Task             v0.6.4 (sec)    Current (sec)    Gain factor
    ---------------------------------------------------------------------
			
    100 x compiler.r	      41.871            0.463	           90
    1M short integers	      14.295            0.071	          201
    1M long integers	      18.105            0.159	          114
    1M dates	              29.319	        0.389	           75
    1M characters             14.865            0.092             162
    1M escaped characters     14.909	        0.120             124
    1M words	                 n/a	        1.216	          n/a
    100K words	              23.183	        0.070	          331

Notes: 
- Only transcode is used in the loading tasks (system/lexer/transcode in 0.6.4).
- The "1M words" task fails on 0.6.4 as the symbol table expansion time is exponential due to some hashtable bugs. That also explains the big gap for the "100K words" task. Those issues are fixed in the current version and the symbol table further optimized for speed. Though, the execution time increase between 100K and 1M words tests in new lexer is not linear which may be explained by a high number of collisions in the internal hashtable due to limited input variability.
- The 0.6.4's lexer can only process strings as input, while the new lexer only processes internally only UTF-8 binary inputs. The input strings were converted to the lexer's native format in order to more accurately compare their speed. Providing a string instead of a binary series as input to the new lexer incurs on average a ~10% speed penalty.
Scanning
It is now possible to only scan tokens instead of loading them. Basically, that means identifying a token's length and type without loading it (so without requiring extra memory and processing time). This is achieved by using the new scan native.

    >> scan "123"
    == integer!
    >> scan "w:"
    == set-word!
    >> scan "user@domain.com"
    == email!
    >> scan "123a"
    == error!

It is possible to achieve even higher scanning speed by giving up a bit on accuracy. That is the purpose of the scan/fast refinement. It trades maximum performance for type recognition accuracy. You can find the list of "guessed" types in the table there.

    >> scan/fast "123"
    == integer!
    >> scan/fast "a:"
    == word!
    >> scan/fast "a/b"
    == path!

Scanning applies to the first token in the input series. When an iterative application is needed in order to scan all tokens from a given input, the /next refinement can be used for that. It will return the input series past the current token allowing to get the precise token size in the input string. It can be used in combination with /fast if required. For example:

    src: "hello 123 world 456.789"
    until [
        probe first src: scan/next src
        empty? src: src/2
    ]

Outputs:

    word!
    integer!
    word!
    float!

Matching by datatype in Parse
The new lexer enables also matching by datatype directly from Parse dialect. Though, this feature is limited to binary input only.

    >> parse to-binary "Hello 2020 World!" [word! integer! word!]
    == true
    >> parse to-binary "My IP is 192.168.0.1" [3 word! copy ip tuple!]
    == true
    >> ip
    == #{203139322E3136382E302E31}
    >> load ip
    == 192.168.0.1

Notice that the whitespaces in front of tokens are skipped automatically in this matching mode.
Instrumentation
Lexers in Red and Rebol world used to be black boxes, this is no longer the case with Red's new lexer and its tracing capabilities. It is now possible to provide a callback function that will be called upon lexer events triggered while parsing tokens. It gives deeper control to users, for example allowing to:

• Trace the behavior of the lexer for debugging or statistical purposes.
• Catch errors and resume loading by skipping invalid data.
• On-the-fly input transformation (to remove/alter some non-loadable parts).
• Extend the lexer with new lexical forms.
• Process serialized Red data without having to fully load the input.
• Extract line comments that would be lost otherwise.

Lexer's tracing mode is activated by using the /trace refinement on transcode. The syntax is:

    transcode/trace <input> <callback>

    <input>    : series to load (binary! string!).
    <callback> : a callback function to process lexer events (function!).

That function is called on specific events generated by the lexer: prescan, scan, load, open, close, error. The callback function and events specification can be found there. 
A default tracing callback is provided in system/lexer/tracer:

    >> transcode/trace "hello 123" :system/lexer/tracer
    
    prescan word 1x6 1 " 123"
    scan word 1x6 1 " 123"
    load word hello 1 " 123"
    prescan integer 7x10 1 ""
    scan integer 7x10 1 ""
    load integer 123 1 ""
    == [hello 123]

That tracing function will simply print the lexer event info. If a syntax error occurs, it will cancel it and resume on the next character after the error position.
Several more sophisticated examples can be found on our red/code repository.

Implementation notes
This new lexer has been specifically prototyped and designed for performance. It relies on a token-oriented pipelined approach consisting of 3 stages: prescanning, scanning and loading.
Prescanning is achieved using only a tight loop and a state machine (FSM). The loop reads UTF-8 encoded input characters one byte at a time. Each byte is identified as part of a lexical class. The lexical class is then used to transition from one state to another in the FSM, using a big transition table. Once a terminal state (state names with a `T_` prefix) or input's end is reached, the loop exits, leading to the next stage. The result of the prescanning stage is to locate a token begin/end positions and give a pretty accurate guess about the token's datatype. It can also detect some syntax errors if the FSM cannot reach a proper datatype terminal state. This approach provides the fastest possible speed for tokens detection, but it cannot be fully accurate, nor can it validate deeply the token content for some complex types (e.g. dates). 
Adding more states would provide greater accuracy and cover more syntatic forms, but at the cost of growing the transition table a lot due to the need to duplicate many state. Currently the table weights 2440 bytes, which is already quite big to be kept entirely in the CPU data cache (usually 8, 16 or 32KB per core, the lexical table uses 1024 bytes and there two other minor tables used in the tight loop). The data cache also needs to handle the parsed input data and part of the native stack, so the available space is limited.
The tight loop code is also optimized for keeping branch mispredictions as low as possible. It currently only relies on two branchings. The loop code could be also further reduced by, for example, pre-multiplying the state values to avoid the multiplication when calculating the table entry offset. Though, we need to wait for a fully optimizing code generation backend before trying to extract more performance from that loop code, or we might be taking wrong directions.
Scanning stage happens when a token has been identified. It consists in eventually calling a scanner function to deep-check the token for errors and more accurately determine the datatype. Loading stage then follows (unless only scanning was requested by the user). It will eventually call a loader function that will construct the Red value out of the token. In case of any-block series, the scanners will actually do the series construction on reaching the ending delimiter (which requires special handling for paths), so no loader is needed there. Conversely, loaders can be invoked in validating mode only (not constructing the value), in order to avoid code duplication when complex code is required for decoding/validating the token (e.g. date!, time!, strings with UTF-8 decoding,...).
For the record, there was an attempt at creating specific FSM for date! and time! literal forms parsing, to reduce the amount of rules that need to be handled by pure code. The results were not conclusive, as the amount of code required for special case handling was still significant and the performance of the FSM parsing loop was below the current pure code version. This approach can be reexamined once we get the fully optimizing backend.
The FSM states, lexical classes and transitions are documented in lexer-states.txt file. A simple syntax is used to describe the transitions and possible branching from one state to others. The FSM has three possible entry points: S_START, S_PATH and S_M_STRING. Parsing path items requires specific states even for common types. For curly-braced strings, it is necessary to exit the FSM on each occurrence of open/close curly braces in order to count the nested ones and accurately determine where it ends. In both those path and string cases, the FSM needs to be re-entered in a different state than S_START.
In order to build the FSM transition table, there is a workflow that goes from that lexer-states.txt file to the final transition table data in binary. It basically looks like this:

    FSM graph -> Excel file -> CSV file -> binary table

The more detailed steps are:

1. Manually edit changes in the lexer-states.txt file.
2. Port those changes into the lexer.xlsx file by properly setting the transition values.
3. Save that Excel table in CSV format as lexer.csv.
4. Run the generate-lexer-table.red script from Red repo root folder. The lexer-transitions.reds file is regenerated.

The lexer code relies on several other tables for specific types handling like path ending detection, floating point numbers syntax validation, binary series and escaped characters decoding. Those tables are either manually written (not planned to be ever changed) or generated using this script.
Various other points worth mentioning:
- The lexer works natively with UTF-8 encoded binary buffers as input. If a string! is provided as input, there is an overhead for converting internally such string to binary before passing it to the lexer. A unique internal buffer is used for those conversions with support for recursive calls.
- The lexer uses a single accumulative cells buffer for storing loaded values, with an inlined any-block stack.
- The lexer and lexer callbacks are fully recursive and GC-compliant. Currently callbacks can be function! only, this can be extended in the future to support routines also for much faster processing.

It's been a busy start to the year for the team. Work has continued on many fronts, and we have some new team members helping us to keep the momentum going. We announced in January that GTK and the Fast Lexer were close, and they are even closer now. The hard part about making announcements is that some of this work is unpredictable and changes in scope after we do. Or the world steps in and a pandemic throws a wrench into your plans.

@bitbegin has done an enormous amount of work on the GTK branch. You can check out the GTK branch to see some of what goes into supporting a new GUI system. The more features we include in Red, the more that have to be ported and maintained. Unfortunately, most operating systems and UI systems have large, complicated sets of APIs and interactions.

Because GUI systems are so complex, and Red not only has to handle them, but also adds its own reactive framework, there are more places for bugs to hide. And users are involved, which is the worst part. They do nothing but cause problems. To that end, in addition to his normal deep diving and bug hunting, @hiiamboris has been working on an automated view test system, which is no small feat. @9214 has joined him on the hunt, and we will squash a good number of bugs for our next release.

The fast lexer was in near-final testing when we decided that it was worth delaying its merge in order to incorporate some new lexical forms that we had planned to include. Then we looked at some old tickets related to modulo and division operators, and a couple lexing questions came up related to tag!. Suddenly the fast lexer work was back in code mode. New lexical forms usually means new datatypes, and that's the case here.

New Datatypes One we've expected for some time, and thanks to @9214 it's now a reality. Money! is coming. There is a branch for it, but no need to comment at this time. There are a few features, like round still to be completed, but the bulk of the work is done. @BeardPower did some great experimental work that we thought might be used for money, based on Douglas Crockford's Dec64 design, but until Red is fully 64-bit it was only Dec32, and the limited range was determined not to be enough. That work won't go to waste though, it's just waiting for its moment in the sun. The current version of money! is a BCD implementation, but that shouldn't concern anyone outside the core team. What you care about is that it can be used for accurate financial calculations that don't suffer from floating point errors. It will also support an optional currency identifier, e.g. USD or EUR, and automatic group separators when formed.

Another new type is called ref! and is still being designed. The basic concept is simple: @reference is a form most people are familiar with today, just as hashtags are known (though we use the historical name issue!, because the new name hadn't become part of our global lexicon back then). Issues, in Rebol 2, were a string type, but Red made them a word type instead. That has benefits (mainly efficiency), but also costs (symbol table space and lexical limitations). For instance, in R2 you could say #abc:123, but not so in Red. Life is compromise. Ref! will be a string type, making it quite flexible. While we most often think of them as referring to a person, they can refer to, for example, a location in a file. You can do that with strings too, of course, but that's the beauty of rich datatypes. By using a ref!, you can build rich dialects and make your intent clear in the data itself.

Finally, Red is going to add a new raw-string! form. It's a combination of the raw string literals some languages support, and heredoc. The goal is to make it easier to include content that would otherwise require escape sequences that sometimes clutter inline data and lead to errors. A lot of time and effort went into the (sometimes heated) discussion around the need, use cases, and syntax. Right now this is just a new literal form, rather than a whole new datatype. They will still be strings when loaded, until we see how they are used by others and if they deserve to be a separate type.

We don't add new datatypes lightly, and design choices have to keep the big picture in mind as well. Balancing the value of new types against their added complexity in the language is hard work, but satisfying if it makes everyone's life better. To that end, when these new types and features become available, we need your input on how they work, where you use them, and what gaps need to be filled.

Both the fast lexer and money datatype leverage some new features in Red/System, which we'll talk about in a future post.

All Aboard! We're also very excited to announce the imminent release of our first commercial product. We alluded to it at the beginning of the year, and it's almost ready to leave the station. There will be plenty of train-related puns, because it's a Railroad syntax diagram generator. Here's what it looks like:
More details will come soon, but here's a short list of some of it's features:

• Live coded diagramming. You write your grammar and the diagram appears in real time.
• Red parse support (of course), but also support for foreign grammars like ABNF, McKeeman, and YACC. That's right, you can write using those grammars and get the same diagram. 
• When writing parse rules, you can use block level parsing, rather than character level.
• Test inputs. Not only can you test a single input, but also entire directories of test files.
• Test specific rules. Not only can you test from your top-level rule, but any rule in your grammar. You can also find where a particular rule matches part of your input.
• Custom diagram styles, including options for a cleaner, more abstract view, different charset rendering options, and whether actions (parens) are displayed.
• The ability to generate inputs that your grammar will recognize. This opens up many other use cases, including educational ones. Here's one of my favorites, that @toomasv created:

Look at the sentences in the input area. Those were created by clicking the generate button. It's like parsing in reverse. So if you're not sure what inputs your grammar might recognize, or want to see examples, this lets you view things in a whole new way.

Stay Tuned It's an exciting time to be a Reducer, and we're rolling with changes just like everyone else right now. The best way to stay healthy during this pandemic is to stay away from other people and take the Red pill.
Until next time.

Hello and happy new year, friends of Red! We have some exciting projects we’ve been working on that will be available this year, including a new product. Let’s talk a little about what the team has been working on behind the scenes.

(TL;DR: A cool new product with Red in 2020...plus, a robust preliminary draft of Parse documentation can now be previewed...CLI library...fast-lexer to merge soon...GTK on the horizon...and a new native OS calendar widget!)





Documentation for Parse: Red’s Language Construction Tool
Our esteemed forerunner, Rebol, broke new ground with its Parse dialect, which Red has expanded on. Today, in Red, Parse has become an even more powerful built-in dialect (embedded domain-specific language) that processes input series with grammar rules in a clean and simple manner. Other language building tools exist, of course, such as Lexx and Yacc. ANTLR is a modern framework built in Java, and libraries exist for other languages, but the ease of use and power that Red’s Parse offers is unique.

This isn’t a new feature, by any means, with the first public introduction here. Parse is easy enough to use that those basic docs have been enough, for the most part. But over time, with new users joining the Parse chat room to discuss the dialect, and it was time for full reference documentation.

For the uninitiated, Parse can be used to for searching, to surface various patterns; validation, in order to confirm an input’s compliance to a specification; extraction, to sift through data and aggregate values; and modification--that is, changing the input stream itself (insertion of values, removing or transforming matched input). And Parse’s true power lies in language processing (compilers, interpreters, and lexical analyzers), particularly for DSLs; and encoding/decoding, to “translate” data formats from one to another.

What makes Red’s Parse dialect a killer feature is that you can do more than parse at the character level in strings of text. That's what every other tool out there does, at best. But with Parse, you can process blocks of Red values, based on their datatype or specific literal matches, letting Red do all the work applying these layered or nested levels of analysis. By parsing blocks of values rather than character strings, we take language construction to the meta level.

We’re happy to announce that @9214 has completed a robust preliminary draft of Parse documentation, which you can see here. Internal reviews by the core team are in process and public comment will be made possible after more analysis. In the meantime, you can view Parse itself here, and look at open issues there if you’re interested in contributing.

(As a final note, the first question people often ask is how Parse compares to regular expressions. Both are DSLs, to be sure, but regex is designed for flat pattern matching of text inputs. Formats with nesting, things like matching parentheses, etc. are difficult or impossible. Engines differ widely, as well (NFA vs DFA), so your code may have very different performance in different environments. Not all regex syntaxes are compatible either. But the real difference comes in readability. Parse is more verbose, but is very clear, readable, and composable. Regex patterns are not. For fun, look up the regex to match email addresses, written by Jeffrey Friedl, in his book Mastering Regular Expressions. It’s a 6.5Kib regex. Doesn’t seem to be online anymore, but is referenced in this chat. For more discussion on this specific area, https://emailregex.com/ points to https://www.regular-expressions.info/email.html about why there’s no perfect regex for email addresses. The point is not to criticize regular expressions. They are useful, especially in the more limited forms of wildcard matching and file globbing. But they can’t recognize all grammars, and quickly become impossible for a human to read and reason about.)

Fast-Lexer To Be Merged Soon
Over the last few months work on the fast-lexer branch of Red has accelerated, beginning with @dockimbel prototyping the next lexer in R/S, faster than Red’s predecessors, with much smaller and simpler code, and more speed and features to come. Issues with the current lexer had brought the need for a revision: this included the goal of making the lexer instrumentable. The requirements of a callback function with the current non-re-entrant lexer could have caused serious problems.

Another hangup? An analysis of all Red's repos showed that the current lexer code was duplicated twice, beyond the basic lexing needed by load: once in the console code, once again in the VSCode plugin. The redundancy had to be eliminated. A merge of the updated lexer into master should be expected early in 2020.

New Year, New Projects! 
In 2020, Red’s commercial arm will release its first product! It will be helpful to CS programs and their students, researchers, and anyone interested in analyzing the grammar and syntax of languages—which is all of you, since you're here. But it is meant to have much broader appeal as well.

Every developer has probably see json.org, which brought railroad syntax diagrams to everyone’s attention. That’s right, syntax is for data formats too. But why would Douglas Crockford go to all the trouble of creating those diagrams? Communication. Visualizations create a different experience for readers, facilitating new understanding and discussion options. This applies to those designing rich data structures (think GraphQL) and APIs, as well as languages. If you want to look professional, foster adoption, show that your design is formally specified, and help people understand it, this tool is for you.

It does a lot more than create fantastic diagrams, too, to help spec and test your grammar. It’s in invite-only beta test now, but we’ll need a limited number of testers soon, so contact us at pr@red-lang.org if you’re interested.

Some of you may ask: Why is it a commercial product? Because we’re a multifaceted organization, and Open Source doesn’t pay. ;^) The Red Foundation supports the open source work done on the Red language, as well as sponsoring other work that promotes Red’s use and adoption. They’ve been mainly focused on supporting Team Red and core language work, but will expand their reach as resources allow. The first step in that direction was to get behind proving Red’s commercial viability. Redlake Technologies is a new business entity that will use Red to build tools the world needs, and will pay for, with the goal of creating a self-sustaining ecosystem of applications, services, and tokenomics that benefit both the contributors to the language and the world at large, working toward a better model of sustainable open source development.

By Your Command (Line)
Also in the works: prolific community member @hiiamboris has been burning the midnight oil creating a fantastic CLI library, which will enable authors to accept and interpret arguments and define their types easily, in an expectable, standardized manner. Command lines are the API through which a program exposes its facilities to the outside world--to users and utilities. If you’d like to check it out, look at the repo and see how it works! By leveraging Red’s own function spec DSL we can, with this CLI library, turn any Red function into a command line utility with one or two words.

Suppose you have a `grep` function written in Red, that is to be used from the command line. Prefix it with `process-into` (the working CLI function name), add `print` for the output, compile it, and you have a standalone `grep` utility! Your feedback is important, because this design for a CLI library is quite unique.

As with Red itself, you may need to let go of old ideas about how CLI libraries like `get-opt` work (though you could recreate its model easily with what Boris has built), and think like a Reducer. Just write a function with parameters, refinements, types, and doc strings, and it does the rest.

GTK on the Horizon 
The GTK back end is nearing completion, with just GUI console support as the last big piece to be done. @bitbegin has done an enormous amount of work, with help, consultation, and testing from @loziniak and @rcqls. Once the GUI console is in place, GTK should have feature parity with MacOS.

Just In Time… 
The New Year will see one more surprise, in the form of a calendar widget with support for Windows and Mac. It’ll just show up one day--a native OS widget joining you in the latest automatic build--so keep your eyes peeled.

Welcome to December, friends of Red. It's an intent and focused time of year as we wind down 2019, and the core team is making important moves (sometimes literally!) to set us up for an ambitious 2020. But first, here are just a few things that happened in November.

• First and foremost, it's always great when community members help compile resources for use by others, and we'd like to acknowledge @rebolek for his excellent compendium of historic automatic builds: https://rebolek.com/builds/ (they weren't available for a hot minute, but now they're back). They can be useful if you're in need of a previous version for a specific project. Of course, you can always go here for Red's daily automated builds. But seeing as how we've a goal of being a self-sustaining, self-selecting group of do-ers, this spirit of providing collective resources is perfectly aligned with the Red-Lang we always want to be. 

• From the community, to outreach: thanks to community member @loziniak, Red is now on Stackshare, so be sure to follow us there and chime in. As a repo with 4.1k GitHub stars (and infinite possibilities), Red has a lot to offer the wider community of developers and engineers, and Stackshare is a great place to help compare and contrast us with other languages. 

• Now, a challenge! In the coming new year we'll be needing beta testers willing to lend their expertise in refining a new product built with Red. You read that right! If you think you'd like to be one of the contributors to spearhead a move into our next phase, we want YOU! Drop a line to @greggirwin to get in on the ground floor.
  
• An appreciation to @hiiamboris for his deeply thought out proposal regarding "series evolution," a framework for standardizing and testing the functions we use in Red for manipulating series. Design is hard, and we have a number of initiatives in the works taking a lot of brain power right now.
  
• Over 60 commits were made to Red's GTK branch in November, making it almost ready for "prime time." The product of a lot of work by a fair handful of the core team, heavy lifter @bitbegin says that merging to master branch is possibly the next step.
  
• And, to close, a bit of a tease: Watch this space for some snazzy website changes coming soon.

All the warmest wishes for the upcoming holidays, from the Red family to yours. <3

-Lucinda.

The original commentary was posted in Red's Gitter channel, here, by Gregg Irwin, one of our core team members, in response to various requests for the ability to create new datatypes in Red.

As a writer, Red has always appealed to me because of its flexibility; but, of course, "the [lexicon] devil is in the details," as the idiom goes (okay, I edited that idiom a little, but it was too cool a link to pass up). It means the more specific we try to be, the more challenges and limitations we encounter, and we can lose some of the amazing versatility of the language. On the other hand, precision and refinement--the "exact right word at the exact right time," can powerfully enhance a language's utility. The dynamic tension between what he calls "generality and specificity, human friendliness and artifice," in the text below, can be an energetic ebb and flow that serves to strengthen our language, to make it more robust. 

Two quotes from community members provide some context:
_____________________________
> The real problem is not number of datatypes, but the lexical syntax of the new ones. -@Oldes
> ...However if something like utype! is added, nothing prevents you from (ab)using system/lexer/pre-load and reinventing whole syntax. -@Rebolek

"I don't support abusing system/lexer/pre-load, and (in the long view) there will almost certainly be special cases where a new lexical form makes sense. We can't see the future, so we can't rule it out. But, and this is key, how much value does each new one add?

I believe that each new lexical form adds less value, and there is a point of diminishing returns. This is not just a lexical problem for Red, but for humans. We have limited capacity to remember rules, and a constrained hierarchy helps enormously here. Think more like linguists, and less like programmers or mathematicians.

In language we have words and numbers. Numbers can be represented as words, with their notation being a handy shortcut for use in the domain of mathematics. And while we classify nouns, verbs, and adjectives by their use, they are all words, and don't have syntax specific to their particular part of speech. That's important because a single word may be used in more than one context, for more than one purpose.

This is interesting, as a tangent, because human language can be ambiguous, though some synthetic languages try to eliminate that (e.g. Lojban). The funny thing is that it's almost impossible to write poetry or tell jokes in Lojban. Nobodyº speaks Lojban. This ties to programming because, while we all know the strengths and value of strict typing, and even more extreme features and designs meant to promote correctness, dynamic languages are used more at higher levels [such as poetry, songwriting and humor, where even the sounds used in one single word can be employed to evoke specific emotive responses in the listener--the effects of devices like assonance, consonance, and loose associations we make with even single letters, in the way a repeated letter R throughout a line of poetry or literature can subtly impart a sense of momentum and intensity to the text...possibly because it evokes a growl... -Ed.]. Why is that? Humans.

When Carl designed Rebol, it had a goal, and a place in time. He had to choose just how far to go. Even what to call things like email!, which are very specific to a particular type of technology. This is what gives Redbol langs so much of their power. They were designed as a data format, meant for exchanging information. That's the core. What are the "things" we need to exchange information about with other humans, not just other programmers?

Do I want new types? I'm pushing for at least one: Ref! with an @some-name-here syntax. It's not username! or filename+line-number!, or specific in any way. It's very general, as lexical types should be; their use and meaning being context-specific (the R in Redbol, which stands for "relative"). I also think ~ could be a leading numeric sigil to denote approximation. It came mainly from wanting a syntax for floats, to make it clear that they are imprecise; but it's tricky, because it could also be much richer, and has to take variables into account. ~.1 is easy, but what about x = ~n+/-5%? Units are also high value, but they are just a combination of words and numbers. (Still maybe worth a lexical form.)

When we look at what Red should support, and the best way to let users fulfill application and purpose-specific needs, we can learn from the past, and also see that there is no single right answer. Structs, Maps, Objects, data structures and functions versus OOP, strict vs dynamic.

As Forth was all about "Build a vocabulary and write your program in that," think about what constitutes a vocabulary; a lexicon. It's a balance, in Red, between generality and specificity, human friendliness and artifice. So when we ask for things, myself and Nenad included, we should first try to answer our need with what is in Red today, and see where our proposed solution falls on the line of diminishing returns. To this end, we can and should abuse system/lexer/pre-load for experimentation."

What made the fast-lexer branch a priority?
Several things. It started when @dockimbel looked into ticket #3606, which was impossible to fix currently, and we didn't want to give up on the auto-syncing between /text and /data facets. So he had to consider bigger options, including how to make the lexer instrumentable. It was not easy, because the current lexer is not re-entrant, so having the lexer emit events to a callback function could have caused serious problems.
Digging through all Red's repos showed that the current lexer code was duplicated twice, beyond the basic lexing needed by load: once in the console code, once in the VSCode plugin, each time for syntax coloring purposes, and each one lagging behind the original implementation. Not good.
@Dockimbel then considered changing the current lexer to make it instrumentable, but the changes were significant and would have made the parse rules much more complex. At the same time, @qtxie did some benchmarking, and the result showed Red's lexer was ~240 times slower than Rebol's. This is not due to parse, but  rather because the high-level rules were optimized for readability, not performance.
The lexer also caused delays in the VSCode plugin, because of its (lack of) performance. The high level code has served Red well, and was a showcase for parse, but loading larger data is also being used by community members, and data sizes will just keep growing. With some projects we have on the horizon, the lexer's performance became a higher priority.
As planned since the beginning (the lexer used to be R/S-only during the pre-Unicode era), @dockimbel decided the best option was to not postpone the conversion of the lexer to pure R/S code any longer, by porting R3's C-based lexer to R/S. After studying Rebol's lexer in detail, he realized that the code was quite complex in some places (mostly the prescanner), and would lead to less than optimal R/S code that would be hard to maintain.
Evaluating the state of the art in fast parsers for programming languages, he found inspiration in some unpublished papers. He then started prototyping the next lexer in R/S, and realized that it could be several times faster than Rebol's, with the additional benefit of much smaller and simpler code. Then he embarked on the full implementation. Knowing he and @qtxie would not have the opportunity to work on that for probably a year with all the big tasks ahead on the roadmap, he committed to it full time.
Red's new R/S lexer is half the size of Rebol's, far simpler, with more maintainable code, and it performs at similar speeds (sometimes a bit faster, sometimes a bit slower). That is a fantastic result, because it means that with an optimizing backend (Red/Pro), our lexer will be 4-8 times faster than R3's. It should then be possible to load gigabytes of Red data in memory in just a few seconds (using the future 64-bit version). 😉
An additional benefit was brought by @qtxie, who added a hashtable for symbol lookup in Red contexts. That sped up word loading tremendously, and should have a measurable improvement on Red's start up time; especially on slow platforms like Raspberry Pi.
@Dockimbel is almost done with the lexer itself, just date! and time! to add, and it should be possible to replace the old one with the new one after thorough testing and debugging. Then, we'll add the hooks for a user-provided callback, allowing us to instrument the lexer in ways Redbolers could only dream about until now. One application of that will be the ability to implement "predictive loading," which will tell you the type and size of a Red value in a string, without loading it, and at extremely high speed (~370MB/s currently, 1-2GB/s with /Pro). Such a feature will allow us to finally address the #3606 issue with a very clean and efficient solution, while keeping the facet's auto-syncing feature.

Over the last few weeks the Red Lang core team drilled down to make some truly great progress on Red's fast-lexer branch--while we also gained valuable support from the contributions of Red doers and makers as they consolidate a world of useful information and resources.


Fast-Lexer Benchmarks
In the fast-lexer branch of Red, you can see lots of new work from Red creator @dockimbel (Nenad Rakocevic) and core teammate @qxtie. Among other fixes and optimizations, they substituted a hashtable for what had previously been a large array in context!

The numbers so far: Loading 100'000 words (5 to 15 characters, 1MB file): Red (master): 19000ms.  Red (fast-lexer): 150ms. Nenad's observations on further testing:

"FYI, we just [ran] some simple benchmarks on the new low-level lexer for Red using 1M 10-digit integers. The new lexer completes the loading about 100 times faster than the current high-level one. Loading 1M 10-digit integers in one block: Red: 175ms; R2: 136ms; R3: 113ms. 

"We use a faster method than Rebol, relying on several lookup tables and a big FSM with pre-calculated transition table (while Rebol relies on a lot of code for scanning, with many branches, so bad for modern CPU with branch predictions). With an optimizing backend, Red's LOAD should in theory run 2-3 times faster than Rebol's one. (Though, we still need to optimize the symbol table loading in order to reach peak performance).  Given that Rebol relies on optimized C code while Red relies on sub-optimal code from R/S compiler, that speaks volume about the efficiency of our own approach. So, Red/Pro should give us a much faster LOAD.

"The lexer is not finished yet, but the hard part is done. We still need to figure out an efficient way to load keywords, like escaped character names (`^(line), ^(page), ...) and month nouns in dates."

This is a huge accomplishment, and it's shaping up to make future goals even more impressive. The fast-lexer branch is a work in progress, but stay tuned: Nenad has more to say about why it's been prioritized just now, which we will have in an upcoming post.


Red's MVPs Contribute New Resource Material & Tools
If you're new to Red, sometimes the flexibility of the language can leave you uncertain about which aggregate structure to use. In red/red's wiki on github, @9214 contributes a useful guide for those seeking to tease apart the differences. For example, map! works better with data that can be atomized, or framed as a conventional associative array, while hash! lends itself to data that will be queried at a high volume and which will require fewer updates. Learn further linguistic nuances, including object! and block!, as well as a useful comparison table of their algorithmic complexity, here. @Rebolek, meanwhile, has furnished us with loads of useful information, diving deeper into code statistics. His value datatype distribution, here, his unofficial Red build archive here, and his rebolek/red-tools repo containing various tools--line parsers, codecs, APIs and documentation among them--are greatly appreciated. The tools repo has a number of new features you can check out here.


About Those Ports...
Wondering about port!? Here's the latest. We've got port! in the master branch already, but low-level input/output networking abilities aren't complete yet, so we need to focus on this, and your feedback can always help. "We have a working async TCP and TLS ports implementation (both client and server-side)," explains Nenad, "but they still require more work to cover all the target platforms." Here, he goes on to explain the prerequisites for our team to complete this process; your thoughts and code contributions are welcomed.


Games and Experiments
It's a fun one to end this update on: Red community member @GalenIvanov's "Island Alleys," a game of unspooling Hamiltonian paths! A path of this type only allows its line, which inscribes a closed loop, to cross through a vertex within a graph once, a process which can lend itself to neural network-related interpretations. And @planetsizedcpu offers a wintry little spin on this repo. Enjoy, and thanks to all!

Hello to all the great makers, doers and creative people who are using Red, helping the Red Language grow and improve! As always, there's a standing invitation for you to join us on Gitter, Telegram or Github (if you haven't already) to ask questions and tell us about your Red-powered projects.

Here are some recent highlights we’d like to share with you:

1. Tickets Get Priority In the last month, our core team has closed a large number of tickets.We’d like to thank community members rgchris, giesse, and dumblob who are just a few of the passionate contributors putting Red through its paces and providing feedback as fixes and changes occur. @WArP ran the numbers for us, showing a cyclical growth pattern linking bursts of closed issues and some serious Red progress, and September’s not even done yet!...:


2. CSV Codec Available Our newly updated CSV codec has been merged in the master branch and is now a part of the nightly (or automatic) build here. It is in an experimental phase, and we want your feedback.

Should the standard codec only support block results, so it’s as simple as possible? Or do people want and need record and column formats as well (using the load-csv/to-csv helper funcs, rather than load/as)? Including those features as standard means they’re always available, rather than moving them to an extended CSV module; but the downside is added size to the base Red binary.

Applause goes to @rebolek’s excellent organization and his wiki on the codec, which explains the various ways in which Red can represent data matrices. He writes, “Choosing the right format depends on a lot of circumstances, for example, memory usage - column store is more efficient if you have more rows than columns. The bigger the difference, the more efficient.”

You can judge their efficiency here, where @rebolek has laid out the compile time, size and speed of each version, including encapping and lite. Be sure to get the latest build, and chat with everyone on Gitter to tell us what you think.

3. Red has reached 4K stars on GitHub! We’re truly grateful for all the interest and support, and we are proud of the way our growth has been powered by this community.

4. AI + Red Lang Stack: Precision Tuning With Local OR Web-Based Datasets In conversation with @ameridroid:
“Presently, it seems like most AI systems available today either allow building an AI from scratch using low level code (difficult and time-consuming), *OR* using a pre-built AI system that doesn't allow any fine-tuning or low-level programming...with the advent of NPUs (Neural Processing Units) akin to CPUs and GPUs, an AI toolkit would allow specifying what type of AI we want to perform (facial, image or speech recognition, generic neural net for specialized AI functions, etc.), the training data (images, audio, etc.) and then allow us to send it the input data stream and receive an output data stream…[using Red] would also allow us to integrate with the AI system at a low level if we have specific needs not addressed by the higher-level funcs. Red dialects would be a good way to define the AI functionality that's desired (a lot like VID does for graphics), but also allow the AI components, like the learning dataset or output data stream sanitization routines, to be fine-tuned via functions. Red can already work on web-based data using 'read or 'load, or work on local data in the same way; the learning data for a particular AI task could be located on the web or on the local machine. That's not easily possible with a lot of the AI solutions available today.”

Check back in the next few days for an update from @dockimbel!

Ideas, contributions, feedback? Leave a comment here, or c’mon over and join our conversation on Telegram, Gitter, or Github.

As part of the R&D work on port! datatype in port-type branch, we have implemented a GPIO driver for Raspberry Pi boards, as a gpio:// scheme (no third-party library needed). This work helped define the low-level API for ports written in purely Red/System or a mix of Red and Red/System.

The Raspberry Pi is a very popular board with millions of units sold, so this is a market where Red could be potentially helpful to developers. We could run Red on such boards for years but did not have proper GPIO support, so this is now fixed!

The current features supported by the GPIO port are:

• auto-detecting the Raspberry Pi board type
• uses /dev/mem or/dev/gpiomem for direct and fast access.
• reading a GPIO pin state.
• writing to a GPIO pin state.
• hardware PWM output support (on capable pins).
• a simple DSL for sending commands.

Planned (but not scheduled) future features include:

• generating events when a pin state changes.
• software PWM on all GPIO pins.
• a higher-level reactive object layer for a API-less interface.
• various drivers for common peripherals.

The source code for the gpio:// scheme can be found there.

In the short video below, you can find a little pet project meant for testing some of the features. It implements a simple joypad with 4 directions support, a red LED indicating when the pad is active and a green one for notifying when a level is completed. The game is the Red port contributed by Huang Yongzhao of Rebox!, my old clone of BoxWorld written in Rebol. In the video below, it is running on a Raspberry Pi 3 using our work-in-progress red/GTK backend for Red (contributed by Rcqls), locally merged with the red/port-type development branch (EDIT: that branch has been merged into master now).




You can find the breadboard layout below made using Fritzing. Sorry for the messy wiring, it is my first try with such kind of tool. If you have the skills to improve it, here is the sketch file. The buttons rely on the internal pull-down resistors. Note that in the video, the board is reversed.



The source code of Redbox has been modified to add GPIO support, you can find the modified code here. The GPIO-related code is enclosed in a context:

    joypad: context [
        mapping: [
            16 down  20 up
            17 left  21 right
        ]

        row: [(id) state: #[false] direction: (direction)]
        table: collect [
            foreach [id direction] mapping [keep compose row]
        ]

        pins: extract table length? row
        port: none

        acquire: has [pin][
            port: open gpio://
            foreach pin pins [
                insert port [
                    set-mode  pin in
                    pull-down pin
                ]
            ]

            insert port [
                set-mode 18 out
                set-mode  4 out
                set 4 on
            ]
        ]

        pressed?: function [][
            foreach pin pins [
                entry: find table pin
                old: entry/state

                insert port [get pin]
                entry/state: make logic! port/data

                ;-- detect 0-to-1 transitions only, to avoid auto-firing
                if all [not old entry/state][return entry/direction]
            ]
            none
        ]

        show-win: does [insert port [set 18 on]]

        release: does [
            insert port [
                set  4 off
                set 18 off
            ]
            close port
        ]
    ]

The GPIO port has a simple API:

• opening: port: open gpio://
• sending commands: insert port [...commands...]
• closing: close port

The sent commands form a small DSL:

Set the working mode for a given pin:
 
    set-mode <pin> <mode>

    <pin> : pin number (integer!)
    <mode>: in, out, pwm

Write a value on a pin: 
 
    set <pin> <value>
  
    <pin>  : pin number (integer!)
    <value>: true, false, on, off, yes, no, 0, 1

Read a value from a pin (the returned value is in port/data):

    get <pin>

    <pin>: pin number (integer!)    

Manage pull-up/down resistors:

   pull-off <pin>    ;-- disable any pull previously set
   pull-down <pin>   ;-- activate pull-down on the given pin
   pull-up <pin>     ;-- activate pull-up on the given pin   

Write a PWM value on a pin:

   set-pwm <pin> <value>
 
   <pin>  : pin number (integer!)
   <value>: an integer between 0 and 1024, or a percentage for duty cycle.

Fade in/out values on a PWM pin:

   fade <pin> from <start> to <end> <delay>   

   <pin>  : pin number (integer!)
   <start>: starting value (0-1024)
   <end>  : ending value (0-1024)
   <delay>: duration of the whole fading (time!)

Wait for a given duration:

   pause <delay>

   <delay>: integer => pause in miliseconds, float => pause in seconds.

Important notes

• Only BCM numbering for GPIO pins is supported.
• A word, or path or paren expression containing regular Red code can be used in place of any numeric value.
• A single command block can contain an arbitrary number of commands.
• The get command can be used multiple times, a block of corresponding results will then be returned in port/data.
• Use sudo when running your GPIO code if it involves PWM!
• An example of using fade and other commands is available here.
• The PWM range is currently preset to 1024, though, the DSL can be trivially extended to allow a user-provided range value.

Happy hardware hacking! ;-)

We are releasing today the version 0.4.0 of the RED Wallet with several major new features. As a reminder, the RED Wallet aims to be a simple and very secure wallet for the major cryptocurrencies (BTC, ETH, and ERC-20 tokens). Safety is enforced through the mandatory use of a hardware key (Ledger or Trezor keys), which protects against any failure of the wallet app (being it a bug or an attacker). The RED Wallet code is fully written in Red (using the Red/System DSL for USB drivers implementation) and open source.

BTC support

The RED Wallet now supports the bitcoin network. The main features are:

• retrieval of address balances.
• sending transactions (with follow-up on an online block explorer).
• support both SegWit and Legacy addresses.
• support both mainnet and testnet.

The balance retrieval can be quite slow on the bitcoin network, so be patient.


ERC-20 tokens support

ERC-20 tokens are supported since version 0.3.0. Though, Ledger removed the ERC-20 tokens list from the embedded Ethereum app since version 1.2.6, leaving it to the wallet app to manage the tokens list. This release now incorporates that list in the RED Wallet binary directly, resulting in a significantly bigger binary (about 100 KB bigger than 0.3.0). The Nano keys will still check the validity of the tokens addresses internally for extra safety. Another consequence of Ledger's recent firmware updates is noticeably slower addresses retrieval when plugging the key. That's an internal change in the Ledger key, so we cannot do anything to bring back the fast speed we could achieve with older firmware.

As a reminder, BIP44 address derivation support is present in the wallet since 0.3.0. If you are using a key initialized in an older Ledger Live version, the derivation path would be different and in order to make the wallet recognize your key, you need to click on the "(Legacy)" label (we will improve that UI in further versions).

Batch transactions were introduced in 0.2.0, they now work for any ERC-20 tokens in addition to ETH. They can be accessed from a contextual menu using a right click on an ETH account.



TREZOR Key

The RED Wallet has updated the firmware support:

• Wallet v0.3.0: Compatibility with firmware 1.7.1+
• Wallet v0.4.0: Support for the latest firmware of the Trezor key. 1.8

TREZOR model T is fully supported on macOS. On Windows 7, if the key is not recognized, you need to install the TREZOR bridge first.


Bug fixes

• FIX: Ledger key does not work on Windows in some cases.
• FIX: crashes in some cases when using Ledger key on Windows.
• FIX: only ledger key is usable when plugging both ledger key and trezor key.
• FIX: trezor-driver context name is wrong.

Download the RED Wallet 0.4.0


Windows 7/8/10 [494 KB] macOS 10.1x [491 KB]

Just click on the executable to run it (extract the .app file first on macOS), no installation or setup process required. Plug in your Ledger or TREZOR key and enjoy a smooth and simple experience!

If you want to check if your RED Wallet binary has been tampered with in any way, you can simply drag'n drop the wallet executable on our binary checking service. If it's legit, the screen will turn green. If it turns red with a warning message, please notify us on Gitter or Twitter at once.

Only download the RED Wallet app from this page, do not trust any other website for that.

Enjoy!

If you are an active member of the Red community, you will have likely been keeping track of Red's progress through our Gitter rooms and Github repositories, but we wanted to officially bring everyone up to speed with a shiny new blog post. The team has been busy since January on many fronts, multiplexing its efforts on many different branches of development. Here is an overview of what we have been cooking and what is coming.

RED Wallet

We are preparing to release v0.4.0 of the RED Wallet which will feature:

• Bitcoin support with both Legacy and SegWit addresses!
• Updated support for ERC-20 tokens for Ledger Nano keys (after Ledger externalized such token management to the wallet software earlier this year).
• Support for the latest firmware of the Trezor key.
• Some bug fixes.

The new wallet is currently under heavy testing, we plan to release it publicly in about a week.

C3

The first public alpha of the C3/System compiler is coming this summer. It is still under heavy development though. C3/High has made little progress as it needs real-world use-cases from the smart contract ecosystem for modelling the DSL(s). That ecosystem has not spread around much, probably due to the so-called "crypto-winter". As new big players are now entering the space, we are looking forward to the possibilities for providing new backends for the C3/System, like for the Libra network.

We will be soon releasing our ETH connector for Red. The final part missing is the integration of hardware and software key support for signing transactions on real online networks (we use private networks during development). The code for hardware key support is already there in our wallet, so we just need to integrate it inside the connector. For software keys, we have developed specific support for them, but it relies on third-party libraries, which are currently not a good match for our single binary distribution model.

GTK support

Red should offer soon a pretty good GUI backend for Linux, thanks to the community effort led by R cqls. The last PR for red/GTK branch weighs in at about 400 commits and covers most of View features.

The main remaining features to cover are:

• Camera widget support (already available in an experimental branch)
• Rich-text and Draw improvements
• Shape dialect support
• Some stability fixes

Here are some examples of scripts and apps using the GTK backend:

GTK theming is also supported:

Red/System

The ARM backend got many improvements:

• Largely improved ARMhf (hard-float) support.
• Fixes on regressions for ARMsf (soft-float).
• Various libraries loading fixes, now all unit tests are passing again.
• New ARMhf target added to our Travis CI backend.

A few low-level features were added, that would make it easier to write device drivers or operating systems directly in Red/System:

• Ability to save/reload all CPU registers on stack using intrinsics.
• Ability to read/write CPU I/O ports using new intrinsics.
• Ability to inline machine-code pretty much anywhere in R/S source code.
• Replacing hard native support for modulo operation on floats by the soft fmod() import.

VSCode plugin
The Red plugin for VSCode got a major refresh with a full reimplementation using LSP model improving the efficiency of existing features and adding some new ones:

• Smarter code completion
• Realtime syntax checking
• Goto definition support

Other WIP features

JSON support

JSON format is now supported by Red's codec system. So it can be used in load/as and save/as to convert JSON data back and forth to Red values. More info about its usage in the Red 0.6.5 release notes.

Shadows in Draw

Some simple shadow effects in Draw have been implemented. They have the capabilities of the box-shadow model of CSS.



GPIO library for Raspberry Pi

Red has supported Raspberry Pi as a compilation target for many years, but we were still lacking a proper GPIO library, which is now under development. It will be implemented as a gpio:// port for now in the port-type branch. A dedicated library or dialect should be added also as a higher abstraction layer in order to simplify GPIO programming even further. Hobbyists and seasoned programmers will be able soon to build IoT projects on RPi easily using Red with GTK for the GUI support.


Next Red releases

0.6.5: Splitting the console and compiler

This is an important change that will be the new way to use Red and, hopefully, provide an even better experience, especially for newcomers. Not much coding is involved but we still need a prototype before giving the final go-ahead then proceeding with the code and infrastructure changes. We plan to finish those tasks during this summer (no precise ETA for now).

0.7.0: Full async I/O

Work started on this release many months ago but with a low-priority, as we have many other features to release first. The current state on the main features is:

• port! type is almost completed and already usable (e.g. the eth:// port) 
• the native ports TCP, UDP, DNS should be implemented this summer.
• the async API still needs design work before being implemented.

Tokens retro-distribution
We are setting up the last steps for the RED tokens retro-distribution (past contributions) and the first monthly distribution for new contributors. We will publish the first article in July that will provide all the rules that we have established for that process. Then shortly after, we will publish the list of the contributors who were selected according to those rules, with the reward amounts and proceed with the distribution (using the batch transactions feature of the RED Wallet).
For example, as part of our tokens rewarding program for contributors, R cqls should receive a good amount of them in the next distribution,


The road ahead...

The most notable next planned milestones are:

• RED Wallet 0.4.0 in July.
• C3/System alpha release in August!
• Red/Android beta release in September!
• Red/Pro for New year!
• Red 64-bit will be split into two phases:
• cross-compiled from 32-bit platform: end of the year
• toolchain ported to 64-bit: 2020

We plan to give more information about each of those big new additions to the Red family this summer.

As several development tracks are coming to fruition in the next couple of months, new articles should come up more frequently.

Go Red Go!