I was invited to demo Grain DDL at the Handmade Network Expo in Vancouver this year. What followed was 20 minutes sharing what Grain DDL is, and some pretty unique examples of how a code generator can be used.

Come watch the talk on Youtube and check out the newly launched product page.

A designer who worked on a AAA open-world game once remarked that he spent weeks implementing a sweeping behaviour change across every traffic light in the world. A design change made it necessary to re-code every traffic intersection in the game to support right-on-red traffic.

When he proposed that the next game be built with common traffic light logic that applied to all lights, the counterargument was: “if it breaks in one place, it’ll break everywhere”. His response: “If it breaks everywhere, we’ll know exactly where to go to fix it.”

Practising developers find that the truth lies somewhere in between, and it depends on your project. Your tools must empower you to be productive wherever on that spectrum you find yourself.

Grain DDL 0.2.0 (pre-alpha test) now offers a technique called value specialization. This allows you to declare a type that has the exact same fields, in the same order as a base struct, but with the ability to explicitly override the values. No new values can be added.

struct A {  i32 a = 10,  i32 b = 20}struct B copies A {  // a is copied from A.a  i32 b = 30}

This produces the following values:

A.a = 10A.b = 20B.a = 10B.b = 30 

This is “value specialization for PODs” — a flat memory model where only plain old data is subject to being overridden.

Grain DDL is designed to let you generate anything you want from its source data and types. But it keeps you in check about what you can define to keep you generating simple, effective code. In order to use value specialization to its maximum advantage, only the values can change, and not the types. Consider:

struct C copies A {  // error: A.c does not exist and new fields not possible  i32 c = 10,  // error: A.b exists, but it is i32, not f32  f32 b = 10,}

In the case of C.b, you might be asking “why are you asking the user to restate the type in the derived struct, if it has to match? “

When creating just a few instances by copies, this will seem like an unnecessary burden. However, once you have quite a few, a sense of base type calcification naturally occurs: if I change the type of this field in the base, how many downstream things could break?

Grain DDL has implicit casting of expressions resulting in scalar values — albeit with full checks for overflow and underflow at compile time. However, in the event that a type changes in a copied struct, it becomes important to comb over the code and ensure that all of the values are also correctly and adequately typed. This requires a touchpoint from a developer. No silent truncation is possible.

The compiler catches the type mismatch until you visit the line. By changing the type, you are signing off that the declared value is compatible with the type change.

Late-Stage Binding

One of the timeless functions of spreadsheets is relative referencing. If you have a formula like =A1+B1 and drag the selection box downwards, the next field will be =A2+B2. This is used to great effect in calculating the same formula for a different set of values.

Grain DDL uses late-stage binding. When you have a base struct with a formula that has field names in it and it is derived via copies, the formula is calculated in the context of the values in the derived struct.

Consider how sum behaves on override of b:

struct A {  i32 a = 10,  i32 b = 20,  i32 sum = a + b}struct B copies A {    i32 b = 30,  // instead of 20  // sum calculated here with 'b' overridden to 30}

This produces the following values:

A.a = 10A.b = 20A.sum = 30    <- sum of A.a + A.bB.a = 10B.b = 30B.sum = 40    <- sum of B.a + B.b

This is a powerful feature that permits you to model relationships between values in one place, and have that cascade. It is designed to run as a precompile step, so the result of your computation is simply placed in your code and values are easily reviewable.

Said another way, the values are computed at emit time, and can be reviewed as plain numbers.

Putting it Together With Codegen Templates

Grain DDL lets you generate anything you need by specifying a .cgtmpl file at compile time.

When iterating over structs, .copy_name gets the name of the struct:

[_ range .structs  if .copy_name    .name ` copies ` .copy_name ``.  else    .name ` does not copy`.  endend_]

Produces

A does not copyB copies A

Live Sample Code

See example “Copies Keyword” in the Grain IDE for a demonstration of how weapons in a game use value specialization to create one type, calculate DPS from a set of factors, and produce simple C definitions that reduce the complexity of a game runtime.

On Traffic Lights and Best Practices

I believe my colleague’s desire to have all of the traffic lights in an AAA open world operate from the same common logic was directionally correct. However, if you are building a hundred intersections in an open world, I would recommend using a system like Grain DDL’s value specialization to define them, keeping in mind:

• Code-Generate a single base structure — all traffic lights are one fat struct, where the difference is in the values rather than through tagged discrimination.

• Use Grain DDL’s late-stage bindings to formulate values and then generate compile-time asserts to ensure they remain in acceptable ranges to catch problems at compile time rather than QA time.

• Consider that some traffic intersections may be so different they should be special-case coded and not part of the system.

• Use Grain DDL’s type checking system to make incremental, verified structural alterations as needed.

Help Develop Grain

Grain is currently in closed pre-alpha testing. If you would like to try it or share feedback, email grain _at_ frogtoss dot com.

It is deceptively easy to make the mistake of attributing a language’s popularity solely to the syntax and functional features of the language. In reviewing why languages became successful, there is often a retroactive feature-driven mythology wherein we over-attribute the success of a language to its unique syntax and functionality. Simultaneously, we under-attribute the environment the language was in at the time of its early maturity to its eventual uptake.

Ultimately, no programming language has become highly adopted on function and syntax alone, and so it does not make sense to retroactively attribute its success solely to function and syntax. Its ascent must be contextualized with the era of its uptake to be well understood.

The adoption of a language toolchain takes significant effort for a team or an org to achieve, and some threshold must be crossed to make the integration last. While language features can lower this threshold, it is the environment in which the language flourishes that justifies the overcoming of the threshold in the first place. This forward momentum — the why — is essential for adoption.

In practice, the most popular languages we use are the result of the intersection of trends in computing with the function of the language. It is informative to look at the environment in which each of these languages flourished.

• C C was the best language for hardware-agnostic Unix programming at the time when Unix was taking off. Low-overhead Unix syscall compatibility and direct memory access were a fit for the everything-is-a-file Unix trend, whereas Pascal was weaker in these crucial areas.

• C++ By the 1990s, programs were increasingly created by larger teams. C++ offered syntactic boundaries that allowed programs to map to team structure. Additionally, the 1990s were a golden age for selling compilers in boxes at retail, enabling unparalleled awareness of the language. GUIs became commonplace, and the OOP additions to C++ were perceived as useful for composing graphical interfaces.

• Perl CGI needed something more practical for extraction and reporting than Bash. JavaScript was effectively unreliable in web browsers of the era, so sophisticated full-page server-side rendering was the order of the day. Late ‘90s cheap shared hosting democratized server access. The availability of MySQL and Postgres clinched the deal, and the LAMP stack was born.

• Python 2 By the 2000s, scripts were increasingly created by larger teams. Python 2 offered syntactic constructs that made it easy to compose long-running scripts in a more readable and maintainable way than Perl.

• Python 3 PyData/ML stack became institutional. As data science and ML hiring surged in the 2010s, Python 3 became the default research-to-production path. Many new developers may even think of NumPy/SciPy as being intrinsic to the toolchain.

• Ruby (on Rails) Developers needed to iterate quickly in the problem space of web service business. This happened at a time when VCs infused cash into a cultural movement that emphasized iterating quickly to find product-market fit.

• JavaScript The web browser is the open, universal runtime we got, and JavaScript had a twenty-year head start on any real alternative. (Sorry, Java Applets).

• Java The onboarding of businesses to the Internet happened at a time when a memory-safe language was needed for network-aware programming by large numbers of programmers. The influx of many freshly graduated compsci majors who found direct memory manipulation challenging could be put to work to deliver real corporate value on large teams. Java’s timing was serendipitous for the enterprise build-out.

• C# Microsoft, at a time when they were the most dominant software vendor in the world, massively backed developers during “dot com” layoffs and early 2000s corporate downsizing. They emphatically showed the Windows platform-supported way of doing memory-safe programming. Using COM or Java to build Windows was now off-book. Out-of-work developers took them seriously and reskilled with C#. Unity later bet the farm on C# which was a second shot in the arm for the language.

• Rust Modern networked devices have to deal with increasingly diverse untrusted data processing (such as bluetooth packets, fonts, audio streaming). Major industry players such as Microsoft, Google and AWS have sought to reduce memory-safety-related CVEs in “zero-click” attack surfaces without the use of garbage collection.

A lesson we can take from this is that you can’t market a language a position of popularity by simply espousing its features and qualities in public. You can, however, build a language and the foundations of its toolchain and help it hitch onto a greater trend. Changing needs are what prompt teams to overcome the friction involved in upgrading their software development tooling.

As always, timing is key.