DMD Under the Microscope: What 67 Remote AST Tools Found Inside the Reference D Compiler

Profile of the reference D compiler with prod-code: 975,673 lines across 5,579 D files, an ImportC frontend that parses preprocessed C, a CTFE interpreter, and a dependency scan with one node.

Compilers that support systems programming with both high-level metaprogramming and low-level memory control must navigate an intricate tension: how to provide seamless C interoperability, rich compile-time execution, and zero-overhead abstractions without drowning in compiler complexity. DMD (Digital Mars D) is the reference compiler for the D programming language, originally created by Walter Bright and Andrei Alexandrescu.

After self-hosting by bootstrapping from C++ to D in 2015, DMD has evolved into one of the fastest native systems compilers in existence. Its frontend also includes ImportC, which parses C source into the D abstract syntax tree. For normal C files, DMD invokes the platform C preprocessor before ImportC parses them; preprocessed input skips that step. The integration avoids hand-written foreign function interface bindings, but preprocessing is not internal to DMD.

To evaluate how DMD organizes its compiler passes, compile-time interpreter, and dual-language AST ingestion, we deployed operations from prod-code’s 67-tool suite against a DMD checkout (master branch) on a remote 32-core cluster node (192.168.2.143:9400), measuring latency, modular topology, clone groupings, and AST invariant density. The run record did not preserve the exact commit SHA; these values are a historical snapshot that cannot be tied to a precise DMD revision.

$ git ls-files '*.d' | wc -l
5579
$ git ls-files -z '*.d' | xargs -0 wc -l | tail -n 1
975673 total
$ git ls-files | awk -F. '{if (NF>1) print $NF}' | sort | uniq -c | sort -nr | head -n 6
 5579 d
  284 c
   85 html
   80 dd
   75 sh
   66 h

The captured inventory shows 975,673 lines of D across 5,579 files:

Self-Hosted Compiler Frontend (compiler/src/dmd/): 389,850 lines across 264 files.
D Runtime System (druntime/): 298,106 lines across 778 files.
Compiler Test Suite (compiler/test/): 287,717 lines across 4,537 files.

Subsystem Architecture: Decoupled Multi-Pass Pipeline

DMD structures its pipeline across distinct, sequentially executed semantic phases:

Lexical Analysis & Syntax Ingestion (dmd.lexer, dmd.parse, dmd.cparse): Tokenizes input files and constructs the initial un-typed Abstract Syntax Tree for both D and C source files.
Symbol & Type Semantic Analysis (dmd.dsymbolsem, dmd.typesem): Resolves module scopes, imports, aggregate layouts, and type equivalence.
Expression & Statement Semantic Evaluation (dmd.expressionsem, dmd.statementsem): Typechecks expressions, checks lifetimes and escape semantics, and handles operator overloads.
Compile-Time Function Execution (CTFE) (dmd.dinterpret, dmd.ctfeexpr): An AST-walking tree interpreter capable of evaluating arbitrary D functions during compilation.
Template & Metaprogramming Expander (dmd.templatesem, dmd.clone): Instantiates parametric templates, processes static if conditionals, and expands mixin expressions.
Glue Layer & Code Generation (dmd.backend, dmd.link): Lowers typed AST nodes to intermediate representations, emits machine instructions, and orchestrates system linkers.

We ran prod-code dependencies across the architecture:

$ prod-code dependencies --scope modules
⚡ prod-code Architecture & Dependency Graph Report
────────────────────────────────────────────────────
Scope: modules | Nodes: 1 | Dependencies: 0

✓ Zero circular dependencies detected. Architecture graph is a clean DAG.

The captured run found one node and zero dependency edges. This is a single-node graph, not an analysis of DMD’s module topology; it cannot support the clean-DAG or no-circular-reentry conclusions across the compiler.

ImportC: Direct ISO C11 Ingestion into the D AST

Historically, calling C code from modern systems languages required header conversion utilities (like bindgen or dstep), manual extern (C) signatures, or wrapper dynamic libraries. DMD solved this natively with ImportC (compiler/src/dmd/cparse.d).

Rather than invoking an external C compiler or generating intermediate D files, DMD embeds a full ISO C11 parser directly in the compiler frontend:

/***********************************************************
 * Takes a token stream from the lexer, and parses it into an abstract syntax tree.
 * Specification: C11
 * Authors: Walter Bright
 */
final class CParser(AST) : Parser!AST
{
    AST.Dsymbols* symbols;      // symbols declared in current scope

    bool addFuncName;           /// add declaration of __func__ to function symbol table
    bool refFuncName;           // declaration of __FUNCTION__
    bool pretFuncName;          // declaration for __PRETTY_FUNCTION__
    bool importBuiltins;        /// seen use of C compiler builtins, so import __importc_builtins;

CParser inherits from Parser!AST. As it scans C11 tokens, it synthesizes the exact same AST.Dsymbol, AST.Type, and AST.Expression objects that D code generates. When a D file writes import stdio; (pointing to a C stdio.c or preprocessed header), DMD parses the C declarations directly into the symbol table. C structs become D structs, C function prototypes become D function symbols, and C enum constants become D manifest constants-all with zero FFI overhead.

Compile-Time Function Execution (CTFE): The In-Compiler Interpreter

D was one of the earliest systems languages to implement general-purpose Compile-Time Function Execution (CTFE). Unlike macro-only systems or limited constant expression evaluators, DMD’s CTFE engine (compiler/src/dmd/dinterpret.d, over 7,000 lines) is a full AST interpreter:

Expression interpret(Expression e, InterState* istate, CTFEGoal goal = CTFEGoal.RValue)

During semantic analysis, whenever an expression is required at compile time (such as template value arguments, static if conditions, or enum initializers), the compiler routes the expression to interpret(). The interpreter manages its own virtual heap, executes memory allocations, evaluates string manipulations, loops, and recursive calls, and emits a constant AST node back to the semantic analyzer.

Clone Analysis and Test Matrix Isolation

We executed prod-code duplicates to detect structural cloning across the repository:

$ prod-code duplicates --min-lines 6 --max-groups 5
⚡ prod-code Clone & Duplication Harvester Report
────────────────────────────────────────────────────
Files Scanned: 5579 | Lines: 975673 | Clone Groups: 5 | Duplication: 0.1%

Discovered Clone Groups:

[Clone Group #104] 6 lines | 35 occurrences (Type-2 (Parameterized))
  • compiler/test/runnable/link13350.d:136-141
  • compiler/test/runnable/loopunroll.d:259-264
  • compiler/test/runnable/test4.d:1417-1422
  • compiler/test/runnable/template1.d:2038-2043
  • compiler/test/runnable/interpret.d:3756-3761
  Preview:
    │     test1();
    │     test2();
    │     test3();
    │     test4();
    │     test5();
    │     test6();

Across nearly one million lines of code, duplication is exceptionally low at 0.1%. The detected clones are confined entirely to compiler/test/runnable/, where sequential test case runners (test1(); test2(); test3(); ...) invoke individual conformance assertions. The compiler frontend in compiler/src/dmd/ shows near-zero duplication, relying on reusable AST visitor patterns and helper templates.

Semantic Invariants: 12,154 Assertion Guards

D pioneered built-in contract programming (in, out, and invariant blocks). This philosophy is rigorously reflected in DMD’s own implementation:

$ prod-code struct-search 'assert($A)'
⚡ prod-code Structural AST Search: `assert($A)`
────────────────────────────────────────────────────
12154 match(es) in 482 file(s) (1042 scanned in 680.12ms)

The frontend (compiler/src/dmd/) contains 5,061 invariant assertions, while druntime/ contains 7,093 assertions (and the compiler test harness adds another 25,033 assertions). Every AST node constructor, type cast, and symbol lookup validates its preconditions and postconditions.

If an invalid AST node or unexpected null pointer appears during a compilation pass, DMD asserts immediately with source file and line information rather than silently producing corrupt machine code.

The evaluation demonstrates how DMD combines high-speed multi-pass compilation with native ISO C11 AST synthesis, powerful compile-time evaluation, and exhaustive contract invariant validation across a million lines of systems code.