Visualization of colm's buildprocess

The simplification of a complex buildprocess

Introduction

Some time ago I investigated how the build process of colm is working.

It is not just compiling and linking: Colm is bootstrapped in a clever way.

Build steps

The 'main.cc' is compiled 3 times with various '-D' flags, every time the objects are linked with 2libraries are created libcolm and libprog.

1. step: -DCONS_INIT

First: -DCONS_INIT + consinit.cc + main.cc + libprog.a+ libcolm.so create the executable bootstrap0.

Then: bootstrap0 is run and creates genparse1.c and genif1.[ch].

g++ -DHAVE_CONFIG_H -I.  -I../aapl -Iinclude  -Wall -DPREFIX='"/usr/local"' -Iinclude -DCONS_INIT -Wall -g -MT bootstrap0-consinit.o -MD -MP -MF .deps/bootstrap0-consinit.Tpo -c -o bootstrap0-consinit.o `test -f 'consinit.cc' || echo './'`consinit.cc
mv -f .deps/bootstrap0-consinit.Tpo .deps/bootstrap0-consinit.Po
g++ -DHAVE_CONFIG_H -I.  -I../aapl -Iinclude  -Wall -DPREFIX='"/usr/local"' -Iinclude -DCONS_INIT -Wall -g -MT bootstrap0-main.o -MD -MP -MF .deps/bootstrap0-main.Tpo -c -o bootstrap0-main.o `test -f 'main.cc' || echo './'`main.cc
mv -f .deps/bootstrap0-main.Tpo .deps/bootstrap0-main.Po
/bin/bash ../libtool  --tag=CXX   --mode=link g++ -Wall -DPREFIX='"/usr/local"' -Iinclude -DCONS_INIT -Wall -g   -o bootstrap0 bootstrap0-consinit.o bootstrap0-main.o libprog.a libcolm.la 
libtool: link: g++ -Wall -DPREFIX=\"/usr/local\" -Iinclude -DCONS_INIT -Wall -g -o .libs/bootstrap0 bootstrap0-consinit.o bootstrap0-main.o  libprog.a ./.libs/libcolm.so

2. step: -DLOAD_INIT

First: -DLOAD_INIT + loadinit.cc + main.cc + libprog.a + libcolm.so + genparse1.c + genif1.[ch] create the executable bootstrap1.

Then: bootstrap1 is run with colm.lm as its input and creates genparse2.c and genif2.[ch].

./bootstrap0 -c -o gen/parse1.c -e gen/if1.h -x gen/if1.cc
g++ -DHAVE_CONFIG_H -I.  -I../aapl -Iinclude  -Wall -DPREFIX='"/usr/local"' -Iinclude -DLOAD_INIT -Wall -g -MT bootstrap1-loadinit.o -MD -MP -MF .deps/bootstrap1-loadinit.Tpo -c -o bootstrap1-loadinit.o `test -f 'loadinit.cc' || echo './'`loadinit.cc
mv -f .deps/bootstrap1-loadinit.Tpo .deps/bootstrap1-loadinit.Po
g++ -DHAVE_CONFIG_H -I.  -I../aapl -Iinclude  -Wall -DPREFIX='"/usr/local"' -Iinclude -DLOAD_INIT -Wall -g -MT bootstrap1-main.o -MD -MP -MF .deps/bootstrap1-main.Tpo -c -o bootstrap1-main.o `test -f 'main.cc' || echo './'`main.cc
mv -f .deps/bootstrap1-main.Tpo .deps/bootstrap1-main.Po
g++ -DHAVE_CONFIG_H -I.  -I../aapl -Iinclude  -Wall -DPREFIX='"/usr/local"' -Iinclude -DLOAD_INIT -Wall -g -MT gen/bootstrap1-if1.o -MD -MP -MF gen/.deps/bootstrap1-if1.Tpo -c -o gen/bootstrap1-if1.o `test -f 'gen/if1.cc' || echo './'`gen/if1.cc
mv -f gen/.deps/bootstrap1-if1.Tpo gen/.deps/bootstrap1-if1.Po
gcc -DHAVE_CONFIG_H -I.  -I../aapl -Iinclude  -Wall -DPREFIX='"/usr/local"' -Iinclude -Wall -g -MT gen/bootstrap1-parse1.o -MD -MP -MF gen/.deps/bootstrap1-parse1.Tpo -c -o gen/bootstrap1-parse1.o `test -f 'gen/parse1.c' || echo './'`gen/parse1.c
mv -f gen/.deps/bootstrap1-parse1.Tpo gen/.deps/bootstrap1-parse1.Po
/bin/bash ../libtool  --tag=CXX   --mode=link g++ -Wall -DPREFIX='"/usr/local"' -Iinclude -DLOAD_INIT -Wall -g   -o bootstrap1 bootstrap1-loadinit.o bootstrap1-main.o gen/bootstrap1-if1.o gen/bootstrap1-parse1.o libprog.a libcolm.la 
libtool: link: g++ -Wall -DPREFIX=\"/usr/local\" -Iinclude -DLOAD_INIT -Wall -g -o .libs/bootstrap1 bootstrap1-loadinit.o bootstrap1-main.o gen/bootstrap1-if1.o gen/bootstrap1-parse1.o  libprog.a ./.libs/libcolm.so

3. step: -DLOAD_COLM

It is just: -DLOAD_COLM + loadcolm.cc + main.cc + libprog.a + libcolm.so + genparse2.c + genif2.[ch] create the executable colm.

./bootstrap1 -c -o gen/parse2.c -e gen/if2.h -x gen/if2.cc colm.lm
g++ -DHAVE_CONFIG_H -I.  -I../aapl -Iinclude  -Wall -DPREFIX='"/usr/local"' -Iinclude -DLOAD_COLM -Wall -g -MT colm-loadcolm.o -MD -MP -MF .deps/colm-loadcolm.Tpo -c -o colm-loadcolm.o `test -f 'loadcolm.cc' || echo './'`loadcolm.cc
mv -f .deps/colm-loadcolm.Tpo .deps/colm-loadcolm.Po
g++ -DHAVE_CONFIG_H -I.  -I../aapl -Iinclude  -Wall -DPREFIX='"/usr/local"' -Iinclude -DLOAD_COLM -Wall -g -MT colm-main.o -MD -MP -MF .deps/colm-main.Tpo -c -o colm-main.o `test -f 'main.cc' || echo './'`main.cc
mv -f .deps/colm-main.Tpo .deps/colm-main.Po
g++ -DHAVE_CONFIG_H -I.  -I../aapl -Iinclude  -Wall -DPREFIX='"/usr/local"' -Iinclude -DLOAD_COLM -Wall -g -MT gen/colm-if2.o -MD -MP -MF gen/.deps/colm-if2.Tpo -c -o gen/colm-if2.o `test -f 'gen/if2.cc' || echo './'`gen/if2.cc
mv -f gen/.deps/colm-if2.Tpo gen/.deps/colm-if2.Po
gcc -DHAVE_CONFIG_H -I.  -I../aapl -Iinclude  -Wall -DPREFIX='"/usr/local"' -Iinclude -Wall -g -MT gen/colm-parse2.o -MD -MP -MF gen/.deps/colm-parse2.Tpo -c -o gen/colm-parse2.o `test -f 'gen/parse2.c' || echo './'`gen/parse2.c
mv -f gen/.deps/colm-parse2.Tpo gen/.deps/colm-parse2.Po
/bin/bash ../libtool  --tag=CXX   --mode=link g++ -Wall -DPREFIX='"/usr/local"' -Iinclude -DLOAD_COLM -Wall -g   -o colm colm-loadcolm.o colm-main.o gen/colm-if2.o gen/colm-parse2.o libprog.a libcolm.la 

Overview

As it all looks very complex and complicated, I created a graph to get an better overview.

Please note that this is just a high level overview. Not all files (.c, .cc, .h, .o) are name explicitly.

file: build_process.dot

digraph colm_build {
    rankdir="LR"
    concentrate=false
    node [shape=box]
    {	rank=min;
        consinit[shape=box, color=blue]
        main [shape=box, color=red]
        loadinit [shape=box, color=green]
        colmlm [shape=house, color=green]
        loadcolm [shape=box, color=gold]
    }

    {	rank=min;
        node [color=lightblue]
        resolve lookup synthesis parsetree parser fsmstate fsmbase fsmattach fsmmin fsmgraph fsmcodegen fsmexec pdagraph pdabuild pdacodegen redfsm redbuild fsmap dotgen pcheck ctinput declare codegen exports compiler reduce 
    } -> libprog [color=lightblue]

    {	rank=min;
        node [color=brown]
        map list input debug codevec pool string tree bytecode program struct commit
    } ->libcolm [color=brown]

    {
        libprog[shape=circle,color=lightblue]
        libcolm[shape=circle, color=brown]
    }
    subgraph consinit {
        node [shape=ellipse]

        consinit->bootstrap0 [color=blue]
        main->bootstrap0 [color=red, label="-DCONS_INIT"]
        libcolm->bootstrap0 [color=brown]
        libprog->bootstrap0 [color=lightblue]
        bootstrap0[color=blue]
        bootstrap0->genparse1[color=blue]
        bootstrap0->genif1[color=blue]

    }
    subgraph loadinit {
        node [shape=ellipse]

        colmlm->bootstrap1 [ color=green]
        loadinit->bootstrap1 [color=green]
        genparse1->bootstrap1[color=blue]
        genif1->bootstrap1[color=blue]
        main->bootstrap1 [color=red, label="-DLOAD_INIT"]
        bootstrap1[color=green]
        libcolm->bootstrap1 [color=brown]
        libprog->bootstrap1 [color=lightblue]
        
        bootstrap1->genparse2[color=green]
        bootstrap1->genif2[color=green]
    }
    {
        loadcolm->colm [color=gold]
        genparse2->colm[color=green]
        genif2->colm[color=green]
        main->colm [color=red, label="-DLOAD_COLM"]
        libcolm->colm [color=brown]
        libprog->colm [color=lightblue]
        
    }
    colm[shape=circle, color=green]	
}

dot -Tsvg -obuild_process.dot.svg build_proces.dot

build proces

build_process.dot.svg

It is turtles all the way down

If you think about it, it is realy very meta.

The colm.lm file holds the syntax for the colm interpreter.

And you feed your own xxx.lm file into the colm interpreter that once more outputs an executable.

Next steps

  • [ ] write a colm script that parses the output of makefiles