Porting D to the Mac Pt. 2
written by Walter Bright
February 1, 2009
I had “hello world” working, but I was using D as a C compiler to do it. All that existed was a main() function. There was a long way to go — functions, data, the runtime library, the test suite, etc. First I’d try and build the runtime library.
Doing so exposed a lot of conditional compilation issues that had no case for OSX. I found that Linux has a bunch of API functions that are missing in OSX, like getline and getdelim, so some of the library functionality had to revert to more generic code for OSX. I had to be careful, because although many system macros had the same functionality and spelling, they had different expansions. Getting these wrong would cause some mysterious behavior, indeed.
After compiling the library modules, they had to be combined into a library file. Normally this is just done with ar, the standard “archive” program, but dmd has a nice feature where it can build the library directly without having to write object files out to disk. Not only is this very fast, but it makes the build process simpler by obviating the need to manage the object files. The archive file format is supposed to be a standard, but of course everyone implements it very differently. The OSX archive format spec is only about half there, and what is there is wrong on a few details. This is where building a file dumper again comes in real handy. By varying the input to ar, dumping the resulting archive, and seeing how things change, one eventually figures out the format. If you get it wrong, the OSX linker ld likes to help you out with a convenient “Bus error” message. To figure those out, it’s back to puzzling out the hex dumps of the archive dmd generated against the one ar generated. I did finally figure it out; the archive format has some definite oddities like entries are aligned on 4 byte boundaries but the object file within each entry must be aligned on 8 byte boundaries, offset by 4. Weird. Anyhow, once that was working and the linker accepted the archive, it was very pleasing to see how fast dmd would chew through the code and build the library.
Unfortunately, none of the library code ran, and I mean none of it. When the compiler was originally designed, I attempted to do the right thing and abstract away the memory model and the object file format. Because I had only one example, the abstraction lines turned out to be in all the wrong places. I coined the term “premature abstraction”, which, like the related term premature optimization, is coming up with all the wrong abstractions because you don’t know what you’re doing. (Andrei suggested I write a column about that, but a quick google search showed I was not original and the term is already in use!)
For example, the data structures being written out were divided into mutable and immutable sections, the immutable stuff being destined for a read-only hardware protected memory section. But on OSX, read only data sections get marked as “TEXT” and put in with the code, where they are expected to be position independent. Position independence means no relocations are possible at program load time, meaning the read only data cannot have any pointers in it. If there was a relocation in the TEXT section, the loader didn’t complain, you just wound up with garbage for a pointer. So, all the data structure generating code had to be carefully gone through and separated into “has pointers” and “no pointers” blocks, and the “has pointers” stuff gets routed to the regular mutable data segment.
Another tricky problem with position independent code on OSX is the way data variables are accessed. If they are defined within the module they are referenced in, one indirection is generated. If defined externally, two are necessary. The trouble is, the D programming language allows forward references to data, so there’s a chicken-and-egg problem in deciding what to resolve first.
The last problem is D needs to generate some tables that are coalesced with other tables, such as the exception handling data. To group stuff together, it is put in a specially named segment. The problem is locating the beginning and end of that segment at run time. The solution is a trick I learned long ago in the DOS world - to always put out that special section as a trio of 3, always in the same order. The linker thankfully will then maintain the same order in the executable, so to find the beginning a global is put in the first and third segments, and the addresses of those globals neatly bracket the table in the second segment. This is the way I’m going to try and do thread local storage, too.
Now that dmd is generating three very different object file formats, where the abstraction lines should be is a lot clearer.
Finally, tonight, I got the runtime to start up, run, and shut down successfully. Next week, I’ll try to get the test suite to run.
Acknowledgements
Thanks to Bartosz Milewski and Andrei Alexandrescu for reviewing this.