{"id":43,"date":"2009-11-04T17:35:23","date_gmt":"2009-11-05T00:35:23","guid":{"rendered":"http:\/\/log.jonriehl.com\/?p=43"},"modified":"2010-01-12T13:09:12","modified_gmt":"2010-01-12T20:09:12","slug":"embedding-llvm-assembly-in-mython","status":"publish","type":"post","link":"http:\/\/log.jonriehl.com\/?p=43","title":{"rendered":"Embedding LLVM Assembly in Mython"},"content":{"rendered":"

Today we’re going to look at how we can use Mython and llvm-py to embed LLVM assembly code into a Mython module.\u00c2\u00a0 For those not familiar with Mython, I wouldn’t worry too much; what we are doing should not look, nor work too different from the following bit of code (which requires Python 2.5<\/a>, LLVM<\/a>, and llvm-py<\/a> to work, by the way):<\/p>\n

import StringIO, llvm, llvm.core, llvm.ee\r\nllvm_asm = \"\"\"\r\n@msg = internal constant [15 x i8] c\"Hello, world.\\\\0A\\\\00\"\r\n\r\ndeclare i32 @puts(i8 *)\r\n\r\ndefine i32 @not_really_main() {\r\n    %cst = getelementptr [15 x i8]* @msg, i32 0, i32 0\r\n    call i32 @puts(i8 * %cst)\r\n    ret i32 0\r\n}\r\n\"\"\"\r\nllvm_module = llvm.core.Module.from_assembly(\r\n                  StringIO.StringIO(llvm_asm))\r\nmp = llvm.core.ModuleProvider.new(llvm_module)\r\nee = llvm.ee.ExecutionEngine.new(mp)\r\nnot_really_main = llvm_module.get_function_named(\r\n                      'not_really_main')\r\nee.run_function(not_really_main, [])<\/pre>\n
This code first defines an LLVM module in a Python string, then builds a LLVM module from the embedded code, and finally uses a JIT to link and run a function from the embedded module.\u00c2\u00a0 I would note two things about this little demo.\u00c2\u00a0 One, while the multiline string allows un-escaped quote characters, developers must still take care to escape backslashes.\u00c2\u00a0 Failing to do this causes the LLVM assembler to reject the string literal.\u00c2\u00a0 Two, the user of this code pays for the assembly of the LLVM code each time it is run.\u00c2\u00a0 Both of these are relatively minor problems, but they illustrate why a developer might prefer Mython over embedding another language as strings in a Python file.\u00c2\u00a0 Later, we shall develop these arguments in more depth.<\/p>\n
This post demonstrates how we can take the infrastructure in llvm-py and use it to embed LLVM source.\u00c2\u00a0 We show how to assemble the embedded LLVM source into LLVM bitcode at compile time.\u00c2\u00a0 We’ll then stash the bitcode for consumption by the LLVM JIT compiler and linker at run time.\u00c2\u00a0 This approach saves us from the bitcode compilation time, and ideally saves some space in the Python bytecode.\u00c2\u00a0 More importantly, this approach ensures that errors in the embedded source are detected at compile time, not run time.<\/p>\n
Preliminaries<\/h1>\n
If you’re not terribly familiar with Python, LLVM, and llvm-py, I’d recommend reading at least the Python tutorial, the LLVM assembly tutorial, and the llvm-py user guide.\u00c2\u00a0 The llvm-py user guide should, in turn, point you at a specific test case for using the LLVM JIT, which the above code follows except it builds the module from assembly source.\u00c2\u00a0 The llvm-py documentation builds the module using wrapper objects for the LLVM intermediate representation (IR).<\/p>\n
At the time of writing, I used LLVM 2.5 (via MacPorts<\/a>), and built llvm-py from the Google code repository<\/a>.\u00c2\u00a0 Originally, I tried the llvm-py port, but the llvm-py 0.5 tarball they use doesn’t build against LLVM 2.5.\u00c2\u00a0 I encountered this problem with llvm-py 0.5 again on Cygwin<\/a>, this time doing a manual build and install of LLVM 2.5 from a source tarball.\u00c2\u00a0 I was also able to build and install the llvm-py Subversion head, but the example code for this post does not work (it can’t dynamically resolve puts()<\/code>).<\/p>\n
Mython introduces a special form of quotation into the Python language.\u00c2\u00a0 The idea is that you can embed raw strings in your source code, and these strings are interpreted into Python code at compile time.\u00c2\u00a0 Quotation blocks look something like this:<\/p>\n
quote [quotefn] name:\r\n\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 ...<\/pre>\n
The quotefn()<\/code> is ideally a function that takes a name, a string, and a dictionary, and returns a 2-tuple containing a list of Python abstract syntax trees (AST’s, specifically, statement nodes), and a dictionary.\u00c2\u00a0 Instead of giving a quick demonstration of how to define and use a quotation function in Mython, let’s go ahead and demonstrate these by embedding LLVM assembly.\u00c2\u00a0 I will explain the Mython code as we go along.<\/p>\n
I recommend you grab a copy of MyFront (which is part of the Basil language framework), and the test1.my<\/code> source file from the Google Code repository<\/a> (see availability, below).\u00c2\u00a0 The following discussion essentially gives the source code for test1.my<\/code>, but lists it out of order.<\/p>\n
Interfacing llvm-py and Mython<\/h1>\n
So now that I’ve discussed the preliminaries, let’s just go ahead and start defining the driver we’ll use to test the compile-time wrapper for the LLVM assembler.\u00c2\u00a0 Let’s assume that we already have a “quotation” function for LLVM assembly.<\/p>\n
quote [llvm_as] llvm_module:\r\n @msg = internal constant [15 x i8] c\"Hello, world.\\0A\\00\"\r\n declare i32 @puts(i8 *)\r\n define i32 @not_really_main() {\r\n     %cst = getelementptr [15 x i8]* @msg, i32 0, i32 0\r\n     call i32 @puts(i8 * %cst)\r\n     ret i32 0\r\n }<\/pre>\n
Our job consists of defining llvm_as()<\/code> to be a quotation function that translates this quotation block into something like the following:<\/p>\n
llvm_module = llvm.core.Module.from_bitcode(\r\n                  StringIO.StringIO(\"...\"))<\/pre>\n
At run time, the above constructs a LLVM module from the elided bitcode in the string literal (the \"...\"<\/code>).\u00c2\u00a0 We therefore need to define a compile-time function that does the following:<\/p>\n