2016/08/12

Perl Parser Plugins - part 1

Today's subject is the parser plugins that were added to perl in 5.14. The parser plugin API itself is relatively small and modest, but the power it grants the user is enormous. To use this power however, you need to know and be familiar with a lot of the internals of the perl core.

This post will be a little different to most of my usual posts. Rather than announcing and demonstrating some code that's already written and available on CPAN or wherever, I shall today be writing about some code I'm currently in the progress of writing. I'm hoping this series of posts will contain useful information about the underlying subject of perl parser plugins and other internal details.

As a parser plugin would be written in C as part of an XS module, the reader is presumed to be at least somewhat familiar with C syntax, and to have at least a basic understanding of what an XS module is and how to go about creating one.

The PL_keyword_plugin Hook

The initial point of entry into the parser plugin system is a function pointer variable that's part of the interpreter core, called PL_keyword_plugin. This variable points to a function that the perl parser will invoke whenever it finds something that looks like it could be a keyword. It's typed as

int (*PL_keyword_plugin)(pTHX_
    char *keyword_ptr, STRLEN keyword_len,
    OP **op_ptr);

How it behaves is that whenever the parser finds a sequence of identifier-like characters that could be a keyword, it first asks the keyword plugin whether it wishes to handle it. The sequence of characters itself is passed in via the pointer and length pair. Because this is a pointer directly into the parser's working-space buffer, the keyword itself will not be NUL-terminated, and so the length argument is required too. The integer return value works with the op_ptr double-pointer value to let the plugin return its result back to the interpreter in the form of an optree. For today's post we won't concern ourselves with this optree, and instead write a plugin that doesn't actually do anything.

Side note:If you don't recognise what that pTHX_ macro is doing at the start of the argument list, don't worry too much about it. It exists to pass around the perl interpreter state if the perl is built with threads enabled. Just think of it as a curious quirk of writing C functions in the perl interpreter; the detail of its operation does not concern us too closely here.

To use this plugin API, we need to define our own custom handling function, and set this variable to its address so that the perl parser will invoke it. This following example implements a trivially tiny plugin; one that declines to process any keyword, but as a side-effect prints its name to the standard error stream, so we can see it in action.

static int MY_keyword_plugin(pTHX_ char *kw, STRLEN kwlen,
    OP **op_ptr)
{
  fprintf(stderr, "Keyword [%.*s]\n", kwlen, kw);
  return KEYWORD_PLUGIN_DECLINE;
}

MODULE = tmp  PACKAGE = tmp

BOOT:
  PL_keyword_plugin = &MY_keyword_plugin;

To test this out, I've compiled this little blob of XS code into a tiny module called tmp. The plugin's output can clearly be seen:

$ perl -Mtmp -e 'sub main { print "Hello, world\n" }  main()'
Keyword [sub]
Keyword [print]
Keyword [main]
Hello, world

Here we can see that the keyword was first informed about the sub. Because we declined to handle it, this was taken by the core perl parser which consumes the main identifier and opening brace in its usual manner. Next up is another candidate for being a keyword, print, which we also decline. The entire string literal does not look like a keyword, so the plugin wasn't invoked here. It is finally informed again of the main that appears at the top-level of the program, which it also declines.

From this initial example, we can see already how the parser plugin API is a lot more robust than earlier mechanisms, such as source filters. We haven't had to take any special precautions against false matches of keywords inside string literals, comments, or anything else like that. The perl parser core itself already invokes the plugin only in places where it expects there to be a real keyword, so all of that is taken care of already.

Hook Chains

The plugin we have created above has one critical flaw in it - it is very rude. It takes over control of the PL_keyword_plugin pointer, overwriting whatever else was there. If the containing program had already loaded a module that wanted to provide custom keyword parsing, our module has just destroyed its ability to do that.

The way we solve this in practice is to wrap the pre-existing parser hook with our own function; saving the old pointer value in a local variable within the plugin code, so if we don't want to handle the keyword we simply invoke the next function down the chain. The perl core helpfully initialises this variable to point to a function that simply declines any input, so the first module loaded doesn't have to take special precautions to check if it was set. It can safely chain to the next one in any circumstance.

We can now update our code to do this:

static int (*next_keyword_plugin)(pTHX_ char *, STRLEN, OP **);

static int MY_keyword_plugin(pTHX_ char *kw, STRLEN kwlen,
    OP **op_ptr)
{
  fprintf(stderr, "Keyword [%.*s]\n", kwlen, kw);

  return (*next_keyword_plugin)(aTHX_ kw, kwlen, op_ptr);
}

MODULE = tmp  PACKAGE = tmp

BOOT:
  next_keyword_plugin = PL_keyword_plugin;
  PL_keyword_plugin = &MY_keyword_plugin;

The plugin now behaves exactly as it did before, but this time if there had been any more plugins loaded before this one, they will be given a chance to run as well. Hopefully the next one that's loaded does this same logic, so that ours continues to work.

Well, I say "work".

Aside from spying on the keywords in the program, this parser plugin doesn't do anything useful yet. I shall start to explore what things the plugin can actually do in part 2.

<< First | < Prev | Next >

No comments:

Post a Comment