bison & flex

Flex is a Fast Lexical Analyzer.

Flex is a fast lexical analyzer generator. It is a tool for generating programs that perform pattern-matching on text. Flex is a non-GNU free implementation of the well known Lex program.

Flex is a tool for generating scanners: programs which recognized lexical patterns in text. flex reads the given input files, or its standard input if no file names are given, for a description of a scanner to generate. The description is in the form of pairs of regular expressions and C code, called rules. flex generates as output a C source file, `lex.yy.c', which defines a routine `yylex()'. This file is compiled and linked with the `-lfl' library to produce an executable. When the executable is run, it analyzes its input for occurrences of the regular expressions. Whenever it finds one, it executes the corresponding C code.

Some simple examples

First some simple examples to get the flavor of how one uses flex. The following flex input specifies a scanner which whenever it encounters the string "username" will replace it with the user's login name: 

%%
username printf( "%s", getlogin() );

By default, any text not matched by a flex scanner is copied to the output, so the net effect of this scanner is to copy its input file to its output with each occurrence of "username" expanded. In this input, there is just one rule. "username" is the pattern and the "printf" is the action. The "%%" marks the beginning of the rules. 

Here's another simple example: 

int num_lines = 0, num_chars = 0;

%%
n ++num_lines; ++num_chars;
. ++num_chars;

%%
main()
{
yylex();
printf( "# of lines = %d, # of chars = %dn",
num_lines, num_chars );
}

This scanner counts the number of characters and the number of lines in its input (it produces no output other than the final report on the counts). The first line declares two globals, "num_lines" and "num_chars", which are accessible both inside `yylex()' and in the `main()' routine declared after the second "%%". There are two rules, one which matches a newline ("n") and increments both the line count and the character count, and one which matches any character other than a newline (indicated by the "." regular expression). 

A somewhat more complicated example: 

/* scanner for a toy Pascal-like language */

%{
/* need this for the call to atof() below */
#include < math.h >
%}

DIGIT [0-9]
ID [a-z][a-z0-9]*

%%

{DIGIT}+ {
printf( "An integer: %s (%d)n", yytext,
atoi( yytext ) );
}

{DIGIT}+"."{DIGIT}* {
printf( "A float: %s (%g)n", yytext,
atof( yytext ) );
}

if|then|begin|end|procedure|function {
printf( "A keyword: %sn", yytext );
}

{ID} printf( "An identifier: %sn", yytext );

"+"|"-"|"*"|"/" printf( "An operator: %sn", yytext );

"{"[^}n]*"}" /* eat up one-line comments */

[ tn]+ /* eat up whitespace */

. printf( "Unrecognized character: %sn", yytext );

%%

main( argc, argv )
int argc;
char **argv;
{
++argv, --argc; /* skip over program name */
if ( argc > 0 )
yyin = fopen( argv[0], "r" );
else
yyin = stdin;

yylex();
}

This is the beginnings of a simple scanner for a language like Pascal. It identifies different types of tokens and reports on what it has seen. 

The details of this example will be explained in the following sections.