Lexical Analysis Questions and Answers

Answer: C

The two statements differ only in the character separating 1 and 25. The scanner cannot decide the nature of DO until it reaches that point, so unbounded lookahead is required. Modern language designs avoid such constructs precisely so that scanning stays simple.

Answer: D

Correct answer: the scanner cannot use white space to delimit one lexeme from the next

If blanks are insignificant then DO 10 I = 1, 25 and DO10I=1,25 are the same statement, and worse, DO10I=1.25 is an assignment to a variable called DO10I. The scanner cannot tell which until it reaches the comma or the point, so it needs arbitrary lookahead - which is exactly what the design of a scanner is meant to avoid.

Answer: A

Discarding them in the scanner keeps the grammar free of any mention of layout. The scanner still counts newlines so that later diagnostics can name the correct source line. Comments are therefore invisible to the grammar while remaining visible to the reader.

Answer: B

Correct answer: counting the nesting depth is beyond the power of a finite automaton

A finite automaton has a fixed number of states and so cannot count to an arbitrary depth; matching nested comment delimiters is the same problem as matching nested brackets, which is not regular. Scanners therefore either forbid nesting, or handle it with an explicit counter written into the action code rather than in the pattern.

Answer: C

Correct answer: False

This is the standard non-regular language. A finite automaton would have to remember how many opening brackets it had seen in order to require the same number of closing ones, and with finitely many states it cannot count without bound. The pumping lemma makes the argument precise, and the same reasoning is why nested comments defeat a pure scanner.

Answer: D

Correct answer: is not regular by exhibiting a string that cannot be pumped

The lemma states a property that every regular language must have, so it is used contrapositively: choose a string in the language long enough that the lemma would apply, show that no way of pumping it stays inside the language, and conclude that the language is not regular. It can never be used to prove that a language is regular.

Answer: A

Nodes are states and edges are moves on input characters. Hand written scanners are often coded directly from such diagrams, with each state becoming a labelled section of code. Generated scanners encode the same diagram as a table rather than as explicit code.

Answer: B

Correct answer: a character was read that belongs to the next lexeme rather than this

The scanner often knows a lexeme has ended only by reading the character after it - the digits of a number end at the first non-digit. That character belongs to the next lexeme, so it must be pushed back for the next call to read. This is why one character of lookahead, and a way of retracting it, is part of every scanner.

Answer: C

Correct answer: a program that recognises the specified tokens in an input stream

Lex takes a specification of patterns with actions and emits source code for a scanner - typically a table-driven automaton together with the driver that runs it. The scanner it produces recognises the specified tokens and executes the corresponding action for each. Producing a parser is the job of a separate tool such as Yacc.

Answer: D

Correct answer: a declarations section, a rules section and a user code section

The three sections are separated by lines containing %%. The declarations hold regular definitions and any C declarations needed; the rules pair each pattern with the action to run when it matches; and the user code section is copied verbatim into the generated file, and is where supporting functions go.