pymacros4py 0.8.2

Installation

You can get and install pymacros4py as follows by using pip:

pip install pymacros4py

Calling pymacros4py

The preprocessor can be accessed in a Python script as follows:

>>> import pymacros4py
>>> pp = pymacros4py.PreProcessor()

Then, the macros in a template file can be expanded as follows:

pp.expand_file_to_file('my_code.template.py', 'my_code.py')

Warning: When pymacros4py expands macros, it executes the expressions, statements and functions of your macro code. You might want to apply it only to files that you have fully under your control. If you write macro code that includes unsafe file content, you can disable macro expansion for this content.

The method expand_file_to_file offers the following keyword parameter:

• diffs_to_result_file: bool = False. True means that the result is not written to the output file, but compared to the content of the file, and the result of the comparison is returned.

>>> pp.expand_file_to_file('tpl/README.rst', 'README.rst', diffs_to_result_file=True)
''

If you need specific arguments for the parameters encoding, errors, or newlines used for opening files (see Python function open), you can set these as attributes of the global object file_options:

>>> pymacros4py.file_options.encoding = "utf-8"

Using pymacros4py together with a code formatter

If you like to apply a code formatter, e.g., black, on files generated by template expansion, pymacros4py also exposes its functionality at a slightly lower level. This is showcased in the following example. It uses all the functionality provided.

Just expand a template, do not write to a file:

>>> template_path = 'tests/data/file_generating_unformatted_code.tpl.py'
>>> expanded = pp.expand_file(template_path)

Process the results of a macro expansion with an external tool:

>>> temp_file_path = pymacros4py.write_to_tempfile(expanded)
>>> pymacros4py.run_process_with_file(["black", temp_file_path], temp_file_path)
>>> formatted = pymacros4py.read_file(temp_file_path, finally_remove=True)

Store the results of the post-processing to a file:

>>> result_path = 'tests/data/file_generating_unformatted_code.py'
>>> pymacros4py.write_file(result_path, formatted)

Compare two texts, e.g., a template and the expanded result, or an expanded result and a formatted form of its content, in the form used by option diffs_to_result_file of method pp.expand_file_to_file:

>>> print(pp.diff(expanded, formatted, "expanded", "formatted"))
*** expanded
--- formatted
***************
*** 1,3 ****
! print(
! "Hello world"
! )
--- 1 ----
! print("Hello world")
<BLANKLINE>

Notes:

• read_file, expand_file, write_file, and write_to_tempfile all use the file_options described above.

• run_process_with_file calls subprocess.run. The first argument is used as args for this function of the Python standard lib. See there for details. The second argument gives the path of the processed file to pymacros4py, so that the library can hint to the file in case an exception is raised during the run.

• If run_process_with_file raises an exception, the temporary file is not removed: The user might want to examine the content of the file the external code formatter got as input, because the reason of the failure could be within the processed file.

Quoted macro code in templates

One way to mark macro code in a template looks similar to a string starting and ending with two dollar characters. Single or double quotes, or triple single or double quotes can be used.

Note, that whitespace between the quotes and the dollar characters is not allowed.

Example: The following lines each show a macro section with ‘v = 0’ as macro code within the macro section.

'$$ v = 0 $$'
"$$ v = 0 $$"
'''$$ v = 0 $$'''
"""$$ v = 0 $$"""

Start and end of macro code is identified only by the special combination of quoting and dollar characters. Thus, both the quotes and the dollars can be freely used in macro code and in application code, as long as they do not occur directly together. This makes the macro recognition quite robust.

Example: Some dollar characters and quotes in application code and in macro sections, but not combined in the special syntax that starts or ends a macro section

print('This is application code with $$ characters and quotes')
'$$ v = 'This is a quoted string within macro code' $$'

A macro section spans quoting, dollars and code together.

If before and after the quotes, there are only space or tab characters, the macro section is a block macro section (otherwise: an inline macro section) and spans the whole line(s), including a trailing line break if present.

Example: Macro section that spans the whole line, including the trailing line break.

# This is a comment in application code
'$$ v = 0  # This macro section spans the whole line $$'
# This is a second comment in application code

Macro code can span several lines. All four possible quoting types can be used for this, but triple quotes are more pythonic here.

Example: Macro section that spans several lines

'''$$ # This comment belongs to the macro code
      v = 'a string'
$$'''

Whitespace and line breaks in the macro section before and after the macro code are ignored.

Example: Identical macro code (‘v = 0’), surrounded by different whitespace and/or line breaks.

'''$$      v = 0       $$'''
'''$$
      v = 0
$$'''

Indentation in macro code

Macro code in a macro section can be indented to an arbitrary local level, independently of other macro sections and surrounding application code. Locally, indentation needs to follow Python syntax. Globally, pymacros4py will establish a valid indentation when combining code of several macro sections, and code generated by mymacros4py itself.

The first (non-whitespace) character of the macro code in a macro section defines the base indentation of the code. Subsequent lines of the macro code need to be indented accordingly: equally indented (by literally the same characters, but with each non-whitespace character replaced by a space character), or with additional indentation characters (following the base indentation), or not indented at all. When pymacros4py re-indents code, it changes only the base indentation, and it keeps non-indented lines non-indented.

Note: This concept supports indentation by space characters, by tabs, and even mixed forms, and does not require fixing the amount of indentation resulting from a tab. But there is one limitation: If macro code is indented by tabs, it needs to start in its own line.

Example: Macro code starts in its own line. Indentation is done by space and/or tab characters. The indentation of the first non-whitespace character (here: ‘v’) defines the base indentation of the macro section, and subsequent lines are indented equally (by an identical indentation string). The third line is locally indented, relative to this base indentation.

'''$$
        v1 = 0
        for i in range(3):
            v1 += i
$$'''

Example: Macro code starts in the first line of the macro section. All indentation is done by space characters. The number of characters left to the ‘v’ determines the base indentation that the second line follows. The third line is locally indented, relative to this base indentation.

'''$       v1 = 0
           for i in range(3):
               v1 += i
$$'''

Example: Multi-line string with non-indented content

'''$$
    v = """
First line of string. No indentation. This will be preserved.
Second line of string. No indentation. This will be preserved.
"""
    # Our code continues at the level of the base indentation
    insert(v)
$$'''

Macro code in a comment

A second option to mark macro code in a template has the form of a comment, starting with a hash character, optionally followed by spaces or tabs, and two dollar characters. The macro code ends with the line. If there are only space and/or tab characters before the hash, the macro section spans the whole line, including a trailing line break.

# $$ v = 0

Hint: Macro code in a comment is valid Python syntax even if it occurs within signatures in application code. This prevents a Python editor from signalling a syntax error in your template, what can be helpful. Quoted macro code has the advantage that it can span multiple lines, and that some editors highlight it in the template, what can also be useful.

Arbitrary Python code as macro code

Macro code is regular Python code. A call to the predefined function insert inserts the results of applying the function str to the arguments of insert at the place of the macro section.

Example: Macro code defining a function that generates code

def a_function_of_our_application():
    '''$$
    # Here, we define a function in macro code
    def return_print_n_times(n, s):
        statement = f'print("{s}")\n'
        return statement * n
    # Now, we call it
    insert(return_print_n_times(3, "Yep."))
    $$'''

The template script derived from this template generates the following result:

def a_function_of_our_application():
    print("Yep.")
    print("Yep.")
    print("Yep.")

Example: Macro code inserting a computation result

def example_function(i: int) -> int:
    # $$ v = 2 * 2
    return '$$ insert(v) $$'

It evaluates to:

def example_function(i: int) -> int:
    return 4

Indentation of macro results

The results of the expansion of a macro section, e.g., the output of calls of function insert, are indented relative to the indentation of the first non-whitespace character of the macro section (i.e., the hash character for macro code in a comment, resp., the first quote in quoted macro code).

Example: Macro sections with different indentation levels

v = True
# $$ # Macro expansion result will be indented to this level
# $$ insert(f'print({1+1})\n')
if v:
    # $$ # Macro expansion result will be indended to this higher level
    # $$ insert(f'print({2+2})\n')

This template is evaluated to the following result:

v = True
print(2)
if v:
    print(4)

For inline macro sections, the first line of the results is inserted without adding indentation. For block macro sections, each line is (re-) indented.

Example: An inline macro section and a block macro section, both with multi-line results

# $$ v = 2
y = 1 + '$$ insert("(\n", v, "\n* w\n)") $$'
z = 11 + (
         '''$$ insert(v+1, "\n*w\n") $$'''
         )

This template is evaluated to the following result:

y = 1 + (
        2
        * w
        )
z = 11 + (
         3
         *w
         )

In the first case, the inline macro section, the expansion result (starting with the opening bracket) is inserted directly after the application code ‘y = 1 + ‘, without indentation.

In the second case, the block macro section, the expansion result (starting with the ‘3’) is inserted with indentation.

If the library detects zero indentation in macro output, this zero indentation is preserved, i.e., no re-indentation happens.

Example: Recognizable zero indentation in macro output is preserved.

if True:
    """$$
        insert("    v = '''\ntext\n'''\n")
    $$"""

This template is evaluated to the following result:

if True:
    v = '''
text
'''

The macro section of the example starts in an indented suite, here, of an if statement. Thus, macro output of the following macro code will be re-indented to this level - except for the case that zero indentation of output is explicitly demanded. So, we can check in the results, if this exception works.

Then, in the macro code, we start with inserting output at a non-zero base indentation, as reference (the spaces before the assignment). So, the library can detect that the subsequent lines require zero-indentation (the text of the string literal is given with zero indentation).

In the expansion result, we see that the macro output starts indented to the level of the start of the macro section: re-indentation happened here. But then, the zero indentation of the lines of the string literal is detected and thus preserved.

Error messages

In case something goes wrong, pymacros4py tries to give helpful error messages.

Example: Wrong indentation within macro code

'''$$
    # first line
  # indentation of second line below base indentation, but not zero
$$'''

This template leads to the following exception:

>>> pp.expand_file_to_file("tests/data/error_wrong_indentation_in_macro.tpl.py", "out.py"
... )   # doctest: +NORMALIZE_WHITESPACE
Traceback (most recent call last):
RuntimeError: File "tests/data/error_wrong_indentation_in_macro.tpl.py", line 2:
Syntax error: indentation of line 1 of the macro code is not an
extension of the base indentation.

Example: Macro section started, but not ended

'''$$

This template leads to the following exception:

>>> pp.expand_file_to_file("tests/data/error_macro_section_not_ended.tpl.py", "out.py"
... )   # doctest: +NORMALIZE_WHITESPACE
Traceback (most recent call last):
RuntimeError: --- File "tests/data/error_macro_section_not_ended.tpl.py", line 1:
Syntax error in macro section, macro started but not ended:
'''$$
<BLANKLINE>

Example: Nesting of multi-section suites of compound statements wrong, unexpected suite end

#$$ if True:
#$$ :end
#$$ :end

This template leads to the following exception:

>>> pp.expand_file_to_file("tests/data/error_unexpected_end.tpl.py", "out.py"
... )   # doctest: +NORMALIZE_WHITESPACE
Traceback (most recent call last):
RuntimeError: --- File "tests/data/error_unexpected_end.tpl.py", line 3:
Nesting error in compound statements with suites spanning several sections,
in macro section:
  :end

Example: Nesting of multi-section suites of compound statements wrong, suite end missing

#$$ if True:

This template leads to the following exception:

>>> pp.expand_file_to_file("tests/data/error_end_missing.tpl.py", "out.py"
... )   # doctest: +NORMALIZE_WHITESPACE
Traceback (most recent call last):
RuntimeError: Syntax error: block nesting (indentation) not correct,
is :end somewhere missing?

Example: Wrong indentation of expansion results

'''$$
  insert("    # First line indented\n")
  insert("  # Second line indented, but less than the first\n")
$$'''

This template leads to an exception:

>>> try:
...     pp.expand_file_to_file("tests/data/error_result_indentation_inconsistent.tpl.py",
...                            "out.py")
... except Exception as e:
...     print(type(e).__name__)  # doctest: +NORMALIZE_WHITESPACE, +ELLIPSIS
RuntimeError

(Depending on the used Python version, the exception contains notes. If there are notes, the doctest module cannot correctly parse them. And if not, the doctest cannot handle this version-specific deviation of the results. So, above, we only check that the expected exception occurs.)

Comparing results

Method expand_file_to_file offers an option diffs_to_result_file that returns the differences between the results of the macro expansion and the current content of the result file. If there are no differences, the empty string is returned.

Example: Showing results of a change in a template

In the following template, we changed the expression with respect to the example of section Templates and template expansion.

# $$ # In the following line, we changed the expression w.r.t. the example of
# $$ # section Templates and template expansion
# $$ v = 3 * 3
x = '$$ insert(v) $$'

Now, we compare against the result we have gotten there:

>>> print(pp.expand_file_to_file("tests/data/diff_templ_and_templ_exp.tpl.py",
...                              "tests/data/doc_templ_and_templ_exp.py",
...                              diffs_to_result_file = True))
*** current content
--- expansion result
***************
*** 1 ****
! x = 6
--- 1 ----
! x = 9
<BLANKLINE>

Viewing the template script

When an exception is raised during the execution of a generated template script, e.g., if there is an error in your Python macro code, the script will be automatically stored (as temporary file, with the platform specific Python mechanisms) and its path will be given in the error message.

Additionally, the method template_script of pymacros4py can be used to see the generated template script anytime.

Example: Getting the template script

>>> print(pp.template_script("tests/data/doc_templ_and_templ_exp.tpl.py")
... )   # doctest: +NORMALIZE_WHITESPACE
_macro_starts(indentation='', embedded=False,
    content_line='File "tests/data/doc_templ_and_templ_exp.tpl.py", line 1')
v = 2 * 3
_macro_ends('File "tests/data/doc_templ_and_templ_exp.tpl.py", line 1')
insert('x = ')
_macro_starts(indentation='    ', embedded=True,
    content_line='File "tests/data/doc_templ_and_templ_exp.tpl.py", line 2')
insert(v)
_macro_ends('File "tests/data/doc_templ_and_templ_exp.tpl.py", line 2')
insert('\n')
<BLANKLINE>

Here, we used the template from section Templates and template expansion. As can be seen, the real template script looks like the one shown there, but has some additional bookkeeping code that marks when macro code starts and ends during the execution of the template script.

Tracing

pymacros4py can write a trace log during parsing of a template and during execution of a template script: The options trace_parsing and trace_evaluation of method expand_file_to_file activate this functionality. We demonstrate this in the following example with method expand_file, which returns the expansion result instead of storing it to a file.

Example: Tracing of the parsing process

>>> r = pp.expand_file("tests/data/doc_templ_and_templ_exp.tpl.py",
...                    trace_parsing=True)   # doctest: +NORMALIZE_WHITESPACE
--- File "tests/data/doc_templ_and_templ_exp.tpl.py", line 1: line_block_macro:
>v = 2 * 3<
<BLANKLINE>
<BLANKLINE>
--- File "tests/data/doc_templ_and_templ_exp.tpl.py", line 2: text:
>x = <
<BLANKLINE>
<BLANKLINE>
--- File "tests/data/doc_templ_and_templ_exp.tpl.py", line 2: embedded_macro:
>insert(v)<
<BLANKLINE>
<BLANKLINE>
--- File "tests/data/doc_templ_and_templ_exp.tpl.py", line 2: text:
>
<
<BLANKLINE>
<BLANKLINE>

Example: Tracing of the evaluation process

>>> r = pp.expand_file("tests/data/doc_templ_and_templ_exp.tpl.py",
...                    trace_evaluation=True)   # doctest: +NORMALIZE_WHITESPACE
'File "tests/data/doc_templ_and_templ_exp.tpl.py", line 1': line_block_macro
>v = 2 * 3<
<BLANKLINE>
<BLANKLINE>
'File "tests/data/doc_templ_and_templ_exp.tpl.py", line 2': text
>x = <
<BLANKLINE>
<BLANKLINE>
'File "tests/data/doc_templ_and_templ_exp.tpl.py", line 2': embedded_macro
>insert(v)<
<BLANKLINE>
<BLANKLINE>
'File "tests/data/doc_templ_and_templ_exp.tpl.py", line 2': text
>
<
<BLANKLINE>
<BLANKLINE>

Changelog

v0.8.2 (2024-03-12)

• Method PreProcessor.diff and functions read_file, write_file, write_to_tempfile, and run_on_tempfile exported / added. They ease applying an external code formatter on content that has been generated by macro expansion.

• Methods import_from and insert_from support paths relative to the path of the template file, not only relative to the current directory.

• Error messages improved.

• Semantic versioning is used.

v0.8.1 (2024-02-11)

• Error messages and format of text differences improved.

• Source formatted with black default 2024.

v0.8.0 (2024-01-21)

• First published version.