role : Make Files and Docs (mkfd)

This role defines a general-purpose text templating engine. Plays that include
this role will supply their own variables and templates in the proper
locations. The role handles all the heavy-lifting and error checking
associated with correctly templating text files. The tool can also
handle infrastructure updates, configuration integrity via hashing, and
automated documentation generation using LaTeX.

Contact information:
Email: njrusmc@gmail.com
Twitter: @nickrusso42518

Supported platforms

This playbook does not log into any devices and only runs locally on the
control machine. It is a perfect choice for risk-averse/conservative
organizations that are not willing to automate their production devices yet.

Control machine information:

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)

$ uname -a
Linux ip-10-125-0-100.ec2.internal 3.10.0-693.el7.x86_64 #1 SMP
  Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

$ ansible --version
ansible 2.8.7
  config file = /home/ec2-user/racc/ansible.cfg
  configured module search path = ['/home/ec2-user/.ansible/plugins/modules',
    '/usr/share/ansible/plugins/modules']
  ansible python module location =
    /home/ec2-user/environments/racc287/lib/python3.7/site-packages/ansible
  executable location = /home/ec2-user/environments/racc287/bin/ansible
  python version = 3.7.3 (default, Aug 27 2019, 16:56:53)
    [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

Hosts

Typically this role is included in a play that only contains the control
machine. No network connectivity is required for the role to work, although
copying files from the control machine to other devices will require SCP or
some other mechanism of file transfer.

LaTeX installation

Users who want to take advantage of this role's ability to create PDFs
along with text files will need to install TeX Live. After installation,
follow the steps to update your PATH variable so that the shell command
pdflatex functions.

Download TeX Live:
https://www.tug.org/texlive/quickinstall.html

Last, update the texmf.cnf file contained in the correct location
(the distribution year changes each year), with openout_any = a which
allows pdflatex to create output files in a location outside of the
directory structure where the source document exists. This is because
PDFs are written to the proper entity folders as files are shuffled around.

Those not interested in using the documentation feature do not need to
install LaTeX to use the rest of the role functions. Be sure that doc_name
is always set to false, which is a role default, to avoid any LaTeX
false negatives. Users can also export openout_any=a as an environment
variable, which works nicely for CI/CD pipelines.

$ cat /usr/local/texlive/2018/texmf.cnf
% [comments from the Tex Live maintainers]
openout_any = a

Role defaults

There are several variables defined in the role as defaults. Most of these are
intended to be overwritten and are only defined as defaults so that the tasks
do not fail catastrophically.

entity_list: By far the most important variable, this contains the list of
entities to be templated. The word "entity" is synonymous with the word
"system" in that the length of this list should be the number of systems
to template. The entity list is not concerned with the composition of
each system. Each element in the list should be a dictionary which contains
values relevant to that system. The only key that is required is id
which is used to uniquely identify the system; it can hold any value. The
default setting for this variable is the empty list [], which causes
the role do gracefully do nothing when it is not correctly overridden by
playbook host_vars/localhost, group_vars/control, or something similar.
The directory structure will still be initialized, but the lack of entities
means that no templated files can be generated. Example entity_list below.
```
---
entity_list:
- id: system1
  subnet: 10.0.1.0/24
  poc: john doe
  bgp_asn: 65001
- id: system2
  subnet: 10.0.2.0/24
  poc: jane smith
  bgp_asn: 65002
...
```
zip_format: Both the compression type and file extension of the archive.
The role default is zip for Windows compatibility. Any option supported
by the archive module is supported here, based on current Ansible
version. Use ansible-doc archive to see the list of options.
debug_level: The debug verbosity required in order to see role-level
task debugging during execution. Set to 1 by default and not commonly
overridden, this allows debug to be revealed with the -v option when
ansible-playbook is invoked. Debugging generally prints the value of
variables processed by the role to stdout for troubleshooting/logging.
current_user: Performs an environment lookup on the USER variable
and stores the result. This string can be used in templates and playbooks
for comments or troubleshooting, and works nicely when paired with DTG.
For example: by "{{ current_user }}" at "{{ DTG }}"
file_mode: The role generates a number of files (discussed more later),
and the permissions of the files can be set when the files are created.
The default value is 0444 which implies all users can read it, but
no one can edit it. This is not commonly overridden, but if it is, a value
of 0664 can be used to enable write permissions to the owner and group
members. This might be useful when configurations are being created
based on limited information (read: need to deliver fast) and this
tool is being used as a 90% solution. Note that post-processing changes to
output files will invalidate the machine-generated SHA-256 hashes.
make_zip: A true/false question which controls whether to archive all of
the resulting text files into a single Windows ZIP bundle. When set to yes,
all source files/folders are deleted, leaving only the final bundle.
The default value is no, which does not create a bundle and retains
all source files/folders. Answering false is an appropriate option for
testing and development to rapidly check the output files for correctness.
Answering true is an appropriate option for production deliverables.
This variable is commonly overloaded within the playbook variables.
zip_name: Optional parameter to provide an alternate, potentially more
descriptive name for the ZIP file output generated when make_zip is
true. By default, the string "archive" is used.
doc_name: When specified, the role looks for a jinja2 file with this
basename with the intent of creating a PDF document using LaTeX. The
file must be present in a playbook directory called doc_inputs/. The
utility is called pdflatex (https://www.tug.org/applications/pdftex/).
Note: Generally speaking, only a .pdf file is accessible after running
this utility, which is appropriate for distribution. When pdflatex
is invoked, it generates many intermediate files, such as:
- .toc: Table of contents
- .lot: List of tables
- .lof: List of figures
- .aux: Auxiliary compilation data
- .tex: LaTeX source file
- .out: Sectional information about outlines
keep_tex: When doc_name is specified, the .tex file generated by
the templater will be retained. This is only valuable for troubleshooting
that the jinja2 templating process succeeded before typesetting the file
into LaTeX using pdflatex. By default, this file is removed as the
end-users should never see it.
skip_sha256: A true/false question that allows an operator to skip
the hashing of output files. The value is false by default since
security should be both assumed and integration with any provisioning
system. Setting this value to true is limited to cases where it is
certain that configurations must be hand-edited after generation to meet
a specific customer demand. These cases are very rare.
scp: Nested dictionary containing two subkeys below.
- user: The SCP username that can write files to the SCP server.
- host: The SCP server FQDN/IP address. Under its root directory, a
  directory called racc/ should be created. This is where the archives
  generated by this playbook are copied.

Role variables

Given the high priority of role variables within Ansible, these are used
for constant values. Any set of known-in-advanced and seldom changing
information can be enumerated here to offload the complexity of
defining constants at the playbook level. Defining complex data
structures, like nested dictionaries, lists of dictionaries, etc ...
can significantly reduce the playbook level variable input process.

Role tasks

The tasks begin by checking that all entities have an id field. The output
is verbose and all entries are checked, even if there are failures early in
the list. The list indices (counting from 0) indicate where the failures occur,
simplifying the remediation process for the playbook designer.

Following this check, some other error checking around directory/file
presence is conducted. The main work is done by populating the following
directories with templates:

config_templates/: Place config file templates here. Each one of these
files is considered a component of an entity. If an entity contains a
router, a switch, and a firewall, there should be one template for each
in this folder. That will create a total of 3 configurations for a given
entity. Nested iteration executes the template module so that the
total number of configuration files generated is as follows:
number of entities * number of files in config_templates/
Given 3 devices in an entity across 2 entities would yield 6 files total.
infra_templates/: Place infrastructure templates here. These templates
embed their own iteration, typically iterating over the entity_list,
to write information that assists updating the infrastructure to support
the addition of these new entities. For example, if 4 new carrier POPs are
provisioned, an infra template might populate a CSV to be imported into
the AAA server to add these new systems, rather than doing it manually.
This folder is only created when infra templates are defined for a
playbook that inherits this role. This keeps the delivery bundle clean.
doc_inputs/: Place any files related to PDF generation here. At a
minimum, it should include a doc_name.j2 file which is used to build
a documentation PDF (optional). This directory should also house any images
or other content referenced by the template required for the final PDF
generation.

As it relates to playbook specific error checking, each playbook can
contribute a pre_check.yml file in the playbook directory. This is a simple
task list to be run immediate after the role-based error checking, such as
ensuring there are no duplicate id fields. If the file is absent, no
playbook specific error checking is done. Note that playbook specific error
checking can be done before or after the role inclusion, but when done before
inclusion, role variables cannot be accessed.

Additionally, an optional pre_config.yml file can be included for each
iteration of generating configuration templates. This file can be used for
additional variable declaration and processing to remove it from the templates.
This is useful for playbooks that have many templates all referencing a shared
set of common variables. If it is not specified, any variable
processing logic must be directly embedded in jinja2 templates. This
works nicely for small projects with few templates and/or few shared
variables across templates. For larger projects with more dependencies
and shared data, including a pre_config.yml file is recommended.

Output files generated by the playbook are written to the files/ folder. Each
entitys configuration files are contained in a subfolder "id_dtg". For
example, an entity with id "system1" and DTG "20171106T122919Z" would have
its configs in files/system1_20171106T122919Z/ for easy organization.

The configuration files themselves are in the format "id_template_dtg". For
example, "system1_EdgeRouter_20171106T122919Z.txt". The files are always
saved with a .txt extension for easy Windows readability and identification.

Once all the files are created, the SHA-256 checksums are calculated for each
file and rolled up into a CSV for easy viewing. This checksum CSV is included
in the bundle, but for security reasons, should also be distributed to the
end-users via some other method.

Archives are written to the archives/ folder and have a format of
bundle_dtg. For example, a templater that executes at 20171106T122919Z would
generate an archive archives/archive_20171106T122919Z.zip file, assuming
a custom zip_name was not defined.

Role handlers

Handlers for this role are only invoked when make_zip is yes an the bundle
changes. The handlers simply delete any residual folders left behind by the
archive module.

Role templates

The role only comes with one template which is used for writing SHA-256
checksums to a CSV file. This is common to any templating process and is
highly generic, and thus need not be implemented in every playbook.

Design guidance

Populating the config_templates/ and infra_templates/ directories with the
relevant templates for a given objective is the responsibility of the playbook
designer. Here are some tips:

Configuration templates typically have these attributes:
- Begin with #jinja2: newline_sequence:'\r\n' for easy Windows viewing
- Assign item[0] to a new var entity for simplified var substitution
- Do not contain iteration; this is handled by the templater tasks
- Try to match the exact text, including white space, of the device
- Embed system specific comments for easy operator troubleshooting
- Generally limited to simple variable substitution
Infrastructure templates typically have these attributes:
- Contain single-depth iteration over the entity_list
- May deal with complex CSV files or other formats as required
- Are optional. When not supplied, the role gracefully skips the process