Subplot → blog → 2020 → 01 → 18 → Iteration planning notes: January 18

What has happened

  • Lars has improved the code generator, and polished the document generator, and written the first self-tests for Subplot.

  • There is now a site for automatically generated example subplots: https://examples.subplot.liw.fi/. (Updated in 2021: moved to https://doc.subplot.liw.fi/.)

  • Overall, Subplot is starting to be ready for some careful real use. Lars will start using it at work for internal tooling.

Discussion

  • Introducing Subplot to co-workers.

    • Lars will write the simplest subplot for an internal tool that is actually useful, and present Subplot to co-workers, and then try iterate fast to make it a more impressive use of the tool.

    • The internal subplot document can be an example when giving a tutorial to co-workers.

    • Lars will also ask co-workers to read Subplot’s own documentation, and gather feedback from them.

  • Approaching other people for feedback on Subplot.

    • After Lars is back from his work trip, he and Daniel will together draft a blog post to introduce Subplot and invite people to try it for real, with the understanding that Subplot is at alpha level.

    • The goal is to find one or two people who’re willing to give Subplot a real try, and willing to tell us what they think, and report any problems they have.

  • Improving the website.

    • https://subplot.liw.fi/ is plain and boring, but that’s OK. It’s structure, what information it presents, in what order, and in what way, could be improved a lot. We feel it’s not something we’re good at.

    • In the long term, some user testing would be a good idea, but a lot of work.

    • In the short term, we’ll try to attract a contributor or two to work on the website.

    • We’ll collect questions people have that aren’t answered well on the site, in lieu of actual user testing.

  • Issue tracking, changing the software.

    • We’ll start advetising the gitlab repository for Subplot (https://gitlab.com/subplot/subplot) as the place to file issues and merge requests.

    • We’ll start filing issues there ourselves.

    • We’ll start changing code using merge requests ourselves.

    • For now, self-merging is OK. We’ll figure out more formal processes later, if and when there are more contributors, and we get more experience with gitlab.

  • Handling errors in docgen.

    • Currently docgen either fails to produce output (pdflatex not installed)¸ or embeds error messages in the output document.

    • In the medium term it’d be good to introduce the concepts of warnings and errors. Warnings get embedded in the output document, errors prevent an output document from happening. Also, the user should have a way to run docgen in a strict mode, where all warnings are errors.

  • Using pandoc_ast for parsing Markdown.

    • Both docgen and codegen are built around the Pandoc abstract syntax tree. Docgen mutates the AST to affect the output document; codegen examines the AST to extract scenarios and information from the document.

    • This is OK otherwise, but Pandoc doesn’t enable very good error messages: nodes in the AST don’t carry any information on where in the source document they come from. Especially for codegen this means the end user running the generated test program has trouble relating a test failure back to the right line in the source document.

    • On the other hand, having docgen and codegen use different parsers might result in them understanding the same input differently, and that would be a problem.

    • The current situation is good enough for alpha.

    • We can improve error messages by giving more structural locations (headings etc).

    • Pandoc may some day give better location information: https://github.com/jgm/pandoc/issues/4565.

    • The https://docs.rs/pulldown-cmark/0.6.1/pulldown_cmark/ crate for Rust would be good if we want to use another parser.

  • Document titles.

    • At the moment the document title in the YAML metadata block is optional. It is treated as the empty string, if not present.

    • For sp-filter, which is a Pandoc filter, and heavily used by Lars for websites via ikiwiki, this is the right thing to do. ikiwiki has another way to specify a page title, and requiring YAML metadata blocks is not acceptable.

    • For docgen and codegen, however, we could make it a warning or an error to not have a title, or default the title to the document filename.

  • Adding support for other languages.

    • codegen now supports Python. The output program is generated using a template, so in principle supporting other languages is straightforward.

    • Some languages, including Rust, would need codegen to support output program consisting of many files.

    • Shell would be an obvious next language to support. Adding the second language would mean adding the infrastructure for codegen to support more than one language (so the user would need to specify the language, etc). After that, more languages should only be a matter of adding a new template.

    • Other possible languages: Rust, Go, Haskell, Perl, Ruby, PHP, Cucumber/Gherkin.

  • Drawing the line between functional tests and acceptance tests

    • Acceptance tests tend to be functional tests as well, but not all functional tests are acceptance tests.

    • A test is an acceptance test is it verifies acceptance criteria by stakeholders. Brugally, if the test checks something the client demands, it’s an acceptance test.

    • Depending on how a project using Subplot structured and organized, some or all functional tests may be implemented using Subplot. Non-acceptance functional tests might be in a second subplot document, which shares the bindings and step function implementations of the acceptance test subplot.

      Subplot tries to be neutral for process and project organization.

  • Subplot marketing.

    • We have a tentative almost-offer of a marketing video for Subplot. We would like that and give the person making the offer free hands, but are available for feedback and discussion as needed.

  • Test program output.

    • At the moment the generated test program output is unstructured. This works fine, for now.

    • There are many test output formats in use in the industry. One simple on is TAP, the Test Anything Protocol.

    • Structure will be necessary once the test programs start running scenarios concurrently, but until then, the current approach will do.

  • Community building.

    • We recognize that we aren’t any good at community building.

    • We hope to start attracting real users relatively soon.

    • We hope that at least some of the users also become contributors and we welcome contributions in any form, not just code.

    • We have a bi-weekly video chat, and would be happy to open that to contributors. We regret that we don’t have the time and energy to support users via video chat.

Actions

  • Lars to introduce Subplot at work.

  • Lars to change website to point people at gitlab for issues, merge requests.

  • Lars to file issue about docgen, codegen needing to handle missing titles in a better way.

  • Lars to file issue about codegen needing to support at least shell as another language.

  • Lars to file issue that Python template should randomize order in which scenarios are run.