Support for definition lists

[Lucas-C]

Hi

I'd like to write a custom token & renderer for definition lists.

A common Markdown extension is to be create definition lists like this:

Term1
: definition1

Term2
: definition2

Inspirational specs:

• https://www.markdownguide.org/extended-syntax/#definition-lists
• https://github.com/Python-Markdown/markdown/blob/master/docs/extensions/definition_lists.md
• https://pandoc.org/MANUAL.html#definition-lists

I'd like to have support for those with mistletoe.

However I'm wondering how to do detect the block start, given that the : prefix occurs only on the seconde line,
and BlockToken start() method only accepts a single text line.

Would you have guidance regarding this, please? 🙂

[pbodnar]

Hi @Lucas-C, you are right, the current mistletoe API doesn't seem to be prepared for this use case when we need to peek at a line ahead the current one. I think the only solution is to change the start() method and pass another parameter lines to it. I.e. something like:

-def start(line):
+def start(line: str, lines: FileWrapper):

But we need to be cautious when doing this, because:

1. The lines object is crucial for functioning of the tokenize_block() method, so we need to make sure its state will be correct after start() method finishes.
2. There is always the question of performance which might be affected by this change.

Do you think you could give it a try yourself and test how it goes?

[Lucas-C]

Do you think you could give it a try yourself and test how it goes?

Sure!

I started a draft PR: #236

It's far from finished yet, but you can already check how I suggest to support both start(line) & start(line, lines) in block_tokenizer.tokenize_block(), and the new FileReader class I introduced to address the 1st point you mentioned.

[Lucas-C]

I think PR #236 is ready for a first review.

I think it's just missing code in the Latex renderer, but I'd like to have your review all the code already written before adding that, if it's Ok with you? 🙂

[pbodnar]

@Lucas-C, thanks for the work. Basically, it looks promising, although there are still problems to cope with. But could you please firstly do the following?

1. Split the PR into 2 - 1st to support peeking ahead of the current line, 2nd to add support for the definition lists. It looks like the 2nd one can be challenging on its own and it is also much harder here on GitHub to review bigger PRs that cover multiple topics.
2. Check the performance impact of the individual PRs - an example how to measure it.

[Lucas-C]

Alright, I opened this other PR, that should merged first: #238