Integration of FastPages with Rmarkdown, Emacs Org mode, Observable

I have a complain to make. Every time I come back to this forum . I discover some wonderful new software to play with which eats away all my time which I need to spend in project. With lock-down in place it is especially hard not to get distracted for hacking into these tools and concentrate on our implementation projects.

Recently I discovered fastpages which is really a clever piece of work capitalizing on Github actions and nbdev which fired my imagination with all the wonderful possibilities. It can do notebooks the way I like , capitalize on markdown and even use word documents all in a fairly automated way.

I quickly put together a blog available here:
https://soliloquium.netlify.app/

A few things I discovered while looking around the forum were hacks to enable plotly rendering and integrating javascript which I captured in my brand new blog here https://soliloquium.netlify.app/jupyter/plotly/visualization/2020/04/24/plotlyrendering

There are a few things which I discovered myself

Now the solution is nearly perfect as it is.

As suggested multiple times in this forum , blogging is a great way to learn things . I intend to use it for the same purpose. This solution would be perfect for me if it can also integrate rmarkdown and org-mode in a more robust way.

Ideas around implementation:

  1. My current solution with org-mode suffers from few issues:

    • Hyperlinks on Table of Content do not work. Reason for this is correct id’s are not created for different sections. There are two ideas to fix it
      - Fix the jekyll-org converter, which is outside my scope ( don’t want to learn ruby)
      - Hack toc.html to fix it for now. However various recommendations on forum suggests not to touch layout and theme , as it might break the upgrade workflow.

A more robust solution would be to capitalize on github actions and emacs org mode publish implementation. We can run commands in docker by using emacs with --script tag. I would love to do this myself but need some guidance on how to approach this. I havenot done a lot of open source contributions or used github actions.

  1. Rmarkdown --> jekyll . From the research I can see most of the solution already exists in form of “knitr”, “blogdown” etc… . I am wondering how much work it would be to come up with combination of docker and github actions to enable the same. I am just starting with R so have limited knowledge on these areas.

I would love to get some feedback and guidance for the above from people in this forum.

Would love to support Rmarkdown

fastpages is built on top of nbdev, which is tightly coupled with Jupyter notebooks and a utility called nbconvert, however maybe it’s possible to emulate the same thing for Rmarkdown? I am not sure there is an easy way to do this? Another option is to forget nbdev completely and just rely on RMarkdown itself

Rmarkdown support would play really well into another flavor of fastpages that is around sharing knowledge like Airbnb’s Knowledge Repo https://github.com/airbnb/knowledge-repo which is focused around sharing knowledge within companies (rather than blog posts)

Anyways I don’t have all the answers those are some initial thoughts

1 Like

I agree. I would rather keep rmarkdown separate and tools designed in R to handle the parsing/conversion part. That way we can capitalize on Rstudio without going back and forth while writing the post. My feeling is we can do so easily if we can somehow leverage rstudio as the daemon and / or some package designed in R (knitr?) to export ‘.Rmd’ files stored in _rmarkdown folder and converted to html/ md and posted in _post folder. But I am not sure . Here is a blogpost where someone has converted Rmd to md.

http://brooksandrew.github.io/simpleblog/articles/blogging-with-r-markdown-and-jekyll-using-knitr/

Since , I am just starting with R. I don’t know how to adapt this effectively to a script which we can use outside Rstudio. With this we may not get all the features, for show/ hiding code etc… But may be we can gradually adapt it

@hamelsmu

I now have a robust solution for org mode. I have captured all my changes in this blog post
Blogging with Org Mode and FastPages

Mainly it includes

  • Additional converter included as a separate docker action in ci.yaml
  • Some changes in docker-compose and Makefile to enable local conversion

These changes are fairly minimal it can be integrated easily to main branch without affecting other portions. Please let me know what do you think
Regards,
Rahul

Why do we need to enable org mode? I’m afraid this would be a dramatic shift from marking up files with nbdev that folks are used to?

If we had both ways to choose from the surface area of what must be maintained would be greatly expanded and I’m not sure I have the resources for that, you would have to test every feature of fastpages against this new target markup language

The nice thing about nbdev is it is actively maintained and fastpages can leverage that. I’m not sure adding a second markup language is sustainable from a maintenance perspective

I’m open minded about it and willing to discuss let me know

I think before I feel comfortable introducing a different markup language I need CI to test every feature of fastpages when the page is rendered

Not sure how to do that as I’m not a web developer but you would have to add this first to make any additional language sustainable from a maintenance perspective. Therefore if you are really passionate about this I would focus on automatically testing what we already have and then think about adding something new when that’s nailed down

I think your point of view is reasonable. I have a working setup now in one of the forks https://github.com/Rahuketu86/fastpages which seems to work atleast on the surface.

  1. I have done minimal setup to fastpages master. With the way things are configured following things work
  • You can use the this as template repository similar to fastpages. [ I didn’t have to change anything on master to make that work]

  • I have not tested update workflow but it should largely work. As my custom actions are in different folder compared to master’s action files.

  • My main changes which collide are following

    • ci.yaml where I add my action for org converter service

    • docker_compose.yml and Makefile to add additional container and docker service.

  • As far as layout is concerned. I have not made any changes there as I generate same Liquid tags for links and boxes.

  • Layout and syntax highlight should largely mark as

    • I use {%% highlight %%} to override default org mode colouring
    • All the other styling is managed by css which is on jekyll side .
  • YAML frontmatter is exactly same as markdown/ notebook so there should be no collision on that side.

What is currently different:-

  • Code folding is currently implemented on all the code blocks.Even though I use same mechanism of wrapping it into detail tag as notebook. I have yet to implement control to enable / disable the same. Approach I have taken is to have semantics native to org mode rather than markdown. In org mode you can set properties on export block (src code or example). I am looking into details to implement the same.
  • There are some org mode specific spans which are exported by native markers. I will have to define some additional custom css to make them look nicers (eg: TODO tag in my blog)
  • Links inside boxes are not working . Ideally should be a simple fix if I can figure out the regex for it.
  • There might be more features which I have currently not encountered which might need fixing.

CI for UI testing

  • I am not sure how to do it myself . What is the current strategy with nbdev/ fastpages? How can we define a minimum set of features which need to pass the acceptance criteria. Even if we don’t have exactly same html [div/ spans] if it is functionally equivalent it should not matter.

  • I also agree with maintenance part. I am happy to continue with my fork as long as we can resolve some of these issues.

  • At some point I also want to attack Rmarkdown in similar fashion. R for now will work on org mode but rmarkdown in Rstudio is extremely nice. Currently I am looking at this one http://brendanrocks.com/htmlwidgets-knitr-jekyll/

  • My strategy is not to reimplement entire nbdev but use native tools generating html and modify them enough to generate output which is functionally close to jekyll markdown/ html and liquid template used in fastpages. If I get some additional features available in R / org mode which are not possible on python side, I try to best match the structure of fastpages so there is no collision.

Rahul
PS - Thank you for your feedback. I think it’s worthwhile to keep this discussion open public forum for others to review and comment.

I am not sure, and I think will require a significant investment of time

I am not sure I can promise we will be able to ultimately merge your fork back into fastpages, as we have to first find a way to minimize the maintenance burden before increasing the surface area like this. Again I would have to do some significant research on how to test user interfaces, etc. on rendered pages. I don’t know how to do this.

Will leave this discussion open incase anyone else wants to volunteer their thoughts.