I have a complain to make. Every time I come back to this forum . I discover some wonderful new software to play with which eats away all my time which I need to spend in project. With lock-down in place it is especially hard not to get distracted for hacking into these tools and concentrate on our implementation projects.
Recently I discovered fastpages which is really a clever piece of work capitalizing on Github actions and nbdev which fired my imagination with all the wonderful possibilities. It can do notebooks the way I like , capitalize on markdown and even use word documents all in a fairly automated way.
As suggested multiple times in this forum , blogging is a great way to learn things . I intend to use it for the same purpose. This solution would be perfect for me if it can also integrate rmarkdown and org-mode in a more robust way.
Ideas around implementation:
My current solution with org-mode suffers from few issues:
Hyperlinks on Table of Content do not work. Reason for this is correct idās are not created for different sections. There are two ideas to fix it
- Fix the jekyll-org converter, which is outside my scope ( donāt want to learn ruby)
- Hack toc.html to fix it for now. However various recommendations on forum suggests not to touch layout and theme , as it might break the upgrade workflow.
A more robust solution would be to capitalize on github actions and emacs org mode publish implementation. We can run commands in docker by using emacs with --script tag. I would love to do this myself but need some guidance on how to approach this. I havenot done a lot of open source contributions or used github actions.
Rmarkdown --> jekyll . From the research I can see most of the solution already exists in form of āknitrā, āblogdownā etcā¦ . I am wondering how much work it would be to come up with combination of docker and github actions to enable the same. I am just starting with R so have limited knowledge on these areas.
I would love to get some feedback and guidance for the above from people in this forum.
fastpages is built on top of nbdev, which is tightly coupled with Jupyter notebooks and a utility called nbconvert, however maybe itās possible to emulate the same thing for Rmarkdown? I am not sure there is an easy way to do this? Another option is to forget nbdev completely and just rely on RMarkdown itself
Rmarkdown support would play really well into another flavor of fastpages that is around sharing knowledge like Airbnbās Knowledge Repo https://github.com/airbnb/knowledge-repo which is focused around sharing knowledge within companies (rather than blog posts)
Anyways I donāt have all the answers those are some initial thoughts
I agree. I would rather keep rmarkdown separate and tools designed in R to handle the parsing/conversion part. That way we can capitalize on Rstudio without going back and forth while writing the post. My feeling is we can do so easily if we can somehow leverage rstudio as the daemon and / or some package designed in R (knitr?) to export ā.Rmdā files stored in _rmarkdown folder and converted to html/ md and posted in _post folder. But I am not sure . Here is a blogpost where someone has converted Rmd to md.
Since , I am just starting with R. I donāt know how to adapt this effectively to a script which we can use outside Rstudio. With this we may not get all the features, for show/ hiding code etcā¦ But may be we can gradually adapt it
Additional converter included as a separate docker action in ci.yaml
Some changes in docker-compose and Makefile to enable local conversion
These changes are fairly minimal it can be integrated easily to main branch without affecting other portions. Please let me know what do you think
Regards,
Rahul
Why do we need to enable org mode? Iām afraid this would be a dramatic shift from marking up files with nbdev that folks are used to?
If we had both ways to choose from the surface area of what must be maintained would be greatly expanded and Iām not sure I have the resources for that, you would have to test every feature of fastpages against this new target markup language
The nice thing about nbdev is it is actively maintained and fastpages can leverage that. Iām not sure adding a second markup language is sustainable from a maintenance perspective
Iām open minded about it and willing to discuss let me know
I think before I feel comfortable introducing a different markup language I need CI to test every feature of fastpages when the page is rendered
Not sure how to do that as Iām not a web developer but you would have to add this first to make any additional language sustainable from a maintenance perspective. Therefore if you are really passionate about this I would focus on automatically testing what we already have and then think about adding something new when thatās nailed down
I think your point of view is reasonable. I have a working setup now in one of the forks https://github.com/Rahuketu86/fastpages which seems to work atleast on the surface.
I have done minimal setup to fastpages master. With the way things are configured following things work
You can use the this as template repository similar to fastpages. [ I didnāt have to change anything on master to make that work]
I have not tested update workflow but it should largely work. As my custom actions are in different folder compared to masterās action files.
My main changes which collide are following
ci.yaml where I add my action for org converter service
docker_compose.yml and Makefile to add additional container and docker service.
As far as layout is concerned. I have not made any changes there as I generate same Liquid tags for links and boxes.
Layout and syntax highlight should largely mark as
I use {%% highlight %%} to override default org mode colouring
All the other styling is managed by css which is on jekyll side .
YAML frontmatter is exactly same as markdown/ notebook so there should be no collision on that side.
What is currently different:-
Code folding is currently implemented on all the code blocks.Even though I use same mechanism of wrapping it into detail tag as notebook. I have yet to implement control to enable / disable the same. Approach I have taken is to have semantics native to org mode rather than markdown. In org mode you can set properties on export block (src code or example). I am looking into details to implement the same.
There are some org mode specific spans which are exported by native markers. I will have to define some additional custom css to make them look nicers (eg: TODO tag in my blog)
Links inside boxes are not working . Ideally should be a simple fix if I can figure out the regex for it.
There might be more features which I have currently not encountered which might need fixing.
CI for UI testing
I am not sure how to do it myself . What is the current strategy with nbdev/ fastpages? How can we define a minimum set of features which need to pass the acceptance criteria. Even if we donāt have exactly same html [div/ spans] if it is functionally equivalent it should not matter.
I also agree with maintenance part. I am happy to continue with my fork as long as we can resolve some of these issues.
At some point I also want to attack Rmarkdown in similar fashion. R for now will work on org mode but rmarkdown in Rstudio is extremely nice. Currently I am looking at this one http://brendanrocks.com/htmlwidgets-knitr-jekyll/
My strategy is not to reimplement entire nbdev but use native tools generating html and modify them enough to generate output which is functionally close to jekyll markdown/ html and liquid template used in fastpages. If I get some additional features available in R / org mode which are not possible on python side, I try to best match the structure of fastpages so there is no collision.
Rahul
PS - Thank you for your feedback. I think itās worthwhile to keep this discussion open public forum for others to review and comment.
I am not sure, and I think will require a significant investment of time
I am not sure I can promise we will be able to ultimately merge your fork back into fastpages, as we have to first find a way to minimize the maintenance burden before increasing the surface area like this. Again I would have to do some significant research on how to test user interfaces, etc. on rendered pages. I donāt know how to do this.
Will leave this discussion open incase anyone else wants to volunteer their thoughts.
Life took over with a new born, job , covid and an executive mba. Just trying to get back into blogging mode and wondering if I should give nbdev2 + quarto a try. I did write some extensions to link fastpages to notion and telegram. Guess I spent much more time preparing to blog then actual blogging . I also want to upgrade entire emacs stack to doom config.So migrating entire stack needs some time .Nonetheless happy to share my orgmode code / old config if you so desire( repo is private but I can send you org files)
Hi - hey, huge congrat on all those big life changes.
I too am lurking around the edges of quarto but in the meantime Iād be quite happy to have a gander at your orgmode code. malcolm dot cook at gmail dot com is me.
Good luck moving to doom. Iāve done it for others but for myself stay old-school. Maybe again somedayā¦
If youāre interested, Iāve got a recipe for building emacs on centos 7 configured as:
. I also want to upgrade entire emacs stack to doom config.So migrating entire stack needs some time .Nonetheless happy to share my orgmode code / old config if you so desire( repo is private but I can send you org files)