Fast.ai APL study session 9

Fast.ai APL study session 9

Fast.ai APL study session 9

Discussion of the 9th study group can go here. Overview

[ <<< session 8 | [session 10 >>>]]

This is a wiki post - feel free to edit to add links from the lesson or other useful info.

Overview

Study session resources

Study session links

Study session covered:

  • loop
  • Atop vs Beside vs Over functions
  • unique function (unique mask + replicate)
  • inverse function
  • fill element
  • for reading Dyalog documentation (f=⍺⍺, g=⍡⍡, X=⍺, Y=⍡, R=Result)
2 Likes

The inverting power (pun intended) of ⍣ is much greater than you demonstrated. It can both do non-mathematical things and seemingly impossible things.

The following function takes a three-letter word, reverses it, puts a space between the first and second letters, then prepends the letters xyz:

      f←'xyz',1 0 1 1\⌽
      f 'ABC'
xyzC BA

If we apply the inverse function on 'xyzC BA' we should get back our 'ABC', right?

      (f⍣¯1)'xyzC BA'
ABC

It is seemingly impossible to solve 2x5+x4+5x3βˆ’2x2βˆ’6x=100, however:

      f←(2Γ—*∘5)+(*∘4)+(5Γ—*∘3)+(Β―2Γ—*∘2)+(Β―6βˆ˜Γ—)
      (f⍣¯1)100
2
      f 2
100

That’s one valid value for x.

2 Likes

You were trying to use ⍨ for removing the double mention of v in v/⍨≠v. You can indeed do that, as you do want v as both left and right argument to /⍨. However, the right argument version of v much be preprocessed by β‰ . β€œPre-process right”? That’s f∘g:

      v←22 10 22 22 21 10 5 10
      /β¨βˆ˜β‰ β¨v
22 10 21 5
      Uniq←/β¨βˆ˜β‰ β¨
      Uniq v
22 10 21 5
1 Like

Another example of monadic use of ⍨

x ← 1 2 3
⍝ The following three are equivalent and compute the array divided by its length
x ÷ (⍴ x)  
x ÷∘⍴ x
÷∘⍴⍨ x
⍝  Now we can use the above to compute the mean of an array 
mean ← +/∘(÷∘⍴⍨)

Or investigate trains:

mean ← +/∘(÷∘⍴⍨)
mean ← +/(÷∘⍴⍨)    ⍝ atop: f g ←→ f∘g
mean ← +/÷∘⍴⍨      ⍝ simplify: f ←→ (f)
mean ← +/÷⍴        ⍝ distributivity: (f g h)Y ←→ (f Y)g(h Y)
1 Like

Thanks. very cool. But the last one not equivalent and does not compute the mean of array, right?

+/÷⍴ 1 2 3  ⍝ this is just 0.33 

You have to put it in parentheses if you don’t assign it to a name first. You’ll see that @abrudz assigns it to a name.

1 Like

Thanks. trains seem useful (but not obvious). It may be a good concept to cover in the course. Maybe you’ve covered it already. I’ve only watched up to middle of the 10th session.

As much as I enjoy APL (and your way of teaching it, in particular), I find it more like a cute and powerful puzzle framework than a practical programming language. As you said in ArrayCast podcast, I wish it was available as a module in python and swift to be used the same way RegExp are used within the language.

I’m fairly confident that in a few months I’ll have a hard time making sense of APL expressions that I have written.

It’s a real shame that some of the best and most paradigm changing languages come across like so. Maybe it’s because the current emphasis seems to be on winning obfuscated code contests and code golf competitions and making expressions as terse is possible (which, naturally, would scare the bajeezas out of mere mortals) but I think in its heyday, APL was used by regular office workers to do regular work. I think maybe because in those days of submitting jobs to the mainframes, your other options were COBOL, BASIC, PL1 or other beasts.

I found this page by a long time APL’er (Jerry Brennan) whose APL tutorial seems to cover more down to earth day to day problems. I also find Jerry’s tutorial quite approachable for myself (I’m the eternal n00b unfortunately).

I mean, it’s good that some people have the advanced mental capacities to understand the rarified mathematical nuances of each operation, but some of us yerts just want to get some run of the mill stuff accomplished (like reading data in, cleaning it, maybe doing some EDA … ) etc. etc.

Maybe it’s the nature of the beast? The more powerful and magical the language is, the bigger the probability that the practitioners of this β€œmagical art” will become separated from the masses. I’ve seen it in Lisp and Forth as well, but APL is a totally new experience because the language is literally executable Math! I honestly feel like I need to know proper math before I an even use it for regular stuff.

I listen to the arraycast and I understand I’m not the intended audience, but most of the time, I can appreciate that this is great content, but at the same time, I also frequently find myself saying , β€œI have no frikking idea what these dudes are talking about” :joy:

At this stage we’re focused on getting thru all the glpyhs, so no we haven’t done trains yet. I expect after we’ve finished the glyphs we’ll do a session or two on tacit/point-free programming, including glyphs.

1 Like

No-one can remember things that they don’t practice retrieving. So yes, if you don’t keep working on it, you’ll forget. Try using Anki with cards for all the glyphs you use, and for checking your understanding of some important combinations and concepts.

As you said in ArrayCast podcast, I wish it was available as a module in python and swift to be used the same way RegExp are used within the language.

You can use Dyalog’s pynapl that way: GitHub - Dyalog/pynapl: Dyalog APL ←→ Python interface – see Rodrigo’s talk from Dyalog '21:

https://dyalog.tv/Dyalog21/?v=gOUFXBUMv_A

2 Likes

but some of us yerts just want to get some run of the mill stuff accomplished (like reading data in, cleaning it, maybe doing some EDA … ) etc. etc.

I’ve gradually over time rewritten all my boring β€œwork” stuff from Python to APL – an unholy bunch of utility scripts that typically gather data from a set of disparate sources (JSON-over-HTTP APIs mostly), massages, cleans, combines and otherwise aggregates it and spits out some report or other as json or csv. I built these as ]usercommands inside Dyalog, and to me it came as somewhat of a surprise both how nice it was to use APL this way, a million miles away from the cut and thrust of the code golfer’s APL that we often see on display. The other realisation was that having created easy ways to get my data into Dyalog, I’m much faster when it comes to cook up some combination that I hadn’t scripted for already. My codebase easily halved in size, even when writing APL in an expansive style.

I used to scoff at the tradfn-style of APL, but when you’re in β€œget stuffs done” mode, it can be super-helpful. APL (IMHO) is an excellent tool for doing boring data processing.

5 Likes

I totally agree, there’s something to be said about empowering the masses with easily approachable computation! BTW, I’m loving your current book, thanks for making it available to us all! maybe your next book could be β€œDo the boring stuff with APL” :smiley:

1 Like

I am very interested in knowing more about your workflow.
for example, do you get the JSON using python (using http requests) and then somehow feed it to your APL code and then use feed the output back to python? Do you use pynapl for doing this?

HttpCommand β€” Learning APL covers this - you can do http requests + json processing in APL just fine

2 Likes

Thanks. This is very helpful.
I installed pynpal. But from pynapl import APL asks for a password. I’m sure I’m doing something silly

Sure. So the first thing I had to wrap my head around was the concept of Dyalog user commands – at first glance, this (like many other APL things) seem utterly backwards. Normally, I’d expect to interact with scripts I write in the unix shell, and enter switches or sub-commands as parameters. Doing this with APL in a reasonably portable way was until recently at best awkward or at worst hopeless.

Writing Dyalog user commands sort of inverts this pattern: you use Dyalog as the β€œshell” and have it manage configuration and parameter passing, and simply write functions for the guts of your business logic. For this to be smooth, some of it works best as using the trad form, but this is mostly a wrapper.

@abrudz has a nice video on how to write your own user commands, and this is definitely the place to start if this sounds interesting. But basically, code you place in the directory ~/MyUCMDs becomes available inside Dyalog as user commands. Below is a snippet of some of mine. The List and Run functions basically does the command-line processing, and in this example I’ve left in one of my β€œbusiness logic” functions, called defaults, which fetches a json document, does a bit of grouping and reformatting. The weird function dotcreds just grabs the relevant credentials from a file.

To use this, I simply type ]CLUSTERS.defaults into the Dyalog session.

As @rak1507 pointed out, I did write some stuff on how to do HTTP in Learn APL, and also a bit on json and friends. The HttpCommand http client lib is excellent.

:Namespace CLUSTERS
    βŽ•IO←0 β‹„ βŽ•ML←1
    hcβ†βŽ•SE.SALT.Load'HttpCommand' ⍝ Dyalog HTTP client

    base←'https://URL_HERE/' 

    dotcreds←{βŽ•JSON⍠'Dialect' 'JSON5'⊒2⌽ '}}{"":{' , ∊'^;.*' '^\[(.*)]' '^(.*?)\s*=\s*(.*)'βŽ•R'' '},"\1":{' '"\1":"\2",' ⊒ βŠƒβŽ•NGET ⍡}

    creds←dotcreds '/Users/stefan/.creds'
    hd←1 2⍴'Authorization' (βŠƒ,/'Basic '(hc.Base64Encode creds.section.user,':',creds.section.password))

    βˆ‡ r←defaults ;url;regions;names ⍝ Niladic; return cluster data for all region defaults
      url←base,'broker-config/cloud-production' ⍝ Single doc
      reg←(_←hc.GetJSON 'GET' url '' hd).Data.regions
      r←get (βŠƒ,/βˆͺ(reg⍎¨reg.βŽ•NLΒ―9).provisioning_cluster)
    βˆ‡
    ⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝
    ⍝ stuff removed for brevity
    ⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝
    βˆ‡ r←List
      rβ†βŽ•NSΒ¨20β΄βŠ‚β¬
      r[0].(Name Group Desc Parse)←'Cluster'      'CLUSTERS' 'Cluster name'   '1S'
      r[1].(Name Group Desc Parse)←'Defaults'     'CLUSTERS' 'Defaults'       ''
      r[2].(Name Group Desc Parse)←'Plans'        'CLUSTERS' 'Plans'          ''
      r[3].(Name Group Desc Parse)←'List'         'CLUSTERS' 'List'           ''
      r[4].(Name Group Desc Parse)←'Plan'         'CLUSTERS' 'Plan name'      '1S'
    ⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝
    ⍝ stuff removed for brevity
    ⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝
    βˆ‡

    βˆ‡ r←Run(Cmd Input)
      r←⍬
      :Select Cmd
      :Case 'Cluster'
        rβ†βŠƒget Input.Arguments
      :Case 'Defaults'
        r←defaults
    ⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝
    ⍝ stuff removed for brevity
    ⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝
      :Else
        rβ†βŠ‚'Unrecognised command'
      :EndSelect
    βˆ‡

:EndNamespace
1 Like

pynapl is in the process of being modernised; the maintainer goes by the moniker @rgs on the apl orchard, but I’ve not seen them here yet.