Swift matmul odd results? (Error in notebook?)

Part 2 (2019)
1

The code for the swift matmul in the matmul_01 notebook was:

// a and b are the flattened array elements, aDims/bDims are the #rows/columns of the arrays.
func swiftMatmul(a: [Float], b: [Float], aDims: (Int,Int), bDims: (Int,Int)) -> [Float] {
assert(aDims.1 == bDims.0, “matmul shape mismatch”)

var res = Array(repeating: Float(0.0), count: aDims.0 * bDims.1)
for i in 0 ..< aDims.0 {
    for j in 0 ..< bDims.1 {
        for k in 0 ..< aDims.1 {
            res[i+aDims.0*j] += a[i+aDims.0*k] * b[k+bDims.0*j]
        }
    }
}
return res

}

I ended up writing this:

func myMatmul(a: [Float], b: [Float],aDims: (Int,Int),bDims: (Int,Int)) -> ([Float],(Int,Int)) {
var res = Array(repeating: Float(0.0), count: aDims.0*bDims.1)
for i in 0…<aDims.0 {
for j in 0…<bDims.1 {
for k in 0…<aDims.1 {
res[i * bDims.1 + j]+=a[i * aDims.1 + k] * b[k * bDims.1 + j]
}
}
}
return (res,(aDims.0,bDims.1))
}

Obviously these are different, and it seems to be what we switching columns/rows. So I multiplied this by an array of all ones to make sure my hypothesis of this being an understanding of rows/columns was correct:

let (aDims,bDims) = ((5, 784), (784, 10))
let flatA = xTrain[0…<5].scalars//Array(repeating: Float(1.0), count: aDims.0 * aDims.1)
let flatB = Array(repeating: Float(1.0), count: bDims.0 * bDims.1)//weights.scalars
print(myMatmul(a:flatA,b:flatB,aDims:aDims,bDims:bDims))
print(swiftMatmul(a:flatA,b:flatB,aDims:aDims,bDims:bDims))

Here are the results:

([25.925554, 25.925554, 25.925554, 25.925554, 25.925554, 25.925554, 25.925554, 25.925554, 25.925554, 25.925554, 38.106697, 38.106697, 38.106697, 38.106697, 38.106697, 38.106697, 38.106697, 38.106697, 38.106697, 38.106697, -33.34089, -33.34089, -33.34089, -33.34089, -33.34089, -33.34089, -33.34089, -33.34089, -33.34089, -33.34089, 2.2452376, 2.2452376, 2.2452376, 2.2452376, 2.2452376, 2.2452376, 2.2452376, 2.2452376, 2.2452376, 2.2452376, -0.38203776, -0.38203776, -0.38203776, -0.38203776, -0.38203776, -0.38203776, -0.38203776, -0.38203776, -0.38203776, -0.38203776], (5, 10))
[13.410015, 25.3511, 6.9443583, 4.9880714, -18.138931, 13.410015, 25.3511, 6.9443583, 4.9880714, -18.138931, 13.410015, 25.3511, 6.9443583, 4.9880714, -18.138931, 13.410015, 25.3511, 6.9443583, 4.9880714, -18.138931, 13.410015, 25.3511, 6.9443583, 4.9880714, -18.138931, 13.410015, 25.3511, 6.9443583, 4.9880714, -18.138931, 13.410015, 25.3511, 6.9443583, 4.9880714, -18.138931, 13.410015, 25.3511, 6.9443583, 4.9880714, -18.138931, 13.410015, 25.3511, 6.9443583, 4.9880714, -18.138931, 13.410015, 25.3511, 6.9443583, 4.9880714, -18.138931]

Notice that my results repeat in groups of 10 5 times, and the notebook’s results repeat the 5 different results 10 times [13.410015, 25.3511, 6.9443583, 4.9880714, -18.138931]

Admittedly I have gotten a bit confused on what the proper row/column configuration here is… as a thought the shape was (aDims:(row:5,column:784),bDims:(row:784,column:10)) = results:(row:5,column:10)

To me the class notebook seems to have switched the deminsions of the rows/columns for A and B, otherwise multiplying by an array of 1s should have created the pattern you see in mymatmul. I could of course have gotten my understanding of the deminsions mixed up as well, but I think I am right?

3 Likes
2

I’m going to agree with you. The scalars property returns the tensor scalars in row-major order and it appears that the swift notebook is coded as if the arrays are in column-major order. A simple example comparing your results and the swift notebook to what the correct values does seem to verify this.

let tensor1: Tensor<Float> = [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0], [10.0, 11.0, 12.0]]
print(tensor1.scalars)
// output is -> [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0]

let tensor2: Tensor<Float> = [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]]
print(tensor2.scalars)
// output is -> [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]

print(swiftMatmul(a: tensor1.scalars, b: tensor2.scalars, aDims: (4,3), bDims: (3, 3))
// output is -> [38.0, 44.0, 50.0, 56.0, 83.0, 98.0, 113.0, 128.0, 128.0, 152.0, 176.0, 200.0]

print(myMatmul(a: tensor1.scalars, b: tensor2.scalars, aDims: (4,3), bDims: (3,3)))
// output is -> ([30.0, 36.0, 42.0, 66.0, 81.0, 96.0, 102.0, 126.0, 150.0, 138.0, 171.0, 204.0], (4, 3))

The myMatmul output is the correct output.

1 Like
3

I didn’t check which way swift ordered the scalars, since we’re not really using the function for anything. Sorry if I picked the wrong one!

2 Likes
4

Thanks for weighing in! I was not exactly sure what happened as I found the code a bit hard to understand with the aDim/bDim piece. I’ll go ahead and write up a pull request then.

1 Like
5

Going to reference this notebook to make the changes clear but here we go:

broadcastMatmult(m1, m2)
[[ 0.61743975, 4.8894544, -8.850632, 35.932926, 7.6370225, -16.770311, 24.500923,
18.64631, 6.261522, -2.0731914],
[ 19.98177, -2.6094093, -42.0761, 31.599257, -46.588505, -52.206516, 59.992134,
63.70806, -16.270977, -13.350272],
[ 39.885983, 29.144627, -35.939686, -5.2272143, 4.8716965, -34.661488, 9.516182,
-44.97263, -12.7907715, -57.22954],
[ -15.486622, -5.7996454, 0.94989705, 19.238352, 29.101627, -3.7798738, -44.72618,
-23.750109, -38.088398, -11.281395],
[ -9.478125, 73.61498, -24.822008, 35.457195, -2.0014482, 0.56936836, -16.792034,
-8.382481, 58.689396, -12.8543005]]

Next mymatmul (implmented in the notebook’s results on same matrices:

swiftMatmulUnsafe(a:m1.scalars,b:m2.scalars,aDims: (5, 784),bDims: (784, 10))
50 elements

  • 0 : 0.61743975
  • 1 : 4.8894544
  • 2 : -8.850632
  • 3 : 35.932926
  • 4 : 7.6370225
  • 5 : -16.770311
  • 6 : 24.500923
  • 7 : 18.64631
  • 8 : 6.261522
  • 9 : -2.0731914
  • 10 : 19.98177

One that was originally in notebook:

swiftOldMatmulUnsafe(a:m1.scalars,b:m2.scalars,aDims: (5, 784),bDims: (784, 10))
50 elements

  • 0 : -4.86545
  • 1 : 4.7959633
  • 2 : -1.4238247
  • 3 : -26.279905
  • 4 : 6.6342196
  • 5 : 11.465586
  • 6 : 13.487977
  • 7 : -53.683453
  • 8 : -16.127754
  • 9 : -26.66617
  • 10 : -40.300545

Pull Request for fix here: https://github.com/fastai/fastai_docs/pull/126