How could we implement GeneralRelu from lesson 10 or 11 in Swift, I tried the following
public struct GeneralReLU {
public var leakOptional: Float? // slope for X when it's negative
public var subOptional: Float? // value used to substract from final result
public var maxvOptional: Float? // maximum value
public init(_ leak: Float? = nil, _ sub: Float? = nil, _ maxv: Float? = nil) {
self.leakOptional = leak
self.subOptional = sub
self.maxvOptional = maxv
}
@differentiable
func applied(to input: Tensor<Float>) -> Tensor<Float> {
var output = max(input, 0)
if let leak = self.leakOptional {
output += min(input, 0) * leak
}
if let sub = self.subOptional {
output -= sub
}
if let maxv = self.maxvOptional {
output = min(output, maxv)
}
return output
}
}
But I’m getting this error
error: <Cell 27>:14:6: error: function is not differentiable
@differentiable
~^~~~~~~~~~~~~~
<Cell 27>:15:10: note: when differentiating this function definition
func applied(to input: Tensor<Float>) -> Tensor<Float> {
^
<Cell 27>:17:28: note: differentiating control flow is not supported yet
if let leak = self.leakOptional {
^
One simple approach would be to avoid nil by using some defaults, e.g. default to 0 for leak, and to Int.Max for maxv, and to 0 for sub. Then you don’t need any conditionals.
public struct GeneralReLU {
public var leak: Float // slope for X when it's negative
public var sub: Float // value used to substract from final result
public var maxv: Float // maximum value
public init(_ leak: Float = 0.0, _ sub: Float = 0.0, _ maxv: Float =
Float.greatestFiniteMagnitude) {
self.leak = leak
self.sub = sub
self.maxv = maxv
}
@differentiable
func applied(to input: Tensor<Float>) -> Tensor<Float> {
return min(max(input, 0) + (min(input, 0) * leak) - sub, maxv)
}
}
error: <Cell 26>:14:6: error: function is not differentiable
@differentiable
~^~~~~~~~~~~~~~
<Cell 26>:15:10: note: when differentiating this function definition
func applied(to input: Tensor<Float>) -> Tensor<Float> {
^
<Cell 26>:19:20: note: cannot differentiate an external function that has not been marked '@differentiable'
return min(max(input, 0) + min(input, 0) * self.leak - self.sub, self.maxv)
^
error: <Cell 26>:14:6: error: function is not differentiable
@differentiable
~^~~~~~~~~~~~~~
<Cell 26>:15:10: note: when differentiating this function definition
func applied(to input: Tensor<Float>) -> Tensor<Float> {
^
<Cell 26>:19:20: note: expression is not differentiable
return min(max(input, 0) + min(input, 0) * self.leak - self.sub, self.maxv)
^
min(_:_:) and max(_:_:) that take a scalar on one side are defined here:
They are not marked @differentiable yet, but they should be. We’ll make sure it’s part of the next release!
In the meantime, the variants of min(_:_:) and max(_:_:) that take Tensors on both sides do have a derivative. As a workaround, you can turn scalars into a Tensor first.
it turns out having everything in one I got this error
error: <Cell 27>:19:16: error: the compiler is unable to type-check this expression in reasonable time; try breaking up the expression into distinct sub-expressions
return min(max(input, Tensor(0)) + min(input, Tensor(0)) * self.leak - self.sub, Tensor(self.maxv))
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I turned the expression into the following and now it’s not complaining any more
let positives = max(input, Tensor(0))
let leaked = min(input, Tensor(0)) * self.leak
return min(positives + leaked - self.sub, Tensor(self.maxv))
@rxwei Collab kernel is reliably crashing when I make GeneralReLU extend Layer like this
struct GeneralReLU: Layer {
var leak: Float // slope for X when it's negative
var sub: Float // value used to substract from final result
var maxv: Float // maximum value
init(_ leak:Float = 0.0, _ sub:Float = 0.0, _ maxv:Float = Float.greatestFiniteMagnitude) {
self.leak = leak
self.sub = sub
self.maxv = maxv
}
@differentiable
func applied(to input: Tensor<Float>, in context: Context) -> Tensor<Float> {
let positives = max(input, Tensor(0))
let leaked = min(input, Tensor(0)) * self.leak
return min(positives + leaked - self.sub, Tensor(self.maxv))
}
}
If I don’t extend Layer then the cell finishes with no errors!
Same things for this standalone function, kernel crashes reliably:
rxwei
(Richard Wei (Swift for TensorFlow team))
10
I noticed from your API usage (applied(to:in:)) that you are using v0.2, which is an older release of Swift for TensorFlow. The issue may have been fixed in v0.3. I tried it and can’t reproduce the crash.
Here’s a slightly modified version of your code that uses v0.3 APIs (call(_:) instead of applied(to:in:)).
import TensorFlow
struct GeneralReLU: Layer {
var leak: Float // slope for X when it's negative
var sub: Float // value used to substract from final result
var maxv: Float // maximum value
init(_ leak:Float = 0.0, _ sub:Float = 0.0, _ maxv:Float = Float.greatestFiniteMagnitude) {
self.leak = leak
self.sub = sub
self.maxv = maxv
}
@differentiable
func call(_ input: Tensor<Float>) -> Tensor<Float> {
let positives = max(input, Tensor(0))
let leaked = min(input, Tensor(0)) * self.leak
return min(positives + leaked - self.sub, Tensor(self.maxv))
}
}
You can follow the instructions here to get newer builds in Colab:
Thanks again @rxwei I guess you’re right i’m on an older version as you solution does not even compile, I see this error
error: <Cell 11>:4:8: error: type 'GeneralReLU' does not conform to protocol 'Layer'
struct GeneralReLU: Layer {
^
TensorFlow.Layer:2:20: note: protocol requires nested type 'Input'; do you want to add it?
associatedtype Input : Differentiable
^
TensorFlow.Layer:3:20: note: protocol requires nested type 'Output'; do you want to add it?
associatedtype Output : Differentiable
Is there a way (or a variable) that contains current swift for tensforlow version, it could be helpful to print it at the beginning of the kernel.
rxwei
(Richard Wei (Swift for TensorFlow team))
12
It would be nice if the commit hash and version string were exposed as variables in Swift itself so that you can query them from a running swift notebook. I don’t know if they are. Finding and/or exposing them might be a good starter task for someone interested in hacking the Swift compiler/stdlib!