This is not supported. I think there’s a sound argument that it should be. However, there’s a cost. It introduces a generalized computational problem for Guild that includes syntax (presumably a Python expression, which is not too hard to support). A bigger downside is that it starts to move computational logic from the script to the Guild file. I’m inclined to think that’s not a good idea. You can imagine this section becoming quite complex, turning into a Python script in itself!
One tack you might consider is to perform this check in your training operation and log a message that you look for as a fixed string, like
ACCURACY OK, etc. You can parameterize the expected accuracy threshold as a flag to the training script, e.g.
target-accuracy, e.g. with a default value of
0.9 that can be changed as needed when you train. This value is then memorialized in the run record. As a final stage of the training script, you check accuracy against this flag-defined threshold and print a message.
This approach lets you evolve the definition of “test” to whatever you want with as many value tests as you need. Guild then simply looks for a string from you that says the result is okay.
What do you think?