OperatorNotAllowedInGraphError: Error while using Guild run

Troubleshooting
1

Hi,

I am trying to use a code with guild. I checked the code using python and the code runs successfully. However, when I try to use the code using guild run it gives me the following error. On looking at the error closely, I found that the code gives me an error in the model.fit function of keras. I am not sure what is happening when we use guild run in this function that is triggering the error?

Any help or suggestion in this matter would be greatly appreciated.

Thanks in advance,
Vishal

Full error below.

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 1098, in fit
    tmp_logs = train_function(iterator)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 823, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 697, in _initialize
    *args, **kwds))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2855, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 3213, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 3075, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py", line 986, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 600, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py", line 973, in wrapper
    raise e.ag_error_metadata.to_exception(e)
tensorflow.python.framework.errors_impl.OperatorNotAllowedInGraphError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:799 step_function  **
        write_scalar_summaries(outputs, step=model._train_counter)  # pylint: disable=protected-access
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:2757 write_scalar_summaries
        summary_ops_v2.scalar('batch_' + name, value, step=step)
    /usr/local/lib/python3.6/dist-packages/guild/python_util.py:321 wrapper
        cb(self._func, *args, **kw)
    /usr/local/lib/python3.6/dist-packages/guild/plugins/summary_util.py:198 _handle_scalar_ops_v2
        vals = self._summary_values(step)
    /usr/local/lib/python3.6/dist-packages/guild/plugins/summary_util.py:179 _summary_values
        return self._summary_cache.for_step(global_step)
    /usr/local/lib/python3.6/dist-packages/guild/plugins/summary_util.py:246 for_step
        return self._val if step == self._step else None
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:877 __bool__
        self._disallow_bool_casting()
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:487 _disallow_bool_casting
        "using a `tf.Tensor` as a Python `bool`")
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:474 _disallow_when_autograph_enabled
        " indicate you are trying to use an unsupported feature.".format(task))

    OperatorNotAllowedInGraphError: using a `tf.Tensor` as a Python `bool` is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.
2

This is a bug in Guild that I believe has been fixed. What version are you using? You can use guild --version to check.

You can try temporarily upgrading to the pre-release version to see if that resolves the issue.

pip install guildai --pre --upgrade
1 Like
3

Thanks for the reply. I am using version guild 0.7.2. Is the bug fixed after this version?

We got a temporary workaround and it seems that guild did not like the “model.fit” function in the main script. Shifting the model.fit function to the utilities folder and calling the function from utils in the main script worked - magically!

4

The fix is in either 0.7.3 or 0.7.4 (pre-release). If you’re happy with the workaround, that’s great—or if you want to see if the issue is fixed in 0.7.3 or 0.7.4, you can upgrade to see.

1 Like