Fix a uncaught exception upon deallocation for actors (#27637)

As specified here, https://joekuan.wordpress.com/2015/06/30/python-3-__del__-method-and-imported-modules/, the del method doesn't guarantee that modules or function definitions are still referenced, and not GC'ed. That means if you access any "modules", "functions", or "global variables", they may have been garbage collected.

This means we should not access any modules, functions, or global variables inside del method. While it's something we should handle in the sooner future more holistically, this PR fixes the issue in the short term.

The problem was that all of ray actors are decorated by trace_helper.py to make it compatible to open telemetry (maybe we should make it optional). At this time __del__ method is also decorated. When __del__ is invoked, some of functions used within this tracing decorator can be accessed and may have been deallocated (in this case, the _is_tracing_enabled was deallocated). This fixes the issue by not decorating __del__ method from tracing.

python/ray/tests/test_ray_shutdown.py

@@ -7,7 +7,10 @@ import ray
 
 import psutil  # We must import psutil after ray because we bundle it with ray.
 
-from ray._private.test_utils import wait_for_condition, run_string_as_driver_nonblocking
+from ray._private.test_utils import (
+    wait_for_condition,
+    run_string_as_driver_nonblocking,
+)
 
 
 def get_all_ray_worker_processes():

python/ray/util/tracing/tracing_helper.py

@@ -522,6 +522,17 @@ def _inject_tracing_into_class(_cls):
         if is_static_method(_cls, name) or is_class_method(method):
             continue
 
+        # Don't decorate the __del__ magic method.
+        # It's because the __del__ can be called after Python
+        # modules are garbage colleted, which means the modules
+        # used for the decorator (e.g., `span_wrapper`) may not be
+        # available. For example, it is not guranteed that
+        # `_is_tracing_enabled` is available when `__del__` is called.
+        # Tracing `__del__` is also not very useful.
+        # https://joekuan.wordpress.com/2015/06/30/python-3-__del__-method-and-imported-modules/ # noqa
+        if name == "__del__":
+            continue
+
         # Add _ray_trace_ctx to method signature
         setattr(
             method,