Add hiredis dependency on supported platforms (#20437)

<!-- Please add a reviewer to the assignee section when you create a PR. If you don't have the access to it, we will shortly find a reviewer and assign them to your PR. -->

## Why are these changes needed?

This PR adds the hiredis dependency for non M1 machines. 

This removes the `redis < 4.0` pin.

Since hiredis doesn't have M1 mac wheels yet, so users there will have extra warning messages in their outputs if they use redis 4.0.
<!-- Please give a short summary of the change and the problem this solves. -->

## Related issue number

<!-- For example: "Closes #1234" -->

## Checks

- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for https://docs.ray.io/en/master/.
- [ ] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(


Co-authored-by: Alex Wu <alex@anyscale.com>

python/setup.py

@@ -4,6 +4,7 @@ import glob
 import io
 import logging
 import os
+import platform
 import re
 import shutil
 import subprocess
@@ -264,9 +265,20 @@ if setup_spec.type == SetupType.RAY:
         "numpy >= 1.19.3; python_version >= '3.9'",
         "protobuf >= 3.15.3",
         "pyyaml",
-        "redis >= 3.5.0, < 4.0.0",
     ]
 
+if platform.system() == "Darwin" and platform.machine() == "arm64":
+    # TODO (Alex): `hiredis` doesn't have prebuilt M1 mac wheels yet. We can
+    # remove this, either when they add support, we remove redis, or we vendor
+    # redis/hiredis ourselves.
+    setup_spec.install_requires.append("redis >= 3.5.0")
+elif platform.system() == "Windows":
+    # TODO (Alex): Ray is not compatible with redis >= 4.0.0. We ened to either
+    # investigate why, or remove the redis dependency.
+    setup_spec.install_requires.append("redis >= 3.5.0, < 4.0")
+else:
+    setup_spec.install_requires.append("redis[hiredis] >= 3.5.0")
+
 
 def is_native_windows_or_msys():
     """Check to see if we are running on native Windows,