Commit graph

50 commits

Author SHA1 Message Date
Lixin Wei
aea3d53545
[Streaming] Supports multiple downstream collector (#9240) 2020-07-03 11:05:07 +08:00
chaokunyang
acd765cb22
[Streaming] fix source loop (#9085) 2020-06-23 11:57:06 +08:00
chaokunyang
5edddf6eac
[Streaming] operator chain (#8910) 2020-06-18 15:11:07 +08:00
chaokunyang
cb6f337372
[Java] Refine python function (#8943) 2020-06-16 16:22:49 +08:00
chaokunyang
dfa4768fc6
[Java] Refactor java api (#8858) 2020-06-12 10:49:01 +08:00
chaokunyang
700d81fa20
[Java] Remove java api sub package from test module (#8853) 2020-06-11 14:59:45 +08:00
Tianyi Chen
ec5ecb661f
[Streaming] Implement streaming job-worker. (#8780) 2020-06-10 14:13:55 +08:00
chaokunyang
d2ef29f0d2
[Streaming] Fix streaming word count tests (#8697) 2020-06-09 11:49:19 +08:00
chaokunyang
31a4d07bc4
[Java] Rename java ObjectRef/ActorHandle (#8799) 2020-06-09 11:40:43 +08:00
chaokunyang
d04953ab3c
[Streaming] Union api (#8612) 2020-06-08 14:28:11 +08:00
chaokunyang
bcdbe2d3d4
Streaming rich function (#8602) 2020-05-27 18:36:07 +08:00
chaokunyang
8da00af74e
[Streaming ]print streaming error when tests fail (#8594) 2020-05-27 11:04:08 +08:00
Max Fitton
00325eb2b2
Rename max_reconstructions to max_restarts and use -1 for infinite (#8274)
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
2020-05-14 10:30:29 -05:00
mehrdadn
4bdef78e2e
Various CI fixes and cleanup (#8289) 2020-05-05 10:47:49 -07:00
chaokunyang
91f630f709
[Streaming] Streaming Cross-Lang API (#7464) 2020-04-29 13:42:08 +08:00
WuTao
32c2055c99
Streaming state (#7348) 2020-04-28 10:36:32 +08:00
chaokunyang
5cf49d5edd
Fix streaming ci (#8159) 2020-04-26 20:56:58 +08:00
Tianyi Chen
0204dff1e9
[streaming]Add master and scheduler. (#8044) 2020-04-22 14:43:56 +08:00
mehrdadn
c8b9a357f2
Try to fix dependency issue (#8065)
Co-authored-by: Mehrdad <noreply@github.com>
2020-04-19 16:09:29 -07:00
mehrdadn
8cf37726d2
Fix missing Java dependency (#8067)
Co-authored-by: Mehrdad <noreply@github.com>
2020-04-17 10:43:02 -05:00
mehrdadn
f15618033d
Remove --no-transfer-progress as it appears to be unsupported (#8066)
Co-authored-by: Mehrdad <noreply@github.com>
2020-04-17 16:30:37 +02:00
Richard Liaw
118d960e1c
[hotfix] Java Lint Broken (#8048) 2020-04-16 13:58:33 -07:00
wanxing
9345d03ffb
[Streaming] Streaming data transfer supports cross language. (#7961)
* add init parameters for java

* fix bug

* cython

* fix compile

* fix test_direct_tranfer

* comment

* ChannelCreationParameter

* fix comment

* builder

* lint and fix tests

* fix single process test

* fix checkstyle and lint

* checkstyle

* lint python

Co-authored-by: wanxing <wanxing@B-458DMD6M-1753.local>
2020-04-16 15:16:48 +08:00
JianZhangYang
7b0518b993
[streaming] Async changes for resourcemanager part (#7955) 2020-04-16 14:15:45 +08:00
mehrdadn
ba00c29b67
Factor out Travis 'install' sections for use with GitHub Actions (#7988) 2020-04-15 08:10:22 -07:00
Qing Wang
98bfcd53bc
[Java] Rename group id and package name. (#7864)
* Initial

* Change streaming's

* Fix

* Fix

* Fix org_ray

* Fix cpp file name

* Fix streaming

* Fix

* Fix

* Fix testlistening

* Fix missing sth in python

* Fix

* Fix

* Fix SPI

* Fix

* Fix complation

* Fix

* Fix CI

* Fix checkstyle

Fix checkstyle

* Fix streaming tests

* Fix streaming CI

* Fix streaming checkstyle.

* Fix build

* Fix bazel dep

* Fix

* Fix ray checkstyle

* Fix streaming checkstyle

* Fix bazel checkstyle
2020-04-12 17:59:34 +08:00
mehrdadn
4aa68b82fa
[CI] Various Improvements to Travis Scripts (#7956)
* Delete LINT section of install-ray.sh since it appears unused

* Delete install.sh since it appears unused

* Delete run_test.sh since it appears unused

* Put environment variables on separate lines in .travis.yml

* Move --jobs 50 out of install-ray.sh

* Delete upgrade-syn.sh since it appears unused

* Move CI bazel flags to .bazelrc via --config

* Make installations quieter

* Get rid of verbose Maven messages

* Install Bazel system-wide for CI so that there's no need to update PATH

* Recognize Windows as valid platform

Co-authored-by: Mehrdad <noreply@github.com>
2020-04-10 13:26:28 -07:00
Tianyi Chen
c5bf9cc472
[streaming] Sync changes for graph part. (#7827) 2020-04-09 12:30:44 +08:00
fyrestone
fc6259a656
Cross language serialization for primitive types (#7711)
* Cross language serialization for Java and Python

* Use strict types when Python serializing

* Handle recursive objects in Python; Pin msgpack >= 0.6.0, < 1.0.0

* Disable gc for optimizing msgpack loads

* Fix merge bug

* Java call Python use returnType; Fix ClassLoaderTest

* Fix RayMethodsTest

* Fix checkstyle

* Fix lint

* prepare_args raises exception if try to transfer a non-deserializable object to another language

* Fix CrossLanguageInvocationTest.java, Python msgpack treat float as double

* Minor fixes

* Fix compile error on linux

* Fix lint in java/BUILD.bazel

* Fix test_failure

* Fix lint

* Class<?> to Class<T>; Refine metadata bytes.

* Rename FST to Fst; sort java dependencies

* Change Class<?>[] to Optional<Class<?>>; sort requirements in setup.py

* Improve CrossLanguageInvocationTest

* Refactor MessagePackSerializer.java

* Refactor MessagePackSerializer.java; Refine CrossLanguageInvocationTest.java

* Remove unnecessary dependencies for Java; Add getReturnType() for RayFunction in Java

* Fix bug

* Remove custom cross language type support

* Replace Serializer.Meta with MutableBoolean

* Remove @SuppressWarnings support from checkstyle.xml; Add null test in CrossLanguageInvocationTest.java

* Refine MessagePackSerializer.pack

* Ray.get support RayObject as input

* Improve comments and error info

* Remove classLoader argument from serializer

* Separate msgpack from pickle5 in Python

* Pair<byte[], MutableBoolean> to Pair<byte[], Boolean>

* Remove public static <T> T get(RayObject<T> object), use RayObject.get() instead

* Refine test

* small fixes

Co-authored-by: 刘宝 <po.lb@antfin.com>
Co-authored-by: Hao Chen <chenh1024@gmail.com>
2020-04-08 21:10:57 +08:00
Kai Yang
48b48cc8c2
Support multiple core workers in one process (#7623) 2020-04-07 11:01:47 +08:00
Tianyi Chen
f889f938e5
[streaming] Use enum to define resource type. (#7813) 2020-03-31 00:03:49 +08:00
fyrestone
a1ae935839
Java call Python use structured function descriptors (#7634) 2020-03-20 17:29:45 +08:00
Kai Yang
6b888b0247
[Java] Make both RayActor and RayPyActor inheriting from BaseActor (#7462) 2020-03-17 21:45:56 +08:00
fyrestone
7697ea2be2
Java call Python actor method use actor.call (#7614) 2020-03-17 14:52:43 +08:00
Qing Wang
f4656d8cc3
[Java] Enable direct call by default. (#7408)
* WIP

* Address comments.

* Linting

* Fix

* Fix

* Fix test

* Fix

* Fix single process ci

* Fix ut

* Update java/test/src/main/java/org/ray/api/test/PlasmaFreeTest.java

* Address comments

* Fix linting

* Minor update comments.

* Fix streaming CI
2020-03-13 12:25:30 +08:00
Tianyi Chen
6993a471f1
[Streaming] Move resource-manager and scheduler to master package. (#7582) 2020-03-13 12:24:37 +08:00
Hao Chen
fe7820fec9
[Java] New Java actor API (#7414) 2020-03-04 22:39:23 +08:00
chaokunyang
8b6784de06
[Streaming] Streaming Python API (#6755) 2020-02-25 10:33:33 +08:00
Lingxuan Zuo
f995099e00
[Streaming] Support streaming flow control (#7152)
* streaming writer use event driven model.

* add RefreshChannelInfo

* fix name

* minor changes according reviewer comments

* Fix according to reviewer's comments

* fix bazel lint

* code polished

* Add more comments

* rename Stop & Start of EventQueue to Freeze and Unfreeze.

* add override

* fix

* fix return value

* support flow control

* add flow control ut in mock transfer

* minor changes according to comments

* add java and python worker adaption

Co-authored-by: wanxing <wanxing.wwx@alibaba-inc.com>
2020-02-24 23:48:04 +08:00
JianZhangYang
7e115490d9
Streaming scheduler resourcemanager (#7070) 2020-02-24 19:32:50 +08:00
chaokunyang
d7f8d18a86
fix symbols conflict and add symbols check (#7227) 2020-02-20 19:31:16 +08:00
mehrdadn
e09f63ad65
Fix build errors and add more targets to Windows builds (#6811)
* Fix common.fbs rename (due to apache/arrow/commit/bef9a1c251397311a6415d3dc362ef419d154caa)

* Add missing COPTS

* Use socketpair(AF_INET) if boost::asio::local is unavailable (e.g. on Windows)

* Fix compile bug in service_based_gcs_client_test.cc (fix build breakage in #6686)

* Work around googletest/gmock inability to specify override to avoid -Werror,-Winconsistent-missing-override

* Fix missing override on IsPlasmaBuffer()

* Fix missing libraries for streaming

* Factor out install-toolchains.sh

* Put some Bazel flags into .bazelrc

* Fix jni_md.h missing inclusion

* Add ~/bin to PATH for Bazel

* Change echo $$(date) > $@ to date > $@

* Fix lots of unquoted paths

* Add system() call checks for Windows

Co-authored-by: GitHub Web Flow <noreply@github.com>
2020-02-11 16:49:33 -08:00
Tianyi Chen
13882d052d
[Streaming] Streaming scheduler - part1-2: execution graph (#6666) 2020-02-04 11:51:21 +08:00
chaokunyang
289e5e8aff enable maven checkstyle (#6829) 2020-01-20 23:41:54 -08:00
chaokunyang
fa3c513276 [Streaming] Streaming filter transform (#6816)
* add filter transform

* lint

* add new line
2020-01-17 22:05:47 +08:00
Tianyi Chen
9a4da1951e [Streaming] Streaming scheduler - part1-1: job graph (#6712) 2020-01-15 13:12:03 +08:00
chaokunyang
70c7d47c09 [Streaming] java cross lang streaming graph (#6689) 2020-01-08 17:32:35 +08:00
Tianyi Chen
9dacebec1a [Streaming] Add configuration with owner config. (#6687) 2020-01-08 11:19:01 +08:00
Jing Ge
d39e76f2ce rename interface and class for task assigner based on suitable pattern. (#6664) 2020-01-03 11:13:36 +08:00
Chaokun Yang
7bbfa85c66 [Streaming] Streaming data transfer java (#6474) 2019-12-22 10:56:05 +08:00