Commit graph

1599 commits

Author SHA1 Message Date
Devin Petersohn
72a3a6cb02 [DataFrame] Implement where (#1989) 2018-05-09 14:05:52 -07:00
Kunal Gosar
d2c193ed2c [DataFrame] Add direct pandas imports for MVP (#1960)
* Add direct pandas imports for MVP

* rebase artifact
2018-05-08 19:19:32 -07:00
Robert Nishihara
77c8aa7627 Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (#2007)
* Make ActorHandles pickleable, also make proper ActorHandle and ActorClass classes.

* Fix bug.

* Fix actor test bug.

* Update __ray_terminate__ usage.

* Fix most linting, add documentation, and small cleanups.

* Handle forking and pickling differently for actor handles. Fix linting.

* Fixes for named actors via pickling.

* Generate actor handle IDs deterministically in the pickling case.
2018-05-08 19:19:07 -07:00
Kristian Hartikainen
2048b546ff Expand local_dir in Trial init (#2013)
* Fix the case where Trial logs into wrong paths when `local_dir`
argument starts with tilde (~), by expanding the `local_dir` argument
* Add test case for checking that the tilde gets expanded
2018-05-07 21:44:28 -07:00
Devin Petersohn
b1e32ca6c2 Fixing ascii error for Python2 (#2009) 2018-05-07 11:56:40 -07:00
Peter Veerman
1f82a46473 [DataFrame] Implements df.update (#1997)
* Working on fixing update

* Fixing update implementation

* Adding test

* Addressing comments
2018-05-07 08:55:40 -07:00
Peter Veerman
12da021717 [DataFrame] Implements df.as_matrix (#2001)
* Implement df.as_matrix

* Addressing comments

* Addressing comments
2018-05-06 23:36:39 -07:00
Rohan Singh
1848745223 [DataFrame] Implement quantile (#1992)
* added quantile method

* updated init for datetime signatures

* updated documentation for _map_partitions return type

* removed extraneous print call

* updated for simplicity

* fixed dtyping issues and error raising

* updated datetime dtype checking

* Fixing quantile implementation

* Fix minor bug

* Fixing diff
2018-05-06 18:25:13 -07:00
Devin Petersohn
ad1afeb268 [DataFrame] Impement sort_values and sort_index (#1977) 2018-05-06 09:53:29 -07:00
Rohan Singh
9f28529e2c [DataFrame] Implement rank (#1991)
* rank method completed

* added sanity checks

* flake8

* updated sanity checks

* flake8

* updated sanity checks and style

* updated dtype logic

* Fixing test
2018-05-06 09:32:33 -07:00
Hari Subbaraj
857458c37c [DataFrame] Implemented prod, product, added test suite (#1994)
* implemented prod/product, modified declaration for sum, added pandas test suite

* fixed tests

* removed test_analytics file

* implemented nunique, skew

* fixed requested changes

* added nunique, skew

* fixed tests in request

* added newline back

* fixed newlines hopefully

* fixed flake8 issues

* more flake8 issues

* fixed test for prod
2018-05-05 21:25:42 -07:00
Jae Min Kim
36fd64800b [DataFrame] Implemented __setitem__, select_dtypes, and astype (#1941)
* added reindex, __setitem__, select_dtypes, and astype functionality

* readded tests for astype and select_dtypes

* fixed index issue with reindex

* lint spacing

* removing current reindex implementation for future pr

* wrong testing func

* errors now raised in the workers, but suppressing them can be an issue

* updated code for select_dtypes

* Update test_dataframe.py
2018-05-05 20:27:29 -07:00
Rohan Singh
8509a51291 [DataFrame] Implement diff (#1996)
* added diff method

* sanity checks

* flake8

* updated sanity checks'

* rebase and style updates

* updated diff tests and cleaned up code

* updated tests for periods

* flake8
2018-05-05 00:44:52 -07:00
Hari Subbaraj
a58629f53f [DataFrame] Implemented nunique, skew (#1995)
* implemented prod/product, modified declaration for sum, added pandas test suite

* fixed tests

* removed test_analytics file

* implemented nunique, skew

* fixed changes for nunique, skew

* fixed nunique test

* added axis=1 test to skew

* flake8 issues

* more flake8 issues

* resolve some flake issues
2018-05-04 12:22:10 -07:00
Kunal Gosar
4030356b51 [DataFrame] Implements filter and dropna (#1959)
* implement filter

* begin implementation of dropna

* implement dropna

* docs and tests

* resolving comments

* resolving merge

* add error checking to dropna

* fix update inplace call

* Implement multiple axis for dropna (#13)

* Implement multiple axis for dropna

* Add multiple axis dropna test

* Fix using dummy_frame in dropna

* Clean up dropna multiple axis tests

* remove unnecessary axis modification

* Clean up dropna tests

* resolve comments

* fix lint
2018-05-04 12:21:16 -07:00
Peter Veerman
22d4950fae [DataFrame] Implements df.pipe (#1999)
* Add empty df test

* Fix flake8 issues

* rebase with master

* reset master tests

* Implement df.pipe

* fix tests

* Use test_pipe as a pytest.fixture

* Add newline at EOF
2018-05-04 10:16:05 -07:00
Omkar Salpekar
a1d7bb31a4 [DataFrame] Apply() for Lists and Dicts (#1973)
* working for non-string functions and not lists of functions

* works with functions as strings now as well

* fixed linting errors

* throwing a warning if the input is a dictionary

* added dict of lists functionality

* fix minor indexing errors and lint

* removed some commented out code

* some comments and thoughts for apply

* cleaned up code a little bit and added todos

* improved performance

* error checking and code cleanup and comments

* small change

* improved list performance a lot

* agg calls apply for lists

* addressing comments on the PR

* col_metadata change

* updated tests to expect TypeError where appropriate
2018-05-04 10:05:00 -07:00
Alok Singh
cdf94c18a4 Clean up syntax for supported Python versions. (#1963)
* Use set/dict literal syntax

Ran code through [pyupgrade](https://github.com/asottile/pyupgrade). This is
supported in every Python version 2.7+.

* Drop unnecessary string format specification

No need to specify 0,1.. if paramters are passed in order.

* Revert "Drop unnecessary string format specification"

This reverts commit efa5ec85d30ff69f34e5ed93e31343fea7647bcb.

* Undo changes to cloudpickle

Drop use of set literal until cloudpickle uses it.

* Reformat code with YAPF

We need to set up a git pre-push hook to automatically run this stuff.
2018-05-03 07:45:11 -07:00
Kunal Gosar
d85ee0bc04 [DataFrame] Implements mode, to_datetime, and get_dummies (#1956)
* implement mode and fix getitem

* mode broken on misaligned partitions

* fully implement mode

* implement to_datetime

* implement get_dummies

* implement tests

* fix __getitem__

* fix python2 compatibility

* fix getitem bug

* resolving comments

* Adding documentation

* resolving comment

* resolve name change

* speeding up getitem

* complete rebase
2018-05-02 23:21:00 -07:00
Peter Schafhalter
d67b786291 [DataFrame] Fix dtypes (#1930)
* Add map, reduce, merge_dtypes

bug fixes

Unify dtypes on DataFrame creation

Formatting and comments

Cache dtypes

Fix bug in _merge_dtypes

Fix bug

Changed caching logic

Fix dtypes issue in read_csv

Invalidate dtypes cache when inserting column

Simplify unifying dtypes and improve caching

Fix typo

Better caching of dtypes

Fix merge conflicts

* Correct dtypes on initialization
2018-05-02 23:04:19 -07:00
Alok Singh
677b1c85ae keep_dims -> keepdims (#1980)
keep_dims is deprecated.
2018-05-02 22:06:05 -07:00
alonamid
32fa862408 add pthread linking (#1986) 2018-05-02 21:50:29 -07:00
Devin Petersohn
4badc04bb2 [DataFrame] Add layer of abstraction to allow OID instantiation (#1984) 2018-05-02 22:29:52 -04:00
Patrick Yang
5589426484 [DataFrame] Fix blocking issue on _IndexMetadata passing (#1965)
* metadata passing fixes

* fix flake8

* fix test failures

* overhaul indexmetadata

* variable name change

* optimization for building coord df

* addressing comments

* subtle bug fixes
2018-05-01 23:27:49 -07:00
Devin Petersohn
7c1d569a49 [DataFrame] Implement df.merge (#1964)
* Begin merge implementation

* Some cleanup

* Continue cleanup

* Allowing merge on index

* Copy dataframes to clear plasma read-only error

* Make some notes, WIP

* Cleaned up code a bit, still need more error checking

* Adding error checking and addressing comments

* Addressing comment

* Adding test

* Addressing rebase artifact

* Fixing indexing bug

* Some minor cleanup
2018-05-01 21:40:53 -04:00
Alok Singh
06a0898af7 [rllib] Fix PyTorch initialization (#1961)
* Fix typo

* Fix A3C PyTorch agent initialization

`registry` needs to be passed as an argument or else the `super` init will
fail.
2018-05-01 18:39:01 -07:00
Eric Liang
b55f4a7f04 [rllib] Fix broken link in docs (#1967)
* Update README.rst

* Update rllib.rst
2018-04-30 16:02:48 -07:00
Omkar Salpekar
1231aa0582 [DataFrame] Sample implement (#1954)
* implemented sample - need to test

* sample fully working

* added sanity check tests

* added some comments to clarify the _deploy_func call

* some more clarifying comments

* added explanatory comments

* minor change in weights_sum for sample
2018-04-30 10:42:28 -07:00
Devin Petersohn
0c477fbbca [DataFrame] Implement Inter-DataFrame operations (#1937) 2018-04-30 06:42:07 -07:00
eric-jj
34bc6ce6ea remove UniqueIDHasher (#1957)
* remove UniqueIDHasher

* Format the change

* remove unused line

* Fix format

* fix lint error

* fix linting whitespace
2018-04-30 06:31:23 -07:00
Eric Liang
47bc4c3009
[rllib] Add DDPG documentation, rename DDPG2 <=> DDPG (#1946)
* updates

* updates

* updates

* updates

* updates

* updates

* Update rllib.rst

* Update policy-optimizers.rst
2018-04-30 00:18:15 -07:00
Eric Liang
9ad94e33d6
updates (#1958) 2018-04-27 19:44:21 -07:00
Robert Nishihara
1a6b5b04e6 Pin Cython in autoscaler development example. (#1951) 2018-04-26 00:17:29 -07:00
Philipp Moritz
af88fdefcf Incorporate C++ Buffer management and Seal global threadpool fix from arrow (#1950) 2018-04-25 22:53:44 -07:00
Philipp Moritz
dad465a2bf [XRay] Add consistency check for protocol between node_manager and local_scheduler_client (#1944) 2018-04-23 23:51:25 -07:00
Robert Nishihara
3c76461b22 Remove smart_open install. (#1943) 2018-04-23 23:18:09 -07:00
Devin Petersohn
1d1df7bbec [DataFrame] Fully implement append, concat and join (#1932) 2018-04-23 17:09:57 -07:00
Kunal Gosar
29c36f2bce [DataFrame] Fix for __getitem__ string indexing (#1939)
* edge case fixes for __getitem__

* Enable None indexing
2018-04-23 13:13:14 -07:00
Kunal Gosar
7c9f39241e [DataFrame] Implementing write methods (#1918)
* Add in write methods and functionality

* infer highest available pickle version

* Fix import rebase artifact

* formatting changes to test

* fix lint
2018-04-22 21:25:33 -07:00
Roy Fox
baf97e450b [rllib] arr[end] was excluded when end is not None (#1931)
Looks good, thanks!
2018-04-22 15:12:55 -07:00
Devin Petersohn
8f59546ef2 [DataFrame] Implementing API correct groupby with aggregation methods (#1914) 2018-04-21 17:28:16 -07:00
Melih Elibol
8264e64b18 Handle interrupts correctly for ASIO synchronous reads and writes. (#1929)
* handle interrupts correctly.

* linting

* handle interrupts on read_some/write_some.
2018-04-20 22:55:40 -07:00
adgirish
3c48783a16 [DataFrame] Adding read methods and tests (#1712)
* Adding read methods and tests

* Referencing internal partition method so constructors are more canonical with Pandas

* Fixing to reference from_pandas in utils

* Cleaning up unused imports

* rerunning tests

* fixing flake8

* resolving errors

* Added sql and sas test

* updating

* Temporarily phasing out read_csv code for wrapper while diagnosing, added io tests to travis

* Adding travis

* restoring distributed read csv

* resolving rebases

* lint

* Sampling out HD test

* adding dep

* fix pathing

* Flagging out tests

* resolving read_method issues

* fix build issue

* move additional dependencies to extras

* fixing lint

* removing IO dependencies

* updated requirements doc
2018-04-20 18:33:08 -07:00
Robert Nishihara
cffda73da1 Allow task_table_update to fail when tasks are finished. (#1927)
* Allow task_table_update to fail when tasks are finished.

* Add comment.
2018-04-20 11:34:29 -07:00
Jones Wong
c9a7744e52 [rllib] Contribute DDPG to RLlib (#1877)
*  ongoing ddpg

*  ongoing ddpg converged

*  gpu machine changes

*  tuned

*  tuned ddpg specification

*  ddpg

*  supplement missed optimizer argument clip_rewards in default DQN configuration

*  ddpg supports vision env (atari) now

*  revised according to code review comments

*  added regression test case

*  removed irrelevant files

*  validate ddpg on mountain_car_continuous

*  restore unnecessary slight changes

*  revised according to eric's comments

*  added the requested tests

*  revised accordingly

*  revised accordingly and re-validated

*  formatted by yapf

*  fix lint errors

*  formatted by yapf

*  fix lint errors

*  formatted by yapf

*  fix lint error
2018-04-19 22:36:29 -07:00
Stephanie Wang
aa07f1ce4e [xray] Workers blocked in a ray.get release their resources (#1920)
* [xray] Throttle task dispatch by required resources
* Pass in number of initial workers into raylet command
* Workers blocked in a ray.get release resources
2018-04-18 20:59:58 -07:00
Alexey Tumanov
1c965fcfeb Raylet task dispatch and throttling worker startup (#1912)
* separate task placement and task dispatch; throttle task dispatch with locally available resournces

* keep track of worker's being started/in flight and suppress starting extraneous workers

* cleanup comments

* remove early termination in task dispatch to support zero-resource actor tasks

* info -> debug

* add documentation

* linting

* mock the worker pool for testing

* some linting

* kill all workers in flight; clear the worker pool in dtor

* remove fixed todo

* lint
2018-04-18 10:58:11 -07:00
Omkar Salpekar
0728d4719b [DataFrame] Eval fix (#1903)
* eval now works without assignment - helper function a bit hacky

* removed df.copy() from eval_helper

* one test still failing for qury

* all eval tests passing now

* added check to eval arge verification

* added tests to travis

* added optimization and some comments

* added pd.eval and passes all tests

* added ray dataframe back to test file

* optimizations and code cleanup for eval

* changed position of pandas import in __init__

* fixed linting errors

* fixing eval in __init__.py

* fixed travis file - removed extra tests

* removed test directory from linting exclude for travis
2018-04-18 08:48:32 -07:00
Richard Liaw
f833e4da37
[tune] Polishing docs (#1846) 2018-04-17 09:57:35 -07:00
Eric Liang
7ab890f4a1 [tune] [rllib] Automatically determine RLlib resources and add queueing mechanism for autoscaling (#1848) 2018-04-16 16:58:15 -07:00