Commit graph

530 commits

Author SHA1 Message Date
Kunal Gosar
4030356b51 [DataFrame] Implements filter and dropna (#1959)
* implement filter

* begin implementation of dropna

* implement dropna

* docs and tests

* resolving comments

* resolving merge

* add error checking to dropna

* fix update inplace call

* Implement multiple axis for dropna (#13)

* Implement multiple axis for dropna

* Add multiple axis dropna test

* Fix using dummy_frame in dropna

* Clean up dropna multiple axis tests

* remove unnecessary axis modification

* Clean up dropna tests

* resolve comments

* fix lint
2018-05-04 12:21:16 -07:00
Peter Veerman
22d4950fae [DataFrame] Implements df.pipe (#1999)
* Add empty df test

* Fix flake8 issues

* rebase with master

* reset master tests

* Implement df.pipe

* fix tests

* Use test_pipe as a pytest.fixture

* Add newline at EOF
2018-05-04 10:16:05 -07:00
Omkar Salpekar
a1d7bb31a4 [DataFrame] Apply() for Lists and Dicts (#1973)
* working for non-string functions and not lists of functions

* works with functions as strings now as well

* fixed linting errors

* throwing a warning if the input is a dictionary

* added dict of lists functionality

* fix minor indexing errors and lint

* removed some commented out code

* some comments and thoughts for apply

* cleaned up code a little bit and added todos

* improved performance

* error checking and code cleanup and comments

* small change

* improved list performance a lot

* agg calls apply for lists

* addressing comments on the PR

* col_metadata change

* updated tests to expect TypeError where appropriate
2018-05-04 10:05:00 -07:00
Alok Singh
cdf94c18a4 Clean up syntax for supported Python versions. (#1963)
* Use set/dict literal syntax

Ran code through [pyupgrade](https://github.com/asottile/pyupgrade). This is
supported in every Python version 2.7+.

* Drop unnecessary string format specification

No need to specify 0,1.. if paramters are passed in order.

* Revert "Drop unnecessary string format specification"

This reverts commit efa5ec85d30ff69f34e5ed93e31343fea7647bcb.

* Undo changes to cloudpickle

Drop use of set literal until cloudpickle uses it.

* Reformat code with YAPF

We need to set up a git pre-push hook to automatically run this stuff.
2018-05-03 07:45:11 -07:00
Kunal Gosar
d85ee0bc04 [DataFrame] Implements mode, to_datetime, and get_dummies (#1956)
* implement mode and fix getitem

* mode broken on misaligned partitions

* fully implement mode

* implement to_datetime

* implement get_dummies

* implement tests

* fix __getitem__

* fix python2 compatibility

* fix getitem bug

* resolving comments

* Adding documentation

* resolving comment

* resolve name change

* speeding up getitem

* complete rebase
2018-05-02 23:21:00 -07:00
Peter Schafhalter
d67b786291 [DataFrame] Fix dtypes (#1930)
* Add map, reduce, merge_dtypes

bug fixes

Unify dtypes on DataFrame creation

Formatting and comments

Cache dtypes

Fix bug in _merge_dtypes

Fix bug

Changed caching logic

Fix dtypes issue in read_csv

Invalidate dtypes cache when inserting column

Simplify unifying dtypes and improve caching

Fix typo

Better caching of dtypes

Fix merge conflicts

* Correct dtypes on initialization
2018-05-02 23:04:19 -07:00
Alok Singh
677b1c85ae keep_dims -> keepdims (#1980)
keep_dims is deprecated.
2018-05-02 22:06:05 -07:00
Devin Petersohn
4badc04bb2 [DataFrame] Add layer of abstraction to allow OID instantiation (#1984) 2018-05-02 22:29:52 -04:00
Patrick Yang
5589426484 [DataFrame] Fix blocking issue on _IndexMetadata passing (#1965)
* metadata passing fixes

* fix flake8

* fix test failures

* overhaul indexmetadata

* variable name change

* optimization for building coord df

* addressing comments

* subtle bug fixes
2018-05-01 23:27:49 -07:00
Devin Petersohn
7c1d569a49 [DataFrame] Implement df.merge (#1964)
* Begin merge implementation

* Some cleanup

* Continue cleanup

* Allowing merge on index

* Copy dataframes to clear plasma read-only error

* Make some notes, WIP

* Cleaned up code a bit, still need more error checking

* Adding error checking and addressing comments

* Addressing comment

* Adding test

* Addressing rebase artifact

* Fixing indexing bug

* Some minor cleanup
2018-05-01 21:40:53 -04:00
Alok Singh
06a0898af7 [rllib] Fix PyTorch initialization (#1961)
* Fix typo

* Fix A3C PyTorch agent initialization

`registry` needs to be passed as an argument or else the `super` init will
fail.
2018-05-01 18:39:01 -07:00
Eric Liang
b55f4a7f04 [rllib] Fix broken link in docs (#1967)
* Update README.rst

* Update rllib.rst
2018-04-30 16:02:48 -07:00
Omkar Salpekar
1231aa0582 [DataFrame] Sample implement (#1954)
* implemented sample - need to test

* sample fully working

* added sanity check tests

* added some comments to clarify the _deploy_func call

* some more clarifying comments

* added explanatory comments

* minor change in weights_sum for sample
2018-04-30 10:42:28 -07:00
Devin Petersohn
0c477fbbca [DataFrame] Implement Inter-DataFrame operations (#1937) 2018-04-30 06:42:07 -07:00
Eric Liang
47bc4c3009
[rllib] Add DDPG documentation, rename DDPG2 <=> DDPG (#1946)
* updates

* updates

* updates

* updates

* updates

* updates

* Update rllib.rst

* Update policy-optimizers.rst
2018-04-30 00:18:15 -07:00
Eric Liang
9ad94e33d6
updates (#1958) 2018-04-27 19:44:21 -07:00
Robert Nishihara
1a6b5b04e6 Pin Cython in autoscaler development example. (#1951) 2018-04-26 00:17:29 -07:00
Philipp Moritz
af88fdefcf Incorporate C++ Buffer management and Seal global threadpool fix from arrow (#1950) 2018-04-25 22:53:44 -07:00
Devin Petersohn
1d1df7bbec [DataFrame] Fully implement append, concat and join (#1932) 2018-04-23 17:09:57 -07:00
Kunal Gosar
29c36f2bce [DataFrame] Fix for __getitem__ string indexing (#1939)
* edge case fixes for __getitem__

* Enable None indexing
2018-04-23 13:13:14 -07:00
Kunal Gosar
7c9f39241e [DataFrame] Implementing write methods (#1918)
* Add in write methods and functionality

* infer highest available pickle version

* Fix import rebase artifact

* formatting changes to test

* fix lint
2018-04-22 21:25:33 -07:00
Roy Fox
baf97e450b [rllib] arr[end] was excluded when end is not None (#1931)
Looks good, thanks!
2018-04-22 15:12:55 -07:00
Devin Petersohn
8f59546ef2 [DataFrame] Implementing API correct groupby with aggregation methods (#1914) 2018-04-21 17:28:16 -07:00
adgirish
3c48783a16 [DataFrame] Adding read methods and tests (#1712)
* Adding read methods and tests

* Referencing internal partition method so constructors are more canonical with Pandas

* Fixing to reference from_pandas in utils

* Cleaning up unused imports

* rerunning tests

* fixing flake8

* resolving errors

* Added sql and sas test

* updating

* Temporarily phasing out read_csv code for wrapper while diagnosing, added io tests to travis

* Adding travis

* restoring distributed read csv

* resolving rebases

* lint

* Sampling out HD test

* adding dep

* fix pathing

* Flagging out tests

* resolving read_method issues

* fix build issue

* move additional dependencies to extras

* fixing lint

* removing IO dependencies

* updated requirements doc
2018-04-20 18:33:08 -07:00
Jones Wong
c9a7744e52 [rllib] Contribute DDPG to RLlib (#1877)
*  ongoing ddpg

*  ongoing ddpg converged

*  gpu machine changes

*  tuned

*  tuned ddpg specification

*  ddpg

*  supplement missed optimizer argument clip_rewards in default DQN configuration

*  ddpg supports vision env (atari) now

*  revised according to code review comments

*  added regression test case

*  removed irrelevant files

*  validate ddpg on mountain_car_continuous

*  restore unnecessary slight changes

*  revised according to eric's comments

*  added the requested tests

*  revised accordingly

*  revised accordingly and re-validated

*  formatted by yapf

*  fix lint errors

*  formatted by yapf

*  fix lint errors

*  formatted by yapf

*  fix lint error
2018-04-19 22:36:29 -07:00
Stephanie Wang
aa07f1ce4e [xray] Workers blocked in a ray.get release their resources (#1920)
* [xray] Throttle task dispatch by required resources
* Pass in number of initial workers into raylet command
* Workers blocked in a ray.get release resources
2018-04-18 20:59:58 -07:00
Alexey Tumanov
1c965fcfeb Raylet task dispatch and throttling worker startup (#1912)
* separate task placement and task dispatch; throttle task dispatch with locally available resournces

* keep track of worker's being started/in flight and suppress starting extraneous workers

* cleanup comments

* remove early termination in task dispatch to support zero-resource actor tasks

* info -> debug

* add documentation

* linting

* mock the worker pool for testing

* some linting

* kill all workers in flight; clear the worker pool in dtor

* remove fixed todo

* lint
2018-04-18 10:58:11 -07:00
Omkar Salpekar
0728d4719b [DataFrame] Eval fix (#1903)
* eval now works without assignment - helper function a bit hacky

* removed df.copy() from eval_helper

* one test still failing for qury

* all eval tests passing now

* added check to eval arge verification

* added tests to travis

* added optimization and some comments

* added pd.eval and passes all tests

* added ray dataframe back to test file

* optimizations and code cleanup for eval

* changed position of pandas import in __init__

* fixed linting errors

* fixing eval in __init__.py

* fixed travis file - removed extra tests

* removed test directory from linting exclude for travis
2018-04-18 08:48:32 -07:00
Richard Liaw
f833e4da37
[tune] Polishing docs (#1846) 2018-04-17 09:57:35 -07:00
Eric Liang
7ab890f4a1 [tune] [rllib] Automatically determine RLlib resources and add queueing mechanism for autoscaling (#1848) 2018-04-16 16:58:15 -07:00
Eric Liang
ed8c0f1a38
[tune] Allow fetching pinned objects from trainable functions (#1895)
* updates

* lint

* Update util.py

* Update function_runner.py

* updates
2018-04-16 15:54:38 -07:00
Devin Petersohn
3c817ad908 Add slice functionality (#1832) 2018-04-16 08:50:56 -07:00
Patrick Yang
f505f0642f [DataFrame] Pass read_csv kwargs to _infer_column (#1894)
* pass kwargs to _infer_column

* adding small test for non-comma delim

* fix lint
2018-04-16 08:47:30 -07:00
Robert Nishihara
6ca2c2a609 Allow numpy arrays to be passed by value into tasks (and inlined in the task spec). (#1816)
* Allow numpy arrays and larger objects to be passed by value in task specifications.

* Fix bug.

* Fix bug. Inline all bug numpy object arrays.

* Increase size limit for inlining args in task spec.

* Give numpy init different signatures in Python 2 and Python 3.

* Simplify code.

* Fix test.

* Use import_array1 instead of import_array.
2018-04-15 20:36:01 -07:00
Robert Nishihara
7792032ee3 Fix UI issue for non-json-serializable task arguments. (#1892)
* Fix UI issue for non-json-serializable task arguments.

* Simplify approach.
2018-04-15 13:54:42 -07:00
Robert Nishihara
6f8b81d9e5 Allow multiple raylets to be started on a single machine. (#1904) 2018-04-15 13:51:19 -07:00
Peter Schafhalter
1d605e8f8a [DataFrame] Inherit documentation from Pandas (#1727)
* Added _inherit_docstrings

* DataFrame documentation inherits from Pandas

* Fix formatting

* Replace hasattr and document properties

* Fix rebase

* Override documentation for groupby

* Override documentation for series

* Don't overwrite property docstrings

* Fix property __doc__ for python2
2018-04-12 20:30:19 -07:00
Alexey Tumanov
39cf6ff6e1 raylet command line resource configuration plumbing (#1882)
* raylet command line resource configuration plumbing

* Small changes.
2018-04-12 02:37:15 -07:00
Alexey Tumanov
85d3963172 use raylet for remote ray nodes (#1880) 2018-04-11 22:06:46 -07:00
Eric Liang
4dc04374f6
[rllib] Propagate dim option to deepmind wrappers (#1876)
* updates

* updates
2018-04-11 21:38:06 -07:00
alvkao58
15a668dd12 [RLLib] DDPG (#1685) 2018-04-11 15:08:39 -07:00
Philipp Moritz
74162d1492 Lint Python files with Yapf (#1872) 2018-04-11 10:11:35 -07:00
Omkar Salpekar
a3ddde398c [DataFrame] Fixed repr, info, and memory_usage (#1874)
* working with dataframes with too many rows and columns

* repr works for jupyter notebooks now

* added comments and test file

* added repr test file to .travis.yml

* added back ray.dataframe as pd to test file

* fixed pandas importing issues in test file

* getting the front and back of df more efficiently

* only keeping dataframe tests in travis

* fixing numpy array for row and col lengths issue

* doesn't add dimensions if df is small enough

* implemented memory_usage()

* completed memory_usage - still failing 2 tests

* only failing one test for memory_usage

* all repr and dataframes tests passing now

* fixing error related to python2 in info()

* fixing python2 errors

* fixed linting errosr

* using _arithmetic_helper in memory_usage()

* fixed last lint error

* removed testing-specific code

* adding back travis test

* removing extra tests from travis

* re-added concat test

* fixes with new indexing scheme

* code cleanup

* fully working with new indexing scheme

* added tests for info and memory_usage

* removed test file
2018-04-11 08:07:07 -07:00
Devin Petersohn
806b2c844e Fix getattr compat (#1871) 2018-04-10 21:28:59 -07:00
Patrick Yang
521b549e4a [DataFrame] Encapsulate index and lengths into separate class (#1849)
* baseline impl for index_df.py

* added skeleton for index_df.py

* initial impl index_df

* separate out partition and non-partition impls

* add len function

* drop returns index_df slice of dropped indices

* housecleaning

* Integrate index overhaul

* Rename index df to index metadata

* Fix flake8 issues

* Addressing issues

* fix import issue

* Added metadata passing to constructor
2018-04-10 14:30:20 -07:00
Peter Schafhalter
405b05d58a [DataFrame] Implemented __getattr__ (#1753)
* __getattr__ accesses columns

* Added test
2018-04-10 10:19:33 -07:00
adgirish
efeaacbedc Adding support for concat (#1739)
adding tests

fixing flake8

adding init

flake 8 on test

fixing tests, imports, and flake8

handling for index

adding tests for row, index

added more robust error handling for axis

fixing test failures

cleaning up error sfor 2.7

updating travis

resolving import

fixing flake8

moved import order

Fixing to refactor and delaying implementing ray-pd inner concat

resolving ray-pd concat and from_pandas mutation

Revert "resolving ray-pd concat and from_pandas mutation"

This reverts commit 5db43e4e89e328286532f3ef98a4526575c5d08d.
2018-04-09 21:36:24 -07:00
Philipp Moritz
834e594709 [XRay] Register object store and raylet with the GCS (#1860) 2018-04-09 18:56:33 -07:00
Robert Nishihara
7c9e291b4b In the UI, display task breakdowns by default. (#1857) 2018-04-09 13:24:38 -07:00
Jones Wong
6b85d15bd9 supplement missed optimizer argument clip_rewards in default DQN configuration (#1853) 2018-04-08 17:58:05 -07:00