Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: use eval expression parsing as replacement for Term in HDFStore #4155

Closed
wants to merge 48 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
89a03be
ENH: add new computation module and toplevel eval function
cpcloud Jun 16, 2013
bcd17b0
ENH/TST: add new instance testing functions and their tests
cpcloud Jun 16, 2013
81bacd1
BUG: prevent certain index types from joining with DatetimeIndex
cpcloud Jun 16, 2013
e380271
TST/ENH: add 2d bare numpy array and nan support
cpcloud Jun 16, 2013
99a3d28
ENH: add modulus support
cpcloud Jun 17, 2013
4db95fe
TST: add failing modulus tests
cpcloud Jun 17, 2013
6000c89
CLN: use format string for unicode
cpcloud Jun 18, 2013
c25a1d4
CLN: remove engine detection and manip for datetimes
cpcloud Jun 18, 2013
1132bc4
CLN/ENH: add new interface to encapsulate Terms and Constants
cpcloud Jun 20, 2013
54f1897
ENH: allow an already-parsed expression to be passed to eval
cpcloud Jun 20, 2013
e20900a
CLN: add automatic scope creating object
cpcloud Jun 26, 2013
51d80f6
CLN: make the environment an implementation detail
cpcloud Jun 28, 2013
038d79c
DOC: add docstring to eval
cpcloud Jun 28, 2013
599cf32
CLN: cleanup pytables.py a bit
cpcloud Jun 28, 2013
ea769e6
CLN: clean up engines
cpcloud Jun 29, 2013
ff78c08
CLN: clean up eval and have the Scope instance auto create the scope …
cpcloud Jul 4, 2013
f9f7fd7
CLN: add six.string_types checking instead of basestring
cpcloud Jul 4, 2013
48eff13
TST: clean up some tests, add minor assertions where none existed
cpcloud Jul 4, 2013
d87f027
CLN: clean up frame.py a bit
cpcloud Jul 4, 2013
5b58a08
CLN: clean up pytables arguments a bit
cpcloud Jul 4, 2013
7482a27
CLN: use shiny new string mixin to refactor repring
cpcloud Jul 4, 2013
0d40fe1
CLN: move align to its own file
cpcloud Jul 4, 2013
87957d2
CLN: clean up and use new stringmixin for Expr
cpcloud Jul 4, 2013
e35cb5c
ENH/CLN: be more careful about unicode
cpcloud Jul 4, 2013
1ceec39
CLN: run autopep8 on pandas/io/pytables.py
cpcloud Jul 4, 2013
c665a85
DOC: reference future enhancingperf.eval section
cpcloud Jul 4, 2013
cb27934
CLN/DOC: clean up docstrings in pytables
cpcloud Jul 4, 2013
63ba37d
CLN: actually pass fletcher32 in get_store
cpcloud Jul 4, 2013
dcde590
CLN: remove unused variables
cpcloud Jul 4, 2013
3c4e2b3
CLN: more pep8 and get rid of most raise Exception clauses
cpcloud Jul 4, 2013
226c786
CLN: change NameError to match python
cpcloud Jul 4, 2013
79871d8
API: expose the Expr object to top level pandas
cpcloud Jul 5, 2013
84fdb45
CLN/TST: fail with a NotImplementedError on and or not
cpcloud Jul 5, 2013
4d9f9a7
CLN: generlize operator/expression printing
cpcloud Jul 5, 2013
a0d2ce0
CLN: clean up testing and expr
cpcloud Jul 5, 2013
317a153
ENH: add modest type inference
cpcloud Jul 6, 2013
401bc28
ENH: rewrite assignment as equal comparison
cpcloud Jul 6, 2013
22dedcb
ENH: initial commit for adding Expr based terms for pytables support
jreback Jul 6, 2013
441285c
WIP: still some debugging statements in
jreback Jul 7, 2013
05a005f
WIP: conditions working now, filtering still only ok
jreback Jul 7, 2013
22b4a93
TST: more test changes
jreback Jul 7, 2013
ca292c2
BUG: added HDFStore to inherit from Stringmixin
jreback Jul 7, 2013
dfef617
BUG: process visit_Index
jreback Jul 7, 2013
b168fb3
ENH: use non_implemented function call in ExprVisitor
jreback Jul 7, 2013
5fac749
BUG: fixed scoping issues by _ensure_term at the top-level
jreback Jul 7, 2013
c5a3c9f
TST: fixed remaining tests
jreback Jul 7, 2013
71a23a8
BUG: py3 fixes; revise scoping rules to be more broad
jreback Jul 8, 2013
e712762
COMPAT: allow prior 0.12 query syntax for terms, e.g. Term('index','>…
jreback Jul 8, 2013
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
CLN: clean up and use new stringmixin for Expr
  • Loading branch information
cpcloud committed Jul 6, 2013
commit 87957d24f08f09f2f4a8574e435d4a9dad75ec55
36 changes: 13 additions & 23 deletions pandas/computation/expr.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@
import sys
from functools import partial


from pandas.computation.ops import BinOp, UnaryOp, _reductions, _mathops, Mod
from pandas.core.base import StringMixin
from pandas.computation.ops import BinOp, UnaryOp, _reductions, _mathops
from pandas.computation.ops import _cmp_ops_syms, _bool_ops_syms
from pandas.computation.ops import _arith_ops_syms, _unary_ops_syms
from pandas.computation.ops import _resolve_name, Term, Constant
from pandas.computation.ops import Term, Constant


class Scope(object):
Expand Down Expand Up @@ -51,8 +51,8 @@ def __init__(self, env):

def visit(self, node):
if not (isinstance(node, ast.AST) or isinstance(node, basestring)):
raise AssertionError('"node" must be an AST node or a string, you'
' passed a(n) {0}'.format(node.__class__))
raise TypeError('"node" must be an AST node or a string, you'
' passed a(n) {0}'.format(node.__class__))
if isinstance(node, basestring):
node = ast.fix_missing_locations(ast.parse(node))
return super(ExprVisitor, self).visit(node)
Expand Down Expand Up @@ -81,8 +81,7 @@ def visit_UnaryOp(self, node):
return op(self.visit(node.operand))

def visit_Name(self, node):
name = node.id
return Term(_resolve_name(self.env, name), name, self.env)
return Term(node.id, self.env)

def visit_Num(self, node):
return Constant(node.n, self.env)
Expand All @@ -108,16 +107,14 @@ def visit_Call(self, node):
def visit_Attribute(self, node):
raise NotImplementedError("attribute access is not yet supported")

def visit_Mod(self, node):
return Mod


class Expr(object):
class Expr(StringMixin):
"""Expr object"""
def __init__(self, expr, engine='numexpr', env=None, truediv=True):
def __init__(self, expr, engine='numexpr', env=None, truediv=True,
parsing='strict'):
self.expr = expr
self.env = env or Scope(frame_level=2)
self._visitor = ExprVisitor(self.env)
self._visitor = ExprVisitor(self.env, parsing)
self.terms = self.parse()
self.engine = engine
self.truediv = truediv
Expand All @@ -126,19 +123,12 @@ def __call__(self, env):
env.locals['truediv'] = self.truediv
return self.terms(env)

def __repr__(self):
return '{0} -> {1}'.format(self.expr, self.terms)

def __str__(self):
return self.expr
def __unicode__(self):
return unicode(self.terms)

def parse(self):
"""return a Termset"""
try:
visited = self._visitor.visit(self.expr)
except SyntaxError as e:
raise e
return visited
return self._visitor.visit(self.expr)

def align(self):
"""align a set of Terms"""
Expand Down