Means to expose built-in types to Python#205
Conversation
We introduce a class (and its test) that will take argument lists as they might be provided to methods and functions, and arranges them in an array. Variants are given intended to cover any Python 3 calling pattern.
We add the apparatus to describe methods by their Python arguments. This is the intermediate representation on the way to exposing each as an object. We test this intermediate structure holds a good ArgParser. Note that we must ask the compiler to preserve argument names for run time to get a Python-like signature.
|
The next installment of the story involves enumerating the methods of a type we wish to expose. We build an argument parser for each. For this purpose, we supply a set of annotations to pick out the methods in question and make choices that Java does not offer us, such as where the keyword-only parameters begin. We process these to create a specification ( At this stage we do not yet have anything we can call. We just check we get the right |
We add method signatures that include argument collectors (varargs and varkeywords) and switch to using parameterised tests to make selective debugging easier.
We introduce types that represent the methods of built-in types, and their bound counterparts, and which can be created from the type exposer. We demonstrate in a test that we can call the target method through the __call__ method on both the unbound descriptor and the bound method.
|
This installment defines the public-facing classes for method descriptors and functions (or bound methods). We show that we can call them from Java by their classic We cannot yet call from Python byte code. |
We change PyCode and PyFrame so that identifiers (names, keywords) appearing in code are represented as interned Java String, since we expect this to be more efficient in look-up and comparison. The marshal module gives this interpretation to the 't' type, which CPython uses for identifiers.
We decorate a small number of methods in PyUnicode with the annotations from Exposed.java, to define their Python calling semantics (positional parameters, defaults, keywords). We implement the several CALL_* byte codes and demonstrate them in a test.
|
We can now expose methods of built-in ("crafted") Python types where the methods have been annotated. We have (I think) the full expressive power of Python method signatures. Initially I thought I would include exposure of members and get-set attributes, but this is huge already, so that's another PR. We need this, the follow-up members and get-sets, and probably module exposure, to support the AST work, so I propose waiting a week (probably while I release a 2.7.3rc1) then I'll present the next stage as a new PR. |
|
@fwierzbicki : pinging you here more for your interest than to invite a line-by-line review. I think this, and more pertinently a follow-up to support members and attributes, is on the path to being able to generate an AST, since AST nodes surely have exposed methods and attributes. Modules use the same annotations, but I'm struggling a bit with the exposer for those. |
1 participant
This is going to be quite a big PR.
So far, we can expose the special methods of built-in types through slots, so that they may be called by the interpreter. They are recognised by their special name and signature when building a type.
In order to give built-in types their methods, we have to support names and signatures that are arbitrary (within the rules of Python). In Jython 2 we have an exposure mechanism that creates a class for each method in which
__call__is given a body that calls the specific method wrapped, with generated argument coercion (I think). This was the only available approach before theMethodHandlewas added to Java for the benefit of dynamic language implementers. (That's us!)The mechanism for this (the Jython 2 exposer) is not well documented and produces code that cannot be debugged (in my experience). I do not know how I would adapt it to create
MethodHandles. So the Very Slow Jython Project took a different direction, which:MethodHandleone could use in a dynamic call site,This PR ports a big chunk of that to the main Jython repo. We cannot evidence a claim that this mechanism is faster than calls in Jython 2. (I'm quietly confident it won't be slower.)
This PR will introduce the necessary moving parts by tested stages. The first part of the mechanism is a parser for arguments in the form they appear on the interpreter stack, or could be presented from the Java stack. In practice we'll look for short-cuts in common cases (small number of arguments by position) but the general mechanism has to be there first. It is also good for filling the frame of a method defined in Python. The use of a single mechanism means that the full range of Python signatures is available to built-in types and modules.