Feat/comp/get input format (#2315)

* -

* • composition.py:
  - _update_controller: added
  - add_controller and _analyze_graph(): call _update_controller

* -

* • composition.py
  _update_controller:  fixed to loop through all input_ports of comp INPUT nodes

* • test_control.py
  - test_agent_rep_assignement_as_controller_and_replacement:
      updated to test that shadowing projections to state_input_ports
      are properly added and deleted

* • optimizationfunctions.py:
  - _function: refactored to put use aggregation_function at end
  - _grid_evaluate:  still needs to return all_samples

* -

* • composition.py
  - added call to _update_controller to add_node
  - moved test for projections to controller.state_input_ports to run()

* -

* • composition.py:
  moved calls to _update_controller to _complete_init_of_partially_initialized_nodes
  moved _update_controller to ocm._update_state_input_ports

• optimizationcontrolmechanism.py:
  added _update_state_input_ports [**still needed work**]

* • composition.py:
  moved calls to _update_controller to _complete_init_of_partially_initialized_nodes
  moved _update_controller to ocm._update_state_input_ports
  _instantiate_controller_shadow_projections [still needs to be implemented]

• optimizationcontrolmechanism.py:
  added _update_state_input_ports [**still needed work**]

* • composition.py
  added needs_update_controller

* -

* • composition.py:
  - implemented self.needs_update_controller
  - moved implementation of controlsignal projections
    from add_controller to _instantiate_control_projections
    that is called in _complete_init_of_partially_initialized_nodes

Note: still need to set  self.needs_update_controller to False
      after instantiating state_input_ports and projections to them

* -

* -

* -

* -

* -

* -

* -

* -

* • Passing all test_control tests except test_mode_based_num_estimates

* • Passing all test_control tests

* -

* • optimizationcontrolmechanism.py
  - _update_state_input_ports_for_controller: handle nested input nodes

* -

* • optimizationcontrolmechanism.py
  _update_state_input_ports_for_controller:  fixed bug with > 1 INPUT node in Composition

* • test_show_graph.py: passes all tests

* -

* • test_report.py:  passing all tests

* • Passes all tests!

* -

* -

* • composition.py: reorganize with #region and #enregions

* • composition.py: reorganize with #region and #enregions

* • controlmechanism.py, optimizationcontrolmechanism.py:
  - _instantiate_monitor_for_control_input_ports -> _parse_monitor_control_input_ports
  - refactored to support allow_probes option on ocm

* -

* -

* -

* • controlmechanism.py, optimizationcontrolmechanism.py:
  - _instantiate_monitor_for_control_input_ports -> _parse_monitor_control_input_ports
  - refactored to support allow_probes option on ocm

* • controlmechanism.py, optimizationcontrolmechanism.py:
  - _instantiate_monitor_for_control_input_ports -> _parse_monitor_control_input_ports
  - refactored to support allow_probes option on ocm

* -

* • composition.py:
  __init__: move controller to after add_nodes and add_linear_pathway

* -

* - test_control: only test_hanging_control_spec_outer_controller not passing

* -

* -

* -

* -

* -

* -

* • composition.py:
  _instantiate_control_projections:
     weird requirement for double-call to controller._instantiate_control_signal

* • test_paremtercomposition.py:
  restored parameter spec that causes crash ('threshold',Decision2)

* ª Attempt to fix problem with partially overlapping local and ocm control specs

- composition.py
  - _get_control_signals_for_composition:  (see 11/20/21)
      - added (but commented out change) to "if node.controller" to "if not node.controller"
      - changed append to extend
  - _instantiation_control_projection:
      - got rid of try and except double-call to controller._instantiate_control_signals
      -  outdented call to self.controller._activate_projections_for_composition at end

- controlmechanism.py:
    - _check_for_duplicates:  add warning and return duplicates

- optimizationcontrolmechanism._instantiate_control_signals:
    - add call to self.agent_rep._get_control_signals_for_composition() to get local control specs (on mechs in comp)
    - eliminate duplicates with control_signal specs on OCM
    - instantiate local + ocm control_signals

- parameterestimationcomposition.py
  - added context to various calls

* see later commit

* see later commit

* see later commit

* see later commit

* - This branch passes all tests except:
   - test_parameterestimationcomposition
   - test_composition/test_partially_overlapping_control_specs (ADDED IN THIS COMMINT)

- All relevant changes to this branch are marked as "11/21/21."
  However, most are commented out as they break other things.

- The tests above both involve local control specifications (on mechanism within a nested comp)
  and on the OCM for the outer composition, some of which are for the same nested mechs

- Both tests fail with:
   "AttributeError: 'NoneType' object has no attribute '_get_by_time_scale'" (in component.py LINE 3276)
   This may be due to a problem with context setting, since the error is because the modulation Parameter
   of the ControlProjection is returning "None" rather than "multiplicative_param" (when called with get(context)),
   whereas "multiplicative_param" is returned with a call to get() (i.e., with no context specified)

- Most of test_partially_overlapping_control_specs is passed if
   changes marked "11/21/21 NEW" in optimizationcontrolmechanism.py (LINE 1390) are implemented,
   but it does not properly route ControlProjections through parameter_CIMS (see last assert in test).
   Furthermore, test_parameterestimationcompsition fails with the mod param error, even though the
   model has similar structure (i.e., outer composition -- in this case a ParameterEstimationComposition)
   with an OCM that is given control specs that overlap with ones in a nested composition.

- There are also several other things in composition I found puzzling and tried modifying, but that cuased failures:
  - _get_control_signals_for_composition():
      - seems "if node.controller" should be "if **not** node.controller" (emphasis added just for comment)
      - "append" should be "extend"
  - _instantiate_control_projection():
      -  call to self.controller._activate_projections_for_composition (at end of method) should not be indented

* - small mods; don't impact anything relevant to prior commit message

* - small mods; don't impact anything relevant to prior commit message

* - small mods; don't impact anything relevant to prior commit message

* - finished adding formatting regions to composition.py

* -

* • composition.py:
  - rename _check_projection_initialization_status -> _check_controller_initialization_status
  - add _check_nodes_initialization_status(context=context)
    (and calls it with _check_controller_initialization_status)

* • show_graph.py:  addressed bug associated with ocm.allow_direct_probe

* • show_graph.py:  addressed bug associated with ocm.allow_direct_probe

* -

* Composition: add_controller: set METHOD as context source early

* -

* • composition.py
  retore append of control_signals in _instantiate_control_projections()

* • composition.py
  restore append of control_signals in _instantiate_control_projections()

• test_composition.py:
  add test_partially_overlapping_local_and_control_mech_control_specs_in_unnested_and_nested_comp

* • test_partially_overlapping_local_and_control_mech_control_specs_in_unnested_and_nested_comp():
  - added clear_registry() to allow names to be reused in both runs of test

* • composition.py
  docstring:  added projections entry to list of attributes
  - add_controller:  added call to _add_node_aux_components() for controller

* • composition.py
  _add_node_aux_components(): added deletion of item from aux_components if instantiated

* • composition.py
  - comment out _add_node_aux_components() (causing new failures)
  - move _instantiate_control_projections to be with _instantiate_control_projections,
       after self.add_node(self.controller.objective_mechanism (to be more orderly)

* -

* - confirm that it passes all tests exception test_composition/test_partially_overlapping...
  (with addition of _add_aux_components in add_controller commented out)

* • composition.py:  some more fixed to add_controller that now fail only one test:
    - test_agent_rep_assignement_as_controller_and_replacement

* • Passes *all* current tests

* • composition.py:
  - add_controller:  few more minor mods;
  still passes all tests

* -

* -

* -

* • controlmechanism.py:
  - __init__: resrict specification to only one of control, modulatory_signals, or control_signals (synonyms)

* -

* • composition.py:  in progress fix of bug in instantiating shadow projections for ocm.state_input_ports

* • composition.py:
  - _get_original_senders():  added support for nested composition
    needs to be checked for more than one level
    needs to be refactored to be recursive

* • optimizationcontrolmechanism.py
  - _update_state_input_ports_for_controller:  fix invalid_state_features to allow input_CIM of nested comp in agent_rep

* -

* • composition.py
 - _get_original_senders: made recursive

* • test_show_graph.py: update for fixes

* -

* • tests:  passes all in test_show_graph.py and test_report.py

* Passes all tests

* - comment clean-up

* • composition.py
 - add_controller and _get_nested_node_CIM_port:
   added support for forced assignment of NodeRole.OUTPUT for nodes specified in OCM.monitor_for_control,
   but referenced 'allow_probes' attribute still needs to be implemented

* • composition.py, optimizationcontrolmechanism.py:
  allow_probes fully implemented

* • show_graph.py:  fixed bug causing extra projections to OCM

* • composition.py:
  - _update_shadow_projections(): fix handling of deep nesting

* • optimizationcontrolmechanism.py: add agent_rep_type property

* • optimizationcontrolmechanism.py:
  - state_feature_function -> state_feature_functions

* • optimizationcontrolmechanism.py:
  - _validate_params:  validate state_feature_functions
  - _update_state_input_ports_for_controller: implement assignment of state_feature_functions

* -

* -

* • Passes all tests except test_json with 'model_with_control'

* -

* • composition.py
  - add_projection:  delete instantiation of shadow projections (handled by _update_shadow_projections)

* • composition.py
  - add_projection:  delete instantiation of shadow projections (handled by _update_shadow_projections)
  - remove calls to _update_shadows_dict

* • composition.py
  - add_projection:  delete instantiation of shadow projections (handled by _update_shadow_projections)
  - remove calls to _update_shadows_dict

* -

* • test_two_origins_two_input_ports:  crashes on failure of C->B to update

* -

* • composition.py
  - added property shadowing_dict that has shadowing ports as keys and the ports they shadow as values
  - refactored _update_shadowing_projections to use shadowing_dict

* • optimizationcontrolmechanism.py
  - _update_state_input_ports:  modified validations for nested nodes;  still failing some tests

* • optimizationcontrolmechanism.py
  - _update_state_input_ports:
     more careful and informative validation that state_input_ports are in comp or nested comp and are INPUT nodes thereof;
     passes all tests except test_two_origins_two_input_ports as before

* • composition.py
  _get_invalid_aux_components():  defer all shadow projections until _update_shadow_projections

* • composition.py
  _get_invalid_aux_components():  bug fix in test for shadow projections

* Port: _remove_projection_to_port: don't reduce variable below length 1

even ports with no incoming projections have variable at least length 1

* • composition.py
  add_node(): marked (but haven't removed) code block instantiating shadow_projections
   that seems now to be redundant with _update_shadow_projection

* • show_graph.py
  - _assign_cim_components: supress showing projections not in composition

* • composition.py:
  _analyze_graph():  add extra call to _determine_node_roles after _update_shadow_projections
  _run():  moved block of code at beginning initializing scheduler to after
           _complete_init_of_partially_initialized_nodes and _analyze_graph()

• show_graph.py
  - add test to all loops on projections:  "if proj in composition.projection"

* • show_graph.py
  - add show_projections_not_in_composition option for debugging

* • composition.py
  _update_shadow_projections(): delete unused shadow projections and corresponding ports

* • composition.py
  _update_shadow_projections(): fix bug in deletion of unused shadow projections and ports

• test_show_graph:  tests failing, need mods to accomodate changes

* • composition.py:
  _analyze_graph():  add extra call to _determine_node_roles after _update_shadow_projections
  _run():  moved block of code at beginning initializing scheduler to after
           _complete_init_of_partially_initialized_nodes and _analyze_graph()

• show_graph.py
  - add test to all loops on projections:  "if proj in composition.projection"

* • show_graph.py
  fixes; now passes all show_graph tests

* -

* • composition.py
  _update_shadow_projections:  raise error for attempt to shadow INTERNAL Node of nested comp

* -

* -

* • test_composition.py
  implemented test_shadow_nested_nodes that tests shadowing of nested nodes

* -

* -

* -

* -

* • optimizationcontrolmechanism.py: docstring mods

* • composition.py:
  - add allow_probes and exclude_probes_from_output

* • composition.py:
  - docstring mods re: allow_probes

• optimizationcontrolmechanism.py:
  - allow_probes:  eliminate DIRECT setting
  - remove _parse_monitor_for_control_input_ports (no longer needed without allow_probes=DIRECT)

* • composition.py:
  - change "exclude_probes_from_output" -> "include_probes_in_output"

* • composition.py:
  - docstring mods re: allow_probes and include_probes_in_output

* • composition.py:
  - docstring mods re: allow_probes and include_probes_in_output

* • controlmechanism.py:
  - add allow_probes handling (moved from OCM)

• optimizationcontrolmechanism.py:
  - move allow_probes to controlmechanism.py

• composition.py:
  - refactor handling of allow_probes to permit for any ControlMechanism

• objectivemechanism.py:
  - add modulatory_mechanism attribute

* • controlmechanism.py:
  - add allow_probes handling (moved from OCM)

• optimizationcontrolmechanism.py:
  - move allow_probes to controlmechanism.py

• composition.py:
  - refactor handling of allow_probes to permit for any ControlMechanism
  - add _handle_allow_probes_for_control() to reconcile setting on Composition and ControlMechanism

• objectivemechanism.py:
  - add modulatory_mechanism attribute

* • composition.py
  add assignment of learning_mechanism to objective_mechanism.modulatory_mechanism for add_learning methods

* • docstring mods

* -

* -

* • optimizationcontrolmechanism.py: docstring revs

* -

* -

* • test_composition.py:
  - add test_unnested_PROBE
  - add test_nested_PROBES
    TBD: test include_probes_in_output

* -

* • composition.py
  - add_node():  support tuple with required_role

* -

* • composition.py:
  - _determine_node_roles:
     fix bug in which nested comp was prevented from being an OUTPUT Node if,
     in addition to Nodes that qualifed as OUTPUT, it also had nodes that projected
     to Nodes in an outer comp (making it look like it was INTERNAL)

* -

* • composition.py:
  - add_node(): enforce include_probes_in_output = True for nested Compositions
  - execute():
    - replace return of output_value with get_output_value()

* -

* • CompositionInterfaceMechanism.rst:
  - correct path ref

• compositioninterfacemechanism.py:
  - docstring fixes

* • composition.py:
  get_input_format(): add template option to return formatted dict for use as input to run()

* -

* -

* • composition.py, mechanism.py:
  - add external_input_variables properties
  - add get_input_variables() methods
  replace use of input.value with input.varible for validating/sizing inputs

* -

* -

* • function.py:
  Function_Base:
  - add input_shape_template attribute (as experiment for now)

* • mechanism.py:
  - add:
    _input_shape_template, default_input_shape, input_shape, get_input_shape,
    default_external_input_shape, external_input_shape
  - _get_variable_from_input(): refactor to use above instead of variable or value

* • mechanism.py, inputport.py:
  move from mechanism to inputport: _input_shape_template, default_input_shape, input_shape, get_input_shape

* -

* -

* -

* -

* -

* -

* • mechanism.py:
  - get_variable_from_input():
    - refactor to assign each item of input to corresponding input_port.variable
      and each input_port.value to result of executing its function with the variable

* -

* -

* -

* -

* • Passes all tests

* • Passes all tests

• function.py:
  input_shape_template -> changes_shape

* -

* • test_composition.py:
  add: test_get_input_format()

* - Before adding trials and labels to template to Composition.get_input_format()

* - Before adding trials and labels to template to Composition.get_input_format()

* -

* -

* -

* - Before adding trials and labels to template to Composition.get_input_format()

* -

* - Before adding trials and labels to template to Composition.get_input_format()

* - Before adding trials and labels to template to Composition.get_input_format()

* -

* -

* -

* -

* -

* -

* -

* -

* -

* -

* -

* -

* • composition.py:
  - get_input_format():
    - add form option that returns dict formatted for use as **inputs** arg of run() method
    - support dict that specifies labels for entire mech, not ports individually
    - _parse_labels(): fix bugs to support above

• utilities.py:
  - nested_depth(): returns depth of ragged lists and np.ndarrays

• test_composition.py:
  - add test_get_input_format()

Co-authored-by: jdcpni <pniintel55>
Co-authored-by: Katherine Mantel <kmantel@princeton.edu>

psyneulink/core/components/mechanisms/mechanism.py

@@ -3973,7 +3973,7 @@ def output_port(self):
 
     @property
     def output_values(self):
-        return self.output_ports.values
+        return self.get_output_values()
 
     def get_output_values(self, context=None):
         return [output_port.parameters.value.get(context) for output_port in self.output_ports]
@@ -3993,8 +3993,8 @@ def get_output_labels(self, context=None):
         elif context:
             return self.get_output_values(context)
         else:
-            return self.output_values
-
+            # Use this to report most recent value if no context is available
+            return self.output_ports.values
 
     @property
     def ports(self):