yarp_guts.dox

/*
 * Copyright (C) 2010 RobotCub Consortium
 * CopyPolicy: Released under the terms of the LGPLv2.1 or later, see LGPL.TXT
 *
 */

/**
 * @page yarp_guts A trip through the guts of YARP

@author Paul Fitzpatrick

The goal of the tutorial is to help people find their way
around the internals of the YARP implementation.
Most people will never need to read it.
It could be useful if you want to extend or modify YARP in
radical ways.

Beware: classes that are deemed to be "internal" (basically, anything in
namespace yarp::os::impl, yarp::sig::impl, etc) are more likely to
change as time goes by than other classes.

@section yarp_guts_facade Behind the facade

Many YARP user-facing classes have private implementations, that is they
follow this pattern:

\code
class UserFacingClass {
   ...
private:
  void *implementation;
};
\endcode

There are two reasons for this.  One is ABI stability, so that
significant changes can
safely be made to the underlying implementation.
Another reason is to insulate regular users of YARP from
ACE header files.  We use the ACE library in the implementation
of YARP because of its portability,
but it can turn proper compilation into an art-form.
We don't want to force our users to deal with that (and
don't want to deal with the confused emails they will send
us).  So we use ACE in the YARP implementation, but do not
reference it in user-facing header files.

@section yarp_guts_summary Summary of important implementation classes

For every port, there is a yarp::os::impl::PortCore object.
The user-facing yarp::os::Port class is a superficial wrapper around this.

For every connection between two ports (call them the source port
and the target port), there is a
yarp::os::impl::PortCoreOutputUnit object held in a list by the
source port's PortCore, and a yarp::os::impl::PortCoreInputUnit
object held in a list by the target port's PortCore.

Every PortCoreUnit has a yarp::os::InputProtocol
or yarp::os::OutputProtocol object that
manages the overall phases of YARP's network protocol.

The Protocol object delegates to an
object from the yarp::os::Carrier family
(yarp::os::impl::TcpCarrier, yarp::os::impl::McastCarrier, ...) to
manage the peculiarities of a particular connection type.  This
carrier can be replaced during the operation of a connection.
Typically, connections start with a TcpCarrier, and then switch
(after hand-shaking) to whatever carrier the user has selected.

Here's an example diagram of four ports.  Port 1 is connected by
multicast to ports 2 and 3, and by tcp to port 4.

  \dot
  digraph guts_example1 {
      node [shape=box, fontname=Helvetica, fontsize=10];
      edge [fontname=Helvetica, fontsize=10, arrowhead="open", style="solid"];

      subgraph cluster_guts_port1 {
		label = "Port 1";
		color = "blue"; style = "dashed";

		portcore1 [ label="PortCore", shape=ellipse ];
		portcoreoutputunit11 [ label="PortCoreOutputUnit", shape=ellipse ];
		portcoreoutputunit12 [ label="PortCoreOutputUnit", shape=ellipse ];
		portcoreoutputunit13 [ label="PortCoreOutputUnit", shape=ellipse ];
		outputprotocol11 [ label="OutputProtocol", shape=ellipse ];
		outputprotocol12 [ label="OutputProtocol", shape=ellipse ];
		outputprotocol13 [ label="OutputProtocol", shape=ellipse ];
		mcastcarrier11 [ label="McastCarrier", shape=ellipse ];
		mcastcarrier12 [ label="McastCarrier", shape=ellipse ];
		tcpcarrier13 [ label="TcpCarrier", shape=ellipse ];
	}

      subgraph cluster_guts_port2 {
		label = "Port 2";
		color = "blue"; style = "dashed";

		portcore2 [ label="PortCore", shape=ellipse ];
		inputprotocol2 [ label="InputProtocol", shape=ellipse ];
		mcastcarrier2 [ label="McastCarrier", shape=ellipse ];
		portcoreinputunit2 [ label="PortCoreInputUnit", shape=ellipse ];
	}

      subgraph cluster_guts_port3 {
		label = "Port 3";
		color = "blue"; style = "dashed";

		portcore3 [ label="PortCore", shape=ellipse ];
		inputprotocol3 [ label="InputProtocol", shape=ellipse ];
		mcastcarrier3 [ label="McastCarrier", shape=ellipse ];
		portcoreinputunit3 [ label="PortCoreInputUnit", shape=ellipse ];
	}

      subgraph cluster_guts_port4 {
		label = "Port 4";
		color = "blue"; style = "dashed";

		portcore4 [ label="PortCore", shape=ellipse ];
		inputprotocol4 [ label="InputProtocol", shape=ellipse ];
		tcpcarrier4 [ label="TcpCarrier", shape=ellipse ];
		portcoreinputunit4 [ label="PortCoreInputUnit", shape=ellipse ];
	}

      portcore1 -> portcoreoutputunit11;
      portcore1 -> portcoreoutputunit12;
      portcore1 -> portcoreoutputunit13;
      portcoreoutputunit11 -> outputprotocol11;
      portcoreoutputunit12 -> outputprotocol12;
      portcoreoutputunit13 -> outputprotocol13;
      outputprotocol11 -> mcastcarrier11;
      outputprotocol12 -> mcastcarrier12;
      outputprotocol13 -> tcpcarrier13;

      mcastcarrier11 -> mcastcarrier2;
      mcastcarrier12 -> mcastcarrier3;
      tcpcarrier13 -> tcpcarrier4;

      mcastcarrier2 -> inputprotocol2;
      mcastcarrier3 -> inputprotocol3;
      tcpcarrier4 -> inputprotocol4;

      inputprotocol2 -> portcoreinputunit2;
      inputprotocol3 -> portcoreinputunit3;
      inputprotocol4 -> portcoreinputunit4;

      portcoreinputunit2 -> portcore2;
      portcoreinputunit3 -> portcore3;
      portcoreinputunit4 -> portcore4;

      fontname=Helvetica;
      fontsize=10;
      label = "A four-port example, arrows show data flow";
		color = "blue"; style = "dashed";
  }
  \enddot

This diagram shows logical data flow, but omits other relationships.
For example, all McastCarrier objects acting as outputs for the
same PortCore work together so that data only gets sent once on
a shared logical bus.  See \ref yarp_guts_mcast.


@section yarp_guts_create Port creation

Every yarp::os::Port or yarp::os::BufferedPort contains exactly one
yarp::os::impl::PortCore object. This is the main coordinator for
communication.  Every yarp::os::impl::PortCore, on start-up, initiates
one yarp::os::impl::Face object.  This object is currently always
of type yarp::os::impl::TcpFace, which represents a server socket.

The PortCore creates a thread to listen to the Face (the server socket).
Any initial communication with the port is made via the Face.

The PortCore maintains a list of yarp::os::impl::PortCoreUnit objects.
These represent incoming and outgoing connections.  When a request is
received via the Face to connect to some other port, the first
step is to create a new PortCoreUnit (either a
yarp::os::impl::PortCoreInputUnit or a yarp::os::impl::PortCoreOutputUnit).
Depending on the configuration of the port, a new thread may be created
to service that unit.  Inputs always get a new thread.  Outputs get
a new thread only if requested (see yarp::os::Port::enableBackgroundWrite;
this is called for BufferedPorts by default).

@section yarp_guts_connect Connection creation

When the Face object receives a communication from the outside world,
it wraps it in a Protocol object.  Concretely, this means that the TcpFace
server socket takes each socket it receives and gives it to the PortCore
as a Protocol object, which then creates a PortCoreUnit to manage
that Protocol object.  The abstraction is important here because
one of the first steps of the YARP network protocol allows connections
to switch from whatever the carrier used to establish
the connection is (TCP) to something different for transmitting
payloads (multicast, shared memory, etc).
The Protocol object is either
yarp::os::InputProtocol or yarp::os::OutputProtocol
(in fact both of these are interfaces implemented by a single
class, yarp::os::impl::Protocol, but most of YARP does not know that).

The Protocol object deals with switching to a particular carrier
(implemented by sub-types of yarp::os::Carrier), and
the details of the YARP network protocol.

@section yarp_guts_mcast Multicast elections

The two McastCarrier objects owned by Port 1
in the figure above "conspire" behind the
scenes to make sure only one of them sends data, but as far as the
rest of YARP is concerned there's nothing special about them.
This conspiracy is implemented via a global yarp::os::Election
object.

*
*/