Firmata unit testing

[soundanalogous]

It may also be helpful to create unit tests. These tests will run against individual units of code and are helpful in ensuring that everything is working as expected at a lower level. There is a lot of debate as to whether units tests should run on the microcontroller on the PC. The most interesting approaches I've seen are those that run individual or small collections of tests directly on the microcontroller, but also use a script to automatically load and run tests on the board. So you would create a test suite on the PC that loads and executes unit tests on the actual micocontroller board. Here are a couple of projects taking that approach:

• arduino_testsuite (python)
• ceedling (ruby)

Unit tests will also be specific to the microcontroller architecture whereas the integration tests will be architecture agnostic since they will test against the serial interface rather than the platform-specific implementation.

[soundanalogous]

If anyone wants to take this on, please start a discussion here before beginning work.

[mmurdoch]

Firmata can largely be unit tested using the following mechanisms:

• Use Firmata.begin(Stream&) with an in-memory stream to test serial reads and writes
• Attach callback functions for capturing and checking processed input data

The only functions which cannot be tested this way are blinkVersion() and pin13strobe() which directly use Arduino functions (pinMode() and digitalWrite()). Some small refactoring would likely be needed to test them.

Also this method won't test any functionality in StandardFirmata.ino. I'd suggest the best way to get this under test would be to move as much of it into a separate library which can be tested as above. Doing this will likely require wrapping the Arduino functions so that they can be stubbed out for testing. This would make StandardFirmata.ino into a very thin wrapper around this library. There could be overlap with this and the implementation for issue #25 (configurable firmata).

Here's a potential starting unit test sketch using ArduinoUnit (https://github.com/mmurdoch/arduinounit - a library I maintain). I'd be happy to add to these tests if we think this is a sensible approach . All the unit tests are of the form test(<test_name>):

#include <Firmata.h>
#include <ArduinoUnit.h>

// Allows full control of a stream for testing purposes
class InMemoryStream : public Stream
{
public:
  virtual ~InMemoryStream() 
  {
  }

  size_t write(uint8_t val)
  {
    _bytesWritten += (char) val;

    return size_t(1);
  }

  void flush() 
  {
  }

  const String& bytesWritten() 
  {
    return _bytesWritten;
  }

  void nextByte(byte b)
  {
    _nextByte = b;
  }

  int available() 
  {
    return 1;
  }

  int read() 
  {
    return _nextByte;
  }

  int peek()
  {
    return _nextByte;
  }

private:
  String _bytesWritten;
  byte _nextByte;
};

TestSuite suite;

void setup()
{
  Serial.begin(9600);
}

void loop()
{
  suite.run();
}

// Helper function for checking strings
void assertStringsEqual(Test& __test__, const char* expected, const String& actual)
{
  size_t expectedLength = strlen(expected);
  assertEquals(expectedLength, actual.length());
  for (size_t i = 0; i < strlen(expected); i++)
  {
    assertEquals(expected[i], actual[i]);
  }
}

test(beginPrintsVersion)
{
  InMemoryStream stream;

  Firmata.begin(stream);

  char expected[] = 
  {
    0xF9, // Version reporting identifier 
    2,    // Major version number
    3,    // Minor version number
    0
  };
  assertStringsEqual(__test__, expected, stream.bytesWritten());
}

// Helper function to simplify tests
void processMessage(const byte* message, size_t length)
{
  InMemoryStream stream;
  Firmata.begin(stream);

  for (size_t i = 0; i < length; i++)
  {
    stream.nextByte(message[i]);
    Firmata.processInput();
  }
}

// Record digital port and value for checking in a test
byte _digitalPort;
int _digitalPortValue;
void writeToDigitalPort(byte port, int value)
{
  _digitalPort = port;
  _digitalPortValue = value;
}

test(processWriteDigital_0)
{
  Firmata.attach(DIGITAL_MESSAGE, writeToDigitalPort);

  byte message[] = { DIGITAL_MESSAGE, 0, 0 };
  processMessage(message, 3);

  assertEquals(0, _digitalPortValue);
}

test(processWriteDigital_127)
{
  Firmata.attach(DIGITAL_MESSAGE, writeToDigitalPort);

  byte message[] = { DIGITAL_MESSAGE, 127, 0 };
  processMessage(message, 3);

  assertEquals(127, _digitalPortValue);
}

test(processWriteDigitalStripsTopBit)
{
  Firmata.attach(DIGITAL_MESSAGE, writeToDigitalPort);

  byte message[] = { DIGITAL_MESSAGE, B11111111, 0 };
  processMessage(message, 3);

  assertEquals(B01111111, _digitalPortValue);
}

test(processWriteDigital_128)
{
  Firmata.attach(DIGITAL_MESSAGE, writeToDigitalPort);

  byte message[] = { DIGITAL_MESSAGE, 0, 1 };
  processMessage(message, 3);

  assertEquals(128, _digitalPortValue);
}

test(processWriteLargestDigitalValue)
{
  Firmata.attach(DIGITAL_MESSAGE, writeToDigitalPort);

  byte message[] = { DIGITAL_MESSAGE, 0x7F, 0x7F };
  processMessage(message, 3);

  // Maximum of 14 bits can be set (B0011111111111111)
  assertEquals(0x3FFF, _digitalPortValue);
}

test(defaultDigitalWritePortIsZero)
{
  Firmata.attach(DIGITAL_MESSAGE, writeToDigitalPort);

  byte message[] = { DIGITAL_MESSAGE, 0, 0 };
  processMessage(message, 3);

  assertEquals(0, _digitalPort);
}

test(specifiedDigitalWritePort)
{
  Firmata.attach(DIGITAL_MESSAGE, writeToDigitalPort);

  byte message[] = { DIGITAL_MESSAGE + 1, 0, 0 };
  processMessage(message, 3);

  assertEquals(1, _digitalPort);
}

[soundanalogous]

I think this is a good start. I've created a new branch called unit-tests and created a test directory and unit sub directory. Please open a pull request against unit-tests. Thanks!

[ntruchsess]

I guess we also need test-stubs of Servo, I2C and OneWire etc... to unit-test the more advanced features?

[soundanalogous]

Yeah we need to merge dev into configurable or at least copy the test code to the configurable branch. Then we can add a unit test for each feature class (at least in theory... some features may be more difficult to test than others if not impossible without writing extra methods just to test, but I don't want to go down that route with a microcontroller).