Add support for array logging and json format

[MaticPogacnik]

Features in this PR are based on the: https://gist.github.com/anjohnson/eee1d8fce75bb406cf2245e2d8cc274e

New backend was written to support logging in a JSON format (requires EPICS base 7.0.1 or newer).
It also supports arrays and lso/lsi records now.

This is work in progress and is not ready to merge. But the code will not change much from this point.
Any feedback would be greatly appreciated.

TODO:

• Document code (doxygen)
• Add user documentation
• Add unit tests.

[anjohnson]

Hi Matic,

I haven't finished going through the code or tried building it yet, but here are some initial comments.

Thanks,

• Andrew

[MaticPogacnik]

Andrew,

Thank you for the feedback, I resolved the comments and added links to commits for easier navigation.

I still have a problem with logging to a PV, if I use a waveform records it works, but with lso record i have some problems.

This doesn't seem to work:

status = dbPutField(this->pCaPutJsonLogPV, DBR_CHAR, msg.c_str(), msg.length());

Should DBR_CHAR be something else?

[anjohnson]

This is how to log to both waveform and long string PVs:

-        status = dbPutField(this->pCaPutJsonLogPV, DBR_CHAR, msg.c_str(), msg.length());
+        if (this->pCaPutJsonLogPV->field_type == DBR_CHAR)
+            status = dbPutField(this->pCaPutJsonLogPV, DBR_CHAR, msg.c_str(), msg.length());
+        else
+            status = dbPutField(this->pCaPutJsonLogPV, DBR_STRING, msg.c_str(), 1);

Even though it's using DBR_STRING the actual string can be longer than 40 characters.

[anjohnson]

This is how to stop the RSET build warnings:

diff --git a/caPutLogApp/Makefile b/caPutLogApp/Makefile
index 3b9f718..b9272f9 100644
--- a/caPutLogApp/Makefile
+++ b/caPutLogApp/Makefile
@@ -5,6 +5,8 @@ include $(TOP)/configure/CONFIG
 
 LIBRARY_IOC = caPutLog
 
+USR_CPPFLAGS += -DUSE_TYPED_RSET
+
 caPutLog_SRCS += caPutLogTask.c
 caPutLog_SRCS += caPutLogAs.c
 caPutLog_SRCS += caPutLogClient.c

[MaticPogacnik]

This is how to log to both waveform and long string PVs:

-        status = dbPutField(this->pCaPutJsonLogPV, DBR_CHAR, msg.c_str(), msg.length());
+        if (this->pCaPutJsonLogPV->field_type == DBR_CHAR)
+            status = dbPutField(this->pCaPutJsonLogPV, DBR_CHAR, msg.c_str(), msg.length());
+        else
+            status = dbPutField(this->pCaPutJsonLogPV, DBR_STRING, msg.c_str(), 1);

Even though it's using DBR_STRING the actual string can be longer than 40 characters.

I tried this and I am getting only first 39 characters (+ terminator) writen to lso record (\ escape characters must be removed to get this number):

I checked with the debugger and this line is getting executed: https://github.com/epics-base/epics-base/blob/951b6acbbc102a5865592f671d2b410e96c5a4d3/modules/database/src/ioc/db/dbFastLinkConv.c#L85 which is also the reason why the string is clipped?

In the units tests I had to use DBR_CHAR to make it work (tho one is over CA and one not):

    chid pchid;
    const char *lsoPv = "lso.$";
    SEVCHK(ca_create_channel(lsoPv, NULL, NULL, 0, &pchid), "ca_create_channel failed");
    SEVCHK(ca_pend_io (CA_PEND_IO_TIMEOUT), "ca_pend_io failed");
    std::string putValue1("lsostring1234567890123456789012345678901234567890123456789");
    SEVCHK(ca_array_put(DBR_CHAR, putValue1.length(), pchid, (void *) putValue1.c_str()), "ca_array_put error");
        SEVCHK(ca_pend_io (CA_PEND_IO_TIMEOUT), "ca_pend_io error");

I took caput client as an example: https://github.com/epics-base/epics-base/blob/951b6acbbc102a5865592f671d2b410e96c5a4d3/modules/ca/src/tools/caput.c#L507
I believe this code is executed if -S flag is specified.

[anjohnson]

I suspect your caget loglso is what is limiting the length of the string you are seeing. Try using caget -S loglso.$ instead, does that clip the string too?

[MaticPogacnik]

I tried it and this is the output:

[devman@anl-develop ~]$ caget -S loglso.$
loglso.$ {\"date\":\"12-08-2020\",\"time\":\"16:57:08\",
[devman@anl-develop ~]$ caget loglso.LEN
loglso.LEN                     40
[devman@anl-develop ~]$ caget loglso.SIZV
loglso.SIZV                    9000

I also checked the LEN field and it confirm that it's only 40 characters long.

[anjohnson]

Hi Matic,

I get these build warnings from clang (on macOS), which should be easily fixed:

woz$ make -sj
../caPutJsonLogTest.cpp:189:18: warning: in-class initialization of non-static
      data member is a C++11 extension [-Wc++11-extensions]
    bool inArray = false;
                 ^
../caPutJsonLogTest.cpp:190:20: warning: in-class initialization of non-static
      data member is a C++11 extension [-Wc++11-extensions]
    int arrayCount = 0;
                   ^
../caPutJsonLogTest.cpp:191:18: warning: in-class initialization of non-static
      data member is a C++11 extension [-Wc++11-extensions]
    bool isArray = false;
                 ^
../caPutJsonLogTest.cpp:193:21: warning: in-class initialization of non-static
      data member is a C++11 extension [-Wc++11-extensions]
    bool waitingKey = true;
                    ^
../caPutJsonLogTest.cpp:210:17: warning: in-class initialization of non-static
      data member is a C++11 extension [-Wc++11-extensions]
    int newSize = -1;
                ^
../caPutJsonLogTest.cpp:211:17: warning: in-class initialization of non-static
      data member is a C++11 extension [-Wc++11-extensions]
    int oldSize = -1;
                ^
../caPutJsonLogTest.cpp:190:9: warning: private field 'arrayCount' is not used
      [-Wunused-private-field]
    int arrayCount = 0;
        ^
../caPutJsonLogTest.cpp:191:10: warning: private field 'isArray' is not used
      [-Wunused-private-field]
    bool isArray = false;
         ^
8 warnings generated.

To build successfully against Base-3.15 I had to add

#include <recSup.h>

to the file caPutLogApp/caPutLogAs.c

caPutLogApp/caPutLogTask.h doesn't need to #include <epicsVersion.h> any more since you're using the Makefile-defined macro instead.

Thank-you for fixing the documentation, that looks much better when viewed on GitHub now. I haven't tried building the docs using sphinx as I don't have it installed, have you attempted that at all?

In test/Makefile the PROD_LIBS need to be listed in the reverse order for the test code to link properly when building a static binary. The library order doesn't matter when linking against shared libraries or DLLs, but for static linking Com should always come last or the link will fail. Actually you don't have to list all of the Base libraries anyway, the build system defines a variable with all the IOC libraries in it, so do this instead:

PROD_LIBS = caPutLog
PROD_LIBS += $(EPICS_BASE_IOC_LIBS)

The tests generally look good, but there do seem to be a lot of 10 second delays in between bursts of checks, are the delays included to avoid updates generating bursts? If they are required that's fine, I don't need the test code to run any faster, I'm just wondering what's going on while the program is paused.

With the above final fixes I think this looks good and mergeable, thanks!

I will see you at the meeting with Guobao on Wednesday.

[MaticPogacnik]

Thank you for review Andrew,

I get these build warnings from clang (on macOS), which should be easily fixed:

woz$ make -sj
../caPutJsonLogTest.cpp:189:18: warning: in-class initialization of non-static
      data member is a C++11 extension [-Wc++11-extensions]
    bool inArray = false;
                 ^
../caPutJsonLogTest.cpp:190:20: warning: in-class initialization of non-static
      data member is a C++11 extension [-Wc++11-extensions]
    int arrayCount = 0;
                   ^
../caPutJsonLogTest.cpp:191:18: warning: in-class initialization of non-static
      data member is a C++11 extension [-Wc++11-extensions]
    bool isArray = false;
                 ^
../caPutJsonLogTest.cpp:193:21: warning: in-class initialization of non-static
      data member is a C++11 extension [-Wc++11-extensions]
    bool waitingKey = true;
                    ^
../caPutJsonLogTest.cpp:210:17: warning: in-class initialization of non-static
      data member is a C++11 extension [-Wc++11-extensions]
    int newSize = -1;
                ^
../caPutJsonLogTest.cpp:211:17: warning: in-class initialization of non-static
      data member is a C++11 extension [-Wc++11-extensions]
    int oldSize = -1;
                ^
../caPutJsonLogTest.cpp:190:9: warning: private field 'arrayCount' is not used
      [-Wunused-private-field]
    int arrayCount = 0;
        ^
../caPutJsonLogTest.cpp:191:10: warning: private field 'isArray' is not used
      [-Wunused-private-field]
    bool isArray = false;
         ^
8 warnings generated.

I fixed this and removed some other c++11 features as well in c9153ac. Code compiles without any warnings with -std=c++98 flag now.

The tests generally look good, but there do seem to be a lot of 10 second delays in between bursts of checks, are the delays included to avoid updates generating bursts? If they are required that's fine, I don't need the test code to run any faster, I'm just wondering what's going on while the program is paused.

Yes, there is a 5 seconds timeout in the logic, to see if there is any more puts coming. Test code makes 1 caput, and then waits for the log to arrive, but the logger will not send one until that 5 seconds timer expires. I added some testDiag to the tests, with this message.

Thank-you for fixing the documentation, that looks much better when viewed on GitHub now. I haven't tried building the docs using sphinx as I don't have it installed, have you attempted that at all?

Yes, I did build the web version (I do not have latex installed for pdf, ...). Here is compressed build docs directory:
docs.tar.gz

Best regards,
Matic