Cannot use `unistd.h`

[bmcdonnell-ionx]

Description

• Type: Bug
• Priority: Major | Minor

I'm trying to use fsync() and fdatasync(), but I can't build when I #include "mbed.h" and #include <unistd.h> in my source file.

---

Bug

Target
LPC4088

Toolchain:
GCC_ARM

Toolchain version:

• arm-none-eabi-c++.exe (GNU Tools for ARM Embedded Processors 6-2017-q2-update) 6.3.1 20170620 (release) [ARM/embedded-6-branch revision 249437]
• arm-none-eabi-gcc.exe (GNU Tools for ARM Embedded Processors 6-2017-q2-update) 6.3.1 20170620 (release) [ARM/embedded-6-branch revision 249437]
• arm-none-eabi-ld.exe GNU ld (GNU Tools for ARM Embedded Processors 6-2017-q2-update) 2.28.0.20170620
• MCUXpresso IDE v10.1.0 [Build 589] [2017-11-14]

mbed-cli version:
1.2.2

mbed-os sha:
5f65721 (tag: mbed_lib_rev157, tag: mbed-os-5.6.6, origin/mbed-os-5.6) Merge pull request #5634 from ARMmbed/release-candidate

Expected behavior

Successful build, and fsync()/fdatasync() flushing to SD Card when I use sd-driver.

Actual behavior

Fails to build. Name collision between

unsigned _EXFUN(sleep, (unsigned int __seconds )); in unistd.h and
__INLINE static void sleep(void) in mbed_sleep.h.

Invoking: MCU C++ Compiler
arm-none-eabi-c++ -std=gnu++98 -DMBED_BUILD_TIMESTAMP=1513190426.2 -DFEATURE_LWIP=1 -D__MBED__=1 -DDEVICE_I2CSLAVE=1 -DTARGET_LIKE_MBED -DTARGET_NXP -DDEVICE_PORTINOUT=1 -DTARGET_RTOS_M4_M7 -DDEVICE_RTC=1 -DTOOLCHAIN_object -D__CMSIS_RTOS -DDEVICE_DEBUG_AWARENESS=1 -DTOOLCHAIN_GCC -DDEVICE_CAN=1 -DTARGET_CORTEX_M -DTARGET_DEBUG -DTARGET_LIKE_CORTEX_M4 -DDEVICE_ANALOGOUT=1 -DTARGET_M4 -DTARGET_UVISOR_UNSUPPORTED -DDEVICE_SERIAL=1 -DDEVICE_INTERRUPTIN=1 -DTARGET_LPCTarget -DTARGET_LPC408X -DTARGET_CORTEX -DDEVICE_I2C=1 -DDEVICE_PORTOUT=1 -D__CORTEX_M4 -DDEVICE_STDIO_MESSAGES=1 -DTARGET_MCU_LPC4088 -D__FPU_PRESENT=1 -DDEVICE_PORTIN=1 -DTARGET_LPC4088 -D__MBED_CMSIS_RTOS_CM -DDEVICE_SLEEP=1 -DTOOLCHAIN_GCC_ARM -DDEVICE_SPI=1 -DDEVICE_ETHERNET=1 -DDEVICE_SPISLAVE=1 -DDEVICE_ANALOGIN=1 -DDEVICE_PWMOUT=1 -DARM_MATH_CM4 -D__NEWLIB__ [snip]
In file included from c:\nxp\mcuxpressoide_10.1.0_589\ide\tools\arm-none-eabi\include\string.h:10:0,
                 from C:/dev/test-fat32/mbed-os/rtos/TARGET_CORTEX/rtx5/rtx_lib.h:31,
                 from C:/dev/test-fat32/mbed-os/rtos/TARGET_CORTEX/mbed_rtos_storage.h:43,
                 from C:/dev/test-fat32/mbed-os/rtos/rtos.h:28,
                 from C:/dev/test-fat32/mbed-os/mbed.h:38,
                 from ../SDCardTest.cpp:16:
c:\nxp\mcuxpressoide_10.1.0_589\ide\tools\arm-none-eabi\include\sys\unistd.h:209:10: error: conflicting declaration of C function 'unsigned int sleep(unsigned int)'
 unsigned _EXFUN(sleep, (unsigned int __seconds ));
          ^
In file included from C:/dev/test-fat32/mbed-os/drivers/PwmOut.h:24:0,
                 from C:/dev/test-fat32/mbed-os/mbed.h:80,
                 from ../SDCardTest.cpp:16:
C:/dev/test-fat32/mbed-os/platform/mbed_sleep.h:131:22: note: previous declaration 'void sleep()'
 __INLINE static void sleep(void)
                      ^~~~~
make: *** [SDCardTest.o] Error 1

Steps to reproduce

In a source file:

#include "mbed.h"
#include <unistd.h>

[pan-]

Even if unistd.h is accessible by GCC's newlib; it is not the case for other compilers like IAR or the ARM compiler because unistd.h is not a header from the C standard library.

It would be impossible to support all features exposed by unistd.h like the exec* functions however it may make sense to provide fsync and fdatasync. @kjbracey-arm @geky What are your thoughts ?

In the meantime you can use fflush that flush buffers out of the libc.

[bmcdonnell-ionx]

Seems I can't use the functions I was trying to anyway. I tried to wrap them as shown below, but the linker complains of
undefined reference to 'fsync' and
undefined reference to 'fdatasync'.

wrap-unistd.h:

namespace workaround
{
   int fsync    (int);
   int fdatasync(int);
}

wrap-unistd.cpp:

#include <unistd.h>

static int wrapped_fsync(int fd)
{
   return fsync(fd);
}

static int wrapped_fdatasync(int fd)
{
   return fdatasync(fd);
}


namespace workaround
{
   int fsync(int fd)
   {
      return wrapped_fsync(fd);
   }

   int fdatasync(int fd)
   {
      return wrapped_fdatasync(fd);
   }
} // namespace workaround

[bmcdonnell-ionx]

Bigger picture: how can I make a function call to ensure that data written to a file gets synced to the storage medium? fflush() doesn't do it; details below.

@pan-

Even if unistd.h is accessible by GCC's newlib; it is not the case for other compilers like IAR or the ARM compiler because unistd.h is not a header from the C standard library.

But mbed OS exposes / utilizes other POSIX functions in the filesystem arena, like open(), close(), read(), etc. Reference: Docs / mbed OS API References / Storage APIs / File system

I don't think "it's not standard C/C++" in and of itself should be the final answer, when the functions in question are POSIX. Are there other reasons not to implement / expose these functions?

It would be impossible to support all features exposed by unistd.h like the exec* functions however it may make sense to provide fsync and fdatasync. @kjbracey-arm @geky What are your thoughts ?

Good points, thanks.

In the meantime you can use fflush that flush buffers out of the libc.

fflush() doesn't ensure that data is written to disk, which is actually the reason I was trying to use fsync().

Per the fflush(3) man page:

Note that fflush() flushes only the user-space buffers provided by the C library. To ensure that the data is physically stored on disk the kernel buffers must be flushed too, for example, with sync(2) or fsync(2).

[bmcdonnell-ionx]

I'm currently working-around the issue by closing and re-opening the file. However, it seems that even this isn't guaranteed to write to disk. (At least in Linux land.)

Notes

Note that fclose() only flushes the user-space buffers provided by the C library. To ensure that the data is physically stored on disk the kernel buffers must be flushed too, for example, with sync(2) or fsync(2).

https://linux.die.net/man/3/fclose

[geky]

I'm currently working-around the issue by closing and re-opening the file. However, it seems that even this isn't guaranteed to write to disk. (At least in Linux land.)

That's pretty interesting. I guess it makes sense when you consider how much of a performance benefit lazily writing to disk is.

Looking at the open group docs, fclose does have this note:

The fclose() function shall perform the equivalent of a close() on the file descriptor that is associated with the stream pointed to by stream.

Which for us, retargets to the FileHandle::close function. At that point it's up to the filesystem. Currently, FATFileSystem does write out to disk, but this is not guaranteed if the implementation changes. For LittleFileSystem, writing out to disk during a close is guaranteed.

TLDR:

• for FATFileSystem fclose == sync - not guaranteed
• for LittleFileSystem fclose == sync - guaranteed

---

But mbed OS exposes / utilizes other POSIX functions in the filesystem arena, like open(), close(), read(), etc. Reference: Docs / mbed OS API References / Storage APIs / File system

The mention of POSIX was a shortsighted mistake on my part. We've since removed it, although the changes are staged and not yet visible. We only guarantee the standard C/C++ file API.

That being said, the standard C/C++ file API is frustrating. It provides some file operations, but as you've noticed it quickly hits walls where functionality is missing. Other examples are the complete lack of directory operations or filesystem stat functions.

To make things messier, the different toolchains we support are each provided different subsets of the POSIX APIs. The examples you mentioned (open, close, read) are all provided by GCC, they won't work if you switch to the IAR or ARM toolchain.

Because of these problems, some of the POSIX APIs were introduced to cover common areas where users were relying on GCC provided functions. Most notably, errno error codes and directory operations. Although, as @pan- points out, some POSIX functions are a bit unreasonable in mbed. So, in the end, we don't have a clear boundary. To avoid issues with backwards compatibility, it's better for mbed to be conservative here.

---

So how do you sync?

It looks like with the current API, you can't. This is a problem.

There are two fixes I can think of:

Option 1:

Provide fileno and fsync (and fdatasync).

This conforms to the expectations of Linux users, but opens the question of where is open, close, and all the other functions that use filenos.

Option 2:

Provide some sort of FileHandle *mbed_get_filehandle(FILE *) function to access the underlying mbed File object. This lets you call File::sync directly.

@kjbracey-arm, I remember you were looking into this at some point. Did you find a way to convert FILE* -> FileHandle*?

IMO, option 2, the mbed_get_filehandle is more maintainable in the long run. Though I'm interested in other's thoughts.

---

As a workaround, @bmcdonnell-ionx have you considered using the C++ mbed file APIs directly? This would give you access to the low-level File::sync function directly:

FATFileSystem fs;
File f;
f.open(&fs, "test.txt", O_RDWR);
f.write("Hello World!", 12);
f.sync();
f.close();

[bmcdonnell-ionx]

@geky

littlefs - interesting. I hadn't heard of that.

But mbed OS exposes / utilizes other POSIX functions in the filesystem arena, like open(), close(), read(), etc. Reference: Docs / mbed OS API References / Storage APIs / File system

The mention of POSIX was a shortsighted mistake on my part. We've since removed it, although the changes are staged and not yet visible. We only guarantee the standard C/C++ file API.

Are you referring to the reference I linked to? It's more than a mention there. Could you clarify? Which parts of the POSIX filesystem API do you (plural) intend for mbed OS to provide? If any.

As a workaround, ...have you considered using the C++ mbed file APIs directly?

I think I'll stick with my current workaround for now, but it's good to have options - thanks.

[geky]

@bmcdonnell-ionx, littlefs is also in the master mbed-os development branch. It should be in mbed-os-5.7 once the release is out.

Are you referring to the reference I linked to? It's more than a mention there. Could you clarify? Which parts of the POSIX filesystem API do you (plural) intend for mbed OS to provide? If any.

The best I can say right now is that we support the standard C99 and C++03 API (anything from here). As well as any existing functions declared in mbed_retarget.h until a major version bump.

We need to put together an official list along with the appropriate documentation, but we don't have that available yet.

[bmcdonnell-ionx]

@geky, if I understand correctly, you're basically saying that "'these' are the subset of POSIX functions we support" (paraphrased). But I'm asking what is the intent?

I've inferred that the intent was to present a complete API through which files, filesystems, directories, etc. can be managed and manipulated. To be as standards-compliant and portable as possible. Thus you support all the C file I/O functions, and add in the POSIX functions where there are gaps (e.g. directory management).

Have I understood correctly? If so, then wouldn't your "option 1" be most consistent with that intent? If not, what am I missing?

[kjbracey]

Hi guys.

Current setup is that we provide the C standard file operations, and some POSIX directory operations. I believe most people haven't seen a pressing need to have the POSIX (int fd) file operations as well as the C (FILE) ones

My PR #5571 would add the standard POSIX file operations, including open,close,fsync. They're a thin layer on top of FileHandle::sync etc.

Up to now we've had FileHandle as the underlying implementation, which is what filesystem/device implementers write, and it can be used directly by apps who like C++.

There's been an internal glue layer on that creating _sys_read(int) etc to support the C library FILE stuff. I think it makes sense to actually expose that layer as a standard-as-possible POSIX read, so we've got the complete stack: fread->_sys_read->read->FileHandle::read.

Even with that, there's unfortunately no way to fsync given only a FILE, as not all the C libraries have fileno, so you can't portably map from FILE to int or FileHandle. Maybe we could be able to come up with something though - at the minute we're living with the limitations of the off-the-shelf libraries. fileno has to be implemented by the C library (not all do), or they have to expose their FILE structure (not all do), or we'd have to figure it out by reverse engineering...

But if you started with a FileHandle or int, and did fdopen on them to get the FILE, so you already know the underlying handle, then you can do the sync/fsync on your underlying FileHandle/int.

This works now:

FATFileSystem fs;
File f;
f.open(&fs, "test.txt", O_RDWR);
FILE *stream = fdopen(&f, "w");
fprintf(stream, "Hello World!");
fflush(stream)
f.sync();
fclose(stream);

or after #5571 this POSIX-portable code would work:

int fd = open("/sd/test.txt", O_RDWR);
FILE *stream = fdopen(fd, "w");
fprintf(stream, "Hello World!");
fflush(stream);
fsync(fd);
fclose(stream);

(Note you have to fflush before fsyncing to empty the C library buffers first).

[kjbracey]

@geky - FileHandle *mbed_get_filehandle(FILE *).

That takes two steps - the FILE knows an int, and we can map an int to a FileHandle (except for stdin/stdout/stderr now, but those would also be FileHandles after #5571).

In #5571, fileno for FILE->int isn't available - we'd need to hack some of the opaque FILE structures to get it :( It does have an internal FileHandle *get_fhc(int) there, which includes the auto-init of the console, but I didn't bother exposing it. It would make sense to, I guess, but without fileno too it would be of somewhat limited usefulness. I guess turning int to FileHandle would let you downcast the FileHandle to something, and there are a couple of FileHandle APIs that #5571 didn't expose as POSIX like set_blocking.

[kjbracey]

@geky

The examples you mentioned (open, close, read) are all provided by GCC

Are they really? They're present in the <unistd.h> header file in the toolchain, but there's no library implementation, I hope. That allows #5571 to provide them.

Only library collision for #5571 I'm aware of is IAR providing fdopen itself - I have to provide it for the other 2. That's a different case though, as that's logically a C library function like fileno, versus open, which is a system function.

[bmcdonnell-ionx]

@kjbracey-arm, cc @geky,

I believe most people haven't seen a pressing need to have the POSIX (int fd) file operations as well as the C (FILE) ones

The only reason I was asking for it now is to have fsync() (and similar) operations. (EDIT: I'm not sure what other motivations there may also be.)

Even with that, there's unfortunately no way to fsync given only a FILE, as not all the C libraries have fileno, so you can't portably map from FILE to int or FileHandle. Maybe we could be able to come up with something though - at the minute we're living with the limitations of the off-the-shelf libraries. fileno has to be implemented by the C library (not all do), or they have to expose their FILE structure (not all do), or we'd have to figure it out by reverse engineering...

None of those sound like maintainable options, IMO.

after #5571 this POSIX-portable code would work:

int fd = open("/sd/test.txt", O_RDWR);
FILE *stream = fdopen(fd, "w");
fprintf(stream, "Hello World!");
fflush(stream);
fsync(fd);
fclose(stream);

(Note you have to fflush before fsyncing to empty the C library buffers first).

That's pretty good!

I have an idea on how you maybe able to get the following more canonical form to work...

// no open()
FILE *stream = fopen("/sd/test.txt", "w");
fprintf(stream, "Hello World!");
fflush(stream);
fsync(fileno(stream)); // use fileno()
// Do some other stuff...
// Maybe the device loses power now, so we never get to fclose(), but the data has already been
//  written to the storage medium.
fclose(stream);

For the libraries that implement fileno(), use their implementations (via #ifdefs.) For those that don't, you could provide the fileno() function. Have a map of FILE* -> fds inside FileSystem, so that when the user calls fileno(), you can look it up. (Of course the map would not be std::map; likely just a pair of arrays.)

Would this do the trick? Is it too much overhead?

If this is doable, I don't know if there would be any value add for also implementing FileHandle *mbed_get_filehandle(FILE *).

[kjbracey]

Have a map of FILE* -> fds inside FileSystem

The problem is the C library never tells the retarget code its FILE. fopen calls our _sys_open, giving us a filename. We translate the filename to a FileHandle and int, and we give the C library back the int. It must store the int in its FILE, but we don't get to find out what FILE our int got put in!

One pretty sick trick would be for fileno(FILE*) to make a dummy ftell(FILE*) call or something, and find out which integer the C library then passes to our _sys_seek. Could be tricky to synchronise, and the call could conceivably have side-effects or be refused for some reason.

Back to the motivation for the fsync - I would note that in most normal applications explicit fsync probably shouldn't be needed. Synchronisation being a good idea is more often a property of the storage medium or the system, rather than a specific file. So I would expect the system to have some policy to automatically ensure flushes to disk at appropriate intervals for normal apps. If people are feeling the need to put fsync in too much, there might be some sort of general policy problem.

I'd suggest it should normally only be explicitly needed in unusual cases like when you are keeping a file open long-term, but making occasional periodic updates.

[bmcdonnell-ionx]

Back to the motivation for the fsync - I would note that in most normal applications explicit fsync probably shouldn't be needed. Synchronisation being a good idea is more often a property of the storage medium or the system, rather than a specific file. So I would expect the system to have some policy to automatically ensure flushes to disk at appropriate intervals for normal apps. If people are feeling the need to put fsync in too much, there might be some sort of general policy problem.

I'd suggest it should normally only be explicitly needed in unusual cases like when you are keeping a file open long-term, but making occasional periodic updates.

On mbed OS 5.6.6, I have a program that was keeping a log file open. It does an fprintf() to the log file every few to several seconds, and calls fflush() after most of them. The file was created, but nothing was ever written to it. (Whereas, I was trying to keep it in sync with other files that were being written.)

Is this a "policy problem" with mbed OS? Has it changed in 5.7.0?

The problem is the C library never tells the retarget code its FILE. fopen calls our _sys_open, giving us a filename. We translate the filename to a FileHandle and int, and we give the C library back the int. It must store the int in its FILE, but we don't get to find out what FILE our int got put in!

One pretty sick trick would be for fileno(FILE*) to make a dummy ftell(FILE*) call or something, and find out which integer the C library then passes to our _sys_seek. Could be tricky to synchronise, and the call could conceivably have side-effects or be refused for some reason.

Yeah, that does sound tricky.

Could you always count on ftell() calling your _sys_seek() in every case? Are there any other lighter-weight functions that call it?