Image support in IDataView and transforms.

[Ivanidzo4ka]

Currently only way to work with images is by manually loading pixels arrays into array field in the class and wrap that class into collection data source. Which is not pleasant.

I propose to add support for type Image in DataView type system which can be based on System.Drawing.Bitmap This way user can specify Bitmap image in their class and it will be properly consumed by CollectionDataSource.

Also I suggest to implement transform which would accept column with file paths and load this images during pipeline execution. Which allow you to save memory in case of streaming dataview, some trainers support streaming structure of data, and this way you need to keep in memory only one image for current row instead of images for all rows.

Since quite often you need to modify images (Re-scale them, transform image into Grayscale, etc) it make sense to keep Image format as "image" and not directly convert it into vector of floats which we require for learners. Which means we need transform which would convert Image type into feature vector.

To summarize I propose to create following:

Image type based on Bitmap.
ImageLoader transform
ImageResizer transform
ImageGrayscale transform
ImageToPixels transform

@shauheen @TomFinley @glebuk

[TomFinley]

Hi @Ivanidzo4ka , thanks for looking at this.

So, internally we have two image types. One based on Bitmap, the other based on OpenCV. The first was deprecated in favor of the latter. This was at least partially due to cross-platform requirements (this was before System.Drawing.Bitmap became .NET standard 2.0 I guess), but also due at least partially to the problematic nature of GDI+, or at least .NET Framework's wrapping of it. E.g., there was a GDI+ global process lock. This made System.Drawing.Bitmap perhaps not the best choice when it came to bulk image processing (which is an important usage of this type) since multithreading was useless.

I guess the question is: while I know the cross-platform issue seems to have been resolved, have the issues of perf that led us to originally abandon the first solution we had of System.Drawing.Bitmap been resolved satisfactorily? This may partially be a question for @danmosemsft .

Also all this is surely in preparation for something bigger, right? You don't just have images for the sake of resizing and making them grayscale, we must be plugging them into something specific. Like, evaluations of TensorFlow, CNTK, ONNX, or something like this. I feel like I'd like to know what those things are before commenting on the fitness of this.

[Ivanidzo4ka]

I've run few tests with following pipeline:
TextFile with path to images-> ImageLoadingTransform-> ImageResizeTransform->ImagePixelExtractorTransform.
On top of that pipeline, I've create cursor, and access only image vector column.

Bitmap based - 12 threads - 85k images => 1:37.
OpenCV based - 12 threads - 85k images => 1:30.

Bitmap based - 8 threads - 85k images => 1:51.
OpenCV based - 8 threads - 85k images => 1:42 .

Bitmap based - 6 threads - 10k images => 14 sec.
OpenCV based - 6 threads - 10k images => 15 sec.

Bitmap based - 1 thread - 10k images => 1:05.
OpenCV based - 1 thread - 10k images => 1:15.

On bigger dataset - OpenCV looks better, but not significantly. (Not to mention our current implementation with OpenCV leak memory)

Overall I would prefer to stick to Bitmap, since it supported by dotnet foundation rather that implement custom solution.

[eerhardt]

FYI - @safern and @JeremyKuhne. These perf tests and numbers are pretty interesting. Do you have any interest in doing perf investigation on Bitmap?

[safern]

Looks like a good opportunity to improve Bitmap. @JeremyKuhne and I are planning on taking a look at te Drawing code in general to improve perf because we have a lot of pinvokes that are marshaling unnecessary memory, it would be nice if @Ivanidzo4ka could share this tests with us.

[TomFinley]

Hi all. I agree @Ivanidzo4ka seems like Bitmap is "good enough" for this purpose. We ought to shift it then, thanks for running the comparison.

[Ivanidzo4ka]

@eerhardt , @safern since we talking about System.Drawing.
It appears, I need to have libgdiplus on linux machines in order to run System.Drawing, otherwise I get generic "The type initializer for 'Gdip' threw an exception" exception.

@safern do you have page with requirement for System.Drawing for different OS? (I don't have Mac to properly test it, and my google-fu don't give me good results)
@eerhardt how I can make sure this library installed on building machines? And what's the usual practice for dependencies like this? Add few lines in readme?

[safern]

The Basic requirements across distros is to just have libgdiplus installed. That is what we currently do for corefx testing and the Mac machines already have libgdiplus installed. (which I believe your builds share the same machines as corefx's) if not it can be installed with homebrew or other package manager.

Also, we added support to load libgdiplus from nuget packages: https://github.com/dotnet/corefx/blob/master/src/System.Drawing.Common/src/System/Drawing/GdiplusNative.Unix.cs#L53

So an alternative is to put it in our own testing package, or to have mono (who are the owners of the library) publish an official package containing the latest libgdiplus for every distro.

This is the libgdiplus repo: https://github.com/mono/libgdiplus

I know we have some issues on FreeBSD and Centos: https://github.com/dotnet/corefx/issues/25102

[Ivanidzo4ka]

https://ci2.dot.net/job/dotnet_machinelearning/job/master/job/linux_release_prtest/729/testReport/junit/(root)/(empty)/Microsoft_ML_Tests_ImageTests_TestGreyscaleTransformImages/

Linux Debug Pending — Triggered. (6/21 on rhel72-20171003)
Since it mentions RHEL72 i would assume our Linux build machine is centos or something like this.

MESSAGE:
System.TypeInitializationException : The type initializer for 'Gdip' threw an exception.
---- System.DllNotFoundException : Unable to load DLL 'libgdiplus': The specified module could not be found.
+++++++++++++++++++
STACK TRACE:
at System.Drawing.SafeNativeMethods.Gdip.GdipCreateBitmapFromFile(String filename, IntPtr& bitmap)
at System.Drawing.Bitmap..ctor(String filename, Boolean useIcm)

I'm wouldn't call myself an build expert, so any chance you have example in dotnet repositories where people create workaround for this situation?

[safern]

Does this machines have libgdiplus? In our dotnet repositories we pull the libgdiplus dependency from the corresponding package manager in our Docker files. However we do have some distros/OS that doesn't support libgdiplus like Windows Nano. So for this tests we have a filtering system with a ConditionalFactAttribute we added using the xunit extensible features and we condition this tests to if GDI is available in the running machine.

This is the attribute: https://github.com/dotnet/buildtools/blob/master/src/xunit.netcore.extensions/Attributes/ConditionalFactAttribute.cs

This is how we condition tests for this:
https://github.com/dotnet/corefx/blob/master/src/System.Drawing.Common/tests/GraphicsTests.cs#L16

Where we check if we can load the library:
https://github.com/dotnet/corefx/blob/master/src/System.Drawing.Common/tests/Helpers.cs#L23

[Ivanidzo4ka]

Thank you @safern, but I would like to run test on Linux environment rather than skip it.

@eerhardt any chance you can point me towards documentation how I can install required package to a build machine?