Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add DataView Type Register (supporting loading in-memory image and other in-memory custom types) #3263

Merged
merged 25 commits into from
May 28, 2019
Merged

Add DataView Type Register (supporting loading in-memory image and other in-memory custom types) #3263

merged 25 commits into from
May 28, 2019

Conversation

wschin
Copy link
Member

@wschin wschin commented Apr 9, 2019 •
edited
Loading

Fix #3162 by considering image type in our bridge code. Also fix #3723, fix #3369, fix #3274, fix #445, fix #3460, fix #2121, fix #2495, fix #3784.

@wschin wschin self-assigned this Apr 9, 2019
@wschin wschin added the bug Something isn't working label Apr 9, 2019
@wschin wschin requested review from eerhardt and Ivanidzo4ka and removed request for eerhardt April 9, 2019 21:29
@codecov
Copy link

codecov bot commented Apr 9, 2019 •
edited
Loading

Codecov Report

❗ No coverage uploaded for pull request base (master@51b10fc). Click here to learn what that means.
The diff coverage is 84.84%.

@@            Coverage Diff            @@
##             master    #3263   +/-   ##
=========================================
  Coverage          ?   72.64%           
=========================================
  Files             ?      807           
  Lines             ?   145126           
  Branches          ?    16220           
=========================================
  Hits              ?   105424           
  Misses            ?    35281           
  Partials          ?     4421
Flag Coverage Δ
#Debug 72.64% <84.84%> (?)
#production 68.19% <72.72%> (?)
#test 88.93% <96.96%> (?)
Impacted Files Coverage Δ
src/Microsoft.ML.DataView/ImageDataViewType.cs 70.96% <ø> (ø)
...osoft.ML.Data/DataView/InternalSchemaDefinition.cs 57.53% <100%> (ø)
src/Microsoft.ML.Data/Utils/ApiUtils.cs 91.74% <100%> (ø)
src/Microsoft.ML.Data/Data/SchemaDefinition.cs 70.93% <68%> (ø)
src/Microsoft.ML.Data/DataView/TypedCursor.cs 83.38% <75%> (ø)
test/Microsoft.ML.Tests/ImagesTests.cs 98.6% <96.96%> (ø)

@wschin wschin mentioned this pull request Apr 10, 2019
Closed
TomFinley
TomFinley previously requested changes Apr 10, 2019
View reviewed changes
Copy link
Contributor

@TomFinley TomFinley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @wschin, thanks for working on this!

I do not consider it acceptable that our answer to the problem of what prediction engine does with its reflection based mechanism is that this core assembly has to have a direct, actual dependence on the type -- quite apart from the already excellent objections raised by @Ivanidzo4ka and @eerhardt, this is incompatible with IDataView's own principle as having an extensible type system.

I will post more details on what I'd consider at least a more correct alternative in the existing thread.

@glebuk glebuk mentioned this pull request May 14, 2019
Closed
@wschin wschin changed the title Get back images as C# objects DataView Type Register and Image Type Register May 16, 2019
eerhardt
eerhardt approved these changes May 24, 2019
View reviewed changes
Copy link
Member

@eerhardt eerhardt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@wschin wschin changed the title DataView Type Register and Image Type Register Add DataView Type Register (supporting loading in-memory image and other in-memory custom types) May 24, 2019
@yaeldekel yaeldekel mentioned this pull request May 28, 2019
Closed
@artidoro
Copy link
Contributor

artidoro commented May 28, 2019 •
edited by wschin
Loading

The PR looks great, but we will need a sample for this work. It's fine if it comes in another PR. Could you adapt a test and make it a sample? #Resolved

@artidoro
Copy link
Contributor

artidoro commented May 28, 2019 •
edited by wschin
Loading

Also would it be useful to surface the DataViewTypeManager in MLContext? Or some of it's methods? #Pending

@wschin
Copy link
Member Author

wschin commented May 28, 2019

For this advanced functionality, I'd keep it as is for now. If we add an alias for it, we will have to support it for years.

In reply to: 496640481 [](ancestors = 496640481)

@wschin
Copy link
Member Author

wschin commented May 28, 2019

Added two samples.

In reply to: 496639086 [](ancestors = 496639086)

artidoro
artidoro approved these changes May 28, 2019
View reviewed changes
Copy link
Contributor

@artidoro artidoro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@wschin wschin merged commit 02a857a into dotnet:master May 28, 2019
@wschin wschin deleted the support-image branch May 28, 2019 23:17
@eerhardt eerhardt mentioned this pull request Sep 12, 2019
Merged
@antoniovs1029 antoniovs1029 mentioned this pull request Jul 10, 2020
Merged
@ADD-eNavarro
Copy link

Hi!
Is there a date for this to make it into ML.NET? I really could use this feature, but it seems a little stuck.
Thanks!

@johncalvowtg
Copy link

Hello!
We have actually the same problem here at Wisetech. I am trying to consume a TensorFlow model with multiple multi-dimensional inputs . Same message and no workaround on this.

I've tried some of your examples, but none of them worked for me.

Please let us know if you manage to fully support TensorFlow integration with ML.NET.

Regards,
John Calvo-Martinez Ph.D.

@jamsoft
Copy link

jamsoft commented Mar 7, 2021

Phew, it's not just me going mad. I've been trying to use in memory images all day and have so far failed to get any of this to work. I have SO question open as well in case anyone wants to follow.

@Webreaper
Copy link

@jamsoft Did you remove the SO question? I'm also looking for an ML.Net sample for image classification that takes a bitmap directly rather than a folder of images. Seems sad that so many people are asking for this and MSFT's ML.Net team still haven't put together a useful sample.

@michaelgsharp
Copy link
Member

@Webreaper I thought we had a sample or documentation about it, but I just went and looked and couldn't find anything myself. I'll make sure we have something. The basic idea is that you have 2 pipelines. 1 pipeline loads the images and stops when it has the images loaded as bytes, and the second pipeline starts from where you would use these image bytes already.

I can' upload a zip file here since its not a github issue, but I do have a sample interactive notebook that shows it. If you can create a new issue I can get attach it for you there.

@Webreaper
Copy link

Thanks @michaelgsharp. Hopefully this should be sufficient? #5980

@ghost ghost locked as resolved and limited conversation to collaborators Mar 22, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@ericstj ericstj ericstj left review comments

@artidoro artidoro artidoro approved these changes

@eerhardt eerhardt eerhardt approved these changes

@TomFinley TomFinley Awaiting requested review from TomFinley

@Ivanidzo4ka Ivanidzo4ka Awaiting requested review from Ivanidzo4ka

@zeahmed zeahmed Awaiting requested review from zeahmed

@glebuk glebuk Awaiting requested review from glebuk

@shmoradims shmoradims Awaiting requested review from shmoradims

Assignees

@wschin wschin

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

IDataView Type system needs to be extensible to properly support image, date, and other types Schema comprehension fails for type System.Drawing.Bitmap Create sample/test on ImageLoadingEstimator for TensorFlow scoring for loading an in-memory image stream (instead of files from the drive) ImageLoadingEstimator for TensorFlow scoring should allow in-memory image streams as input in addition to images from files on drive Direct API: RoleMappedSchema/Data Cleanup, Improvement
Example code on how to load in-memory images into ONNX models and TensorFlow models Is there any way to pass pre-loaded image into MLContext pipeline? IDataView Type System needs to be extensible to support Image scenarios properly Bitmap data