Allow usage of ConversionService for Jackson HttpMessageConverter [SPR-7054]

[spring-projects-issues]

Ken Egervari opened SPR-7054 and commented

I am having a heck of a time doing complex Ajax stuff with Spring MVC 3.0. While I totally appreciate a lot of the advancements, there are a few features that are just "missing".

The one thing I am having problems with is @RequestBody. It seems that Spring is handling JSON mappings to Strings, ints, longs and other primitive types just fine. Unfortunately... and this is extremely disappointing... Spring does not also use the property editors.

So, in its current state, we can create an object with "long myDomainObjectId" but not "MyDomainObject myDomainObject" and have Spring do the mapping for us. Instead, Spring throws back:

The server refused this request because the request entity is in a format not supported by the requested resource for the requested method ().

In Firebug, it just says, "Unsupported Mime type". And before you say anything, I am sending "application/json" with the charset set to utf8 (as many examples say to do, although I've tried it all kinds of ways). I have used every option conceivable in jQuery - this is a problem with Spring.

So guys... are we supposed to do all this mapping ourselves? What if we are sending json data that is nested 2 or 3 times - do we manually traverse the "long[] ids" arrays and fetch everything ourselves? What if we have to do this for 100 controllers actions? My god, that is insane.

---

Now, I have tried to use GET instead of POST, even though POST is what I want. However, this causes all kinds of other problems with jQuery. For example,

{"activeQuestion":"551","categories":["129","129","128"],"categoryAnswers":["487","489","488"]}

You would think that this would transfer fine... but it doesn't. jQuery creates these GET Request headers:

?activeQuestion=551&
categories[]=129&
categories[]=128&
categoryAnswers[]=487&
categoryAnswers[]=488

As you can see, it doesn't put the indexes for the array notation, and it also chops out a duplicate. The only way to fix this via jQuery is to wrap the numbers in their own objects. Man, talk about bloated! That means we have to create bloated DTO objects that mirror the JSON data... and then convert to the "REAL" domain model afterward.

While this GET approach is slightly better because at least Spring does in fact do use the property editors... we still have to create a bunch of dummy objects that pollute the system for every controller action. Not to mention, we have to use GET for everything - even things that should be POST.

---

I will admit, maybe I don't know enough about Spring 3.0's new features and there is solutions around this stuff... but that only emphasizes the failure of your documentation then... as well as the failure in your "Spring 3.0 Simplications for Ajax" article, which doesn't talk about doing anything complicated.

And I'm just getting started... but it seems all Ajax examples in every framework - JavaScript or otherwise - just show really simple, easy examples. I guess they know their frameworks aren't up to snuff when it comes to something complex - like "a list of numbers".

Guys, why can't we just get this stuff to "Work"? :(

Thanks for listening. I apologize for venting, but I have went from YUI 3.0 -> jQuery... and now because of jQuery's poor API design as well as it's crappy .ajax implementation, I may have to try out Dojo to see if it's any better. YUI 3.0 is bloated beyond belief, it's ajax requests are 10x slower than jQuery's... and ultimately, the new sandbox model is terrible for trying to wrap YUI in an application core framework to make sure it doesn't get exposed to higher levels of the application-specific JavaScript.

It would seem that nobody has gotten this ajax stuff to work seamlessly. Yuck. No wonder I don't want to touch it. Now I have to.

---

Affects: 3.0.2

Issue Links:

• MappingJacksonHttpMessageConverter's ObjectMapper does not use ConversionService when binding JSON to JavaBean properties [SPR-6731] #11397 MappingJacksonHttpMessageConverter's ObjectMapper does not use ConversionService when binding JSON to JavaBean properties ("duplicates")
• Spring dependency injection for Jackson serializers [SPR-10768] #15394 Spring dependency injection for Jackson serializers
• Support jackson mix-in classes in Jackson2ObjectMapperFactoryBean [SPR-12144] #16758 Support jackson mix-in classes in Jackson2ObjectMapperFactoryBean
• Add serializerByType() deserializerByType() and mixIn() to Jackson2ObjectMapperBuilder [SPR-12313] #16918 Add serializerByType() deserializerByType() and mixIn() to Jackson2ObjectMapperBuilder
• Spring MVC Binding Lifecycle differs between @RequestBody arguments and plain JavaBean arguments [SPR-6740] #11406 Spring MVC Binding Lifecycle differs between @RequestBody arguments and plain JavaBean arguments

7 votes, 12 watchers

[spring-projects-issues]

Arjen Poutsma commented

Am I sorry you cannot "get this stuff to work" and I understand your frustration. That said, I am a little unclear as to exactly what you want us to do about it. I do have some background info for you, which might explain why things don't work as you'd like them to.

Conceptually, there is a difference between @RequestBody handling and (form) binding. Binding is where we take string key-value pairs and bind them to bean properties. @RequestBody takes the entire request stream and turn it into an object (using HttpMessageConverters, see http://static.springsource.org/spring/docs/3.0.x/reference/html/remoting.html#rest-message-conversion). As HttpMessageConverters deal with streams (and not strings), using PropertyEditors is not an option.

Spring MVC's JSON support uses Jackson (http://jackson.codehaus.org/) under the hood. You can tweak the JSON support by using annotations (see http://wiki.fasterxml.com/JacksonAnnotations). To see some examples of what Jackson can do, you can take a look at the unit tests (https://fisheye.springframework.org/browse/spring-framework/trunk/org.springframework.web/src/test/java/org/springframework/http/converter/json/MappingJacksonHttpMessageConverterTests.java?r=3141#l43)

[spring-projects-issues]

Ken Egervari commented

Hi Arjen,

Thanks for the response!

First, at the very least, I hope that this message shows what people are trying to do in the field, so that evolutions of the framework... or at least the documentation... can head in a direction that caters to what I think is pretty practical/normal and supports the kinds of use cases that we are doing.

It's nice to have it work for trivial examples... but I get totally scared, frustrated, annoyed, etc. when things start bombing at "a list of numbers". I am thinking, "My god... my app's requirements are like 10x more complex than this... and I am already running into trouble. ARGH!"

There's also the problem that there's just too many ajax frameworks out there... and in my opinion... they all have problems (although I can't speak for Dojo... it's the next on my list to "review").

So I can understand why working with ajax is hard... especially as a dev @ Spring, because I'm sure you check it out with all the top ajax libraries for sanity reasons (I hope so). In the one case about jQuery, I am pretty sure they did something in the latest version of jQuery that enhances the experience under PHP and Rails, but screws over Perl and Java. It turns out - what a shock - that other Java developers were pissed at the devs & jQuery. One can only wonder why they broke working code in 1.3.2 for everyone just to make PHP people happy. Man, I already have so much contempt that I have spent 2-3 days learning about jQuery... maybe for nothing. But I digress.

Now, I can still use jQuery if I use @RequestBody... or I do the simplest of simple things with GET query parameters (no arrays though!).

I guess it would be nice if Spring let us choose if we wanted that Json data to also be treated like form data. I mean, conceptually at a higher level, it's the same thing. With Ajax, we may not even use the form data now... at least we aren't required. In my examples, I am actually reading a whole bunch of <ul>'s and <li>'s from various parts of the page to form a submission to a Spring controller. Conceptually though, it shouldn't be any different than sending stuff through a form.

If we can get that level of abstraction done with Spring's property editors/conversation support, I think it would be a huge win. Man, no custom Jackson coding... AND I can reuse the stuff from before! That would be very nice!

[spring-projects-issues]

Ken Egervari commented

I looked at the Jackson annotations, and while some of them are going to be helpful when I send my domain objects back (the ignore one mainly), this won't help me with my mapping problems whenever I get json from the server.

Arjen, how hard would it be for someone (me, you, whoever) to implement a @Mapping annotation alongside the @RequestBody annotation. Or maybe add a parameter that says, "usePropertyEditors=true".

If I use @RequestBody... and it's looking more and more that I am forced to due to the crappiness of the ajax library world... then I'm sure you can agree that doing the domain object mapping for arrays of arrays of arrays of ids is going to be a pain in the rear... no?

I refuse to code this manually for 100+ controllers. I absolutely refuse. This is just crap code to write.

So would it be possible... even if I am the only person in the world who wants to do this... to create an @Mapping annotation that would make this possible? How hard would it be for me to implement it?

[spring-projects-issues]

Keith Donald commented

I am having a hard time following through the emotion. Before I can assist and arrive at any concrete framework improvements, I need to understand concretely what you're trying to do. Exactly what PropertyEditors do you wish to register, and what should these PropertyEditors do? Can you perhaps attach a small sample app that demonstrates your problem?

[spring-projects-issues]

Ken Egervari commented

Sure Keith ;)

In all of my Spring applications since we could register property editors with spring mvc, I have made a DomainObject property editor that pulls the object out of a database whenever it comes across an id. Here's the code:

public class DomainObjectEditor extends PropertyEditorSupport {

	private BaseDaoSupport dao;
	private SimpleTypeConverter typeConverter = new SimpleTypeConverter();

	public DomainObjectEditor( BaseDaoSupport dao ) {
		this.dao = dao;
	}

	@Override
	public String getAsText() {
		Object obj = getValue();
		if( obj == null ) {
			return null;
		}

		if( obj instanceof DomainObject ) {
			DomainObject domainObject = ( DomainObject ) obj;

			return typeConverter.convertIfNecessary(
				domainObject.getId(), String.class );
		}

		throw new IllegalArgumentException( "Value must be a DomainObject" );
	}

	@Override
	public void setAsText( String text ) {
		if( text == null || 0 == text.length() ) {
			setValue( null );
			return;
		}

		setValue( dao.find( typeConverter.convertIfNecessary( text, Long.class ) ) );
	}

}

As you can see, it's a pretty handle class and I've been using it for a very long time. Generics makes this possible.

To use it, I do stuff like this:

@InitBinder
public void initBinder( WebDataBinder binder ) {
    binder.registerCustomEditor( CategoryAnswer.class,
	    new DomainObjectEditor( categoryAnswerDao ) );
	binder.registerCustomEditor( Category.class,
		new DomainObjectEditor( categoryDao ) );
	binder.registerCustomEditor( ActiveQuestion.class,
		new DomainObjectEditor( activeQuestionDao ) );
}

So, we get good reuse by doing it this way. I can make any string that represents a primary key to the database.

When I use JSON, I also have id's of objects in the database.

So instead of having an object that looks like this:

public class CategoryAnswerSubmission {

	private long activeQuestion;
	private long[] categories;
	private long[] categoryAnswers;
	...
}

It would be really nice to code it like this instead:

public class CategoryAnswerSubmission {

	private ActiveQuestion activeQuestion;
	private List<CategoryAnswer> categoryAnswers;
	private List<Category> categories;
	...
}

Seeing as I can do this with Request query parameters, I don't see why we shouldn't also be able to do this with JSON data as well. Conceptually, we are using JSON in the same way we used to use query parameters or form data. It's totally the same thing.

I hope that clarifies things and separates the emotion.

ARGH!@!@!

(Had to have some emotion in here somewhere though!)

[spring-projects-issues]

Ken Egervari commented

Basically, this would allow me to do this in the controller:

public @ResponseBody ModelMap answerCategory( @RequestBody CategoryAnswerSubmission submission ) {
	myService.myMethod( submission ) );

	ModelMap modelMap = new ModelMap();
	/// generate model and return it
	return model;
}

The idea is that since Spring would do the marshalling (via Jackson, or whatever) as well as do the mapping via property editors, the application developer would only have to code the CategoryAnswerSubmission class in this case... nothing else!

Can you see how nice this would be if we had List of objects that also contained more lists and so on? If we don't have this feature, the developer is FORCED to do the mapping manually. And man, that sucks!!!!

So this is a highly desirable feature ;)

[spring-projects-issues]

Arjen Poutsma commented

Ken,

Just a little follow up on Keith's comment: reading through this issue, and also the other issues you've filed, I think I need to make some things clear.

This is JIRA, our issue tracking system. In it, people file issues they want fixed. An issue can be either:

• a Bug, i.e. "I did A, expected to see B, but got C". Bonus points if you attach a unit tests/project which reproduces the bug. Even more bonus points if you attach a patch that fixes the bug.
• an New Feature, i.e. "I am using A, and it would be really could if you could add B, so that I can do C". Bonus points for a patch.

There are some good guidelines available on the web. See https://developer.mozilla.org/en/Bug_writing_guidelines, for instance.

Now don't be offended, but it is/was rather hard for me and Keith to see where the above issue fits in. The fact that there are non-Spring-related problems (i.e. jQuery) in there certainly doesn't help, as there is nothing we can do about it. Overall, this whole issue seems to be better suited for our community support forums (http://forum.springsource.org/), or - alternatively - you can get commercial support through http://www.springsource.com/services/enterprisesupport.s

Now, I've noticed that you added some additional comments while I was writing this, and they certainly seem to make things more clear. I guess it would have been better if we started out this way.

---

Now, back to the issue at hand: we don't support using PropertyEditors binding for reading JSON, and we probably never will. The reason for this is simple: the web binding mechanism can only deal with Strings Maps, and JSON is a lot more complicated than that. In 3.0, we introduced the new ConversionService which is also capable of converting to other types, but this has not been plugged into our Jackson support (yet).

If you want to customize the Jackson deserialization process, there are some possible options described here. Specifically, you might want to look at JsonDeserializer. For example, you could annotate CategoryAnswerSubmission as follows:

@JsonDeserialize(using=CategoryAnswerSubmissionDeserializer.class)
public class CategoryAnswerSubmission {
	private ActiveQuestion activeQuestion;
	private List<CategoryAnswer> categoryAnswers;
	private List<Category> categories;

}

and then write the following deserializer:

public class CategoryAnswerSubmissionDeserializer extends JsonDeserializer<CategoryAnswerSubmission> {

	@Override
	public CategoryAnswerSubmission deserialize(JsonParser jp, DeserializationContext ctxt)
			throws IOException, JsonProcessingException {
		
		// use Jackson API to read JSON
	}
}

[spring-projects-issues]

Ken Egervari commented

Thanks for response Arjen,

To be fair, while I am using jQuery, this is a Spring related issue.

I see what you mean about the deserializer. Whether it was deserializing the json or fixing the object structure afterward to conform to my domain model, I just didn't want to go down this path. I guess you're saying that we have no choice.

I would HIGHLY recommend the ability to use the new conversation interfaces to plug into Jackson so that the developers can do all sorts of JSON->Domain Model mappings automatically. This would virtually remove almost all of the pain when working with Ajax and HTTP. It would actually feel as if we were developing a thick client... almost. This is certainly an obtainable goal.

In my app, it's going to be so messy to piece together the object graphs of the various JSON data that will be coming from the client. In some cases, there is going to be lists of lists of lists if you know what I mean. Coding and testing these classes is going to take time and it's going to be painfully boring... and just painful.

It just feels like a "small" to me... something that can be improved upon. Something that should be improved upon.

Just think Arjen, with this functionality... you'd probably have the best server-side platform for doing ajax (as far as I know). It would be so much easier to work with complex domain models that Spring would be praised 10x over. It would do wonders for marketing. You could actually say, "The first ajax framework that a Java developers loves and feels at home with." Just imagine Arjen!

As is... the whole process is bloated. My mind cannot comprehend that we are still de-serializing a list of numbers in the year 2010. We are still breaking apart and piecing together primitive data types... like wow. I just can't believe it. I really can't. My mind sees a better way. I can totally visualize it and see it in my mind's eye. I then I go to use this stuff... and I'm like, "Wow... I can't believe how far behind as an industry we are."

That last comment has nothing to do with Spring - just our industry as a whole. It's just amazing, you know?

Anyway, Spring has a golden opportunity to come out on top here by assisting developers to write more natural Java applications that work in the Ajax world without having to commit and base an entire application on a framework like GWT. I think such a feature could really be bragged about in your marketing and really give Spring another push forward.

Take my thoughts for its worth.

[spring-projects-issues]

Ken Egervari commented

A feature that does this mapping automatically also fits in line with Spring's goals - keeping things simple, and only writing code that is part of the application. This deserializing code is clearly framework "to be" code. This is not productive code that advances the application's requirements if you know what I mean.

Telling you... such a feature would be extremely well received.

[spring-projects-issues]

Ken Egervari commented

I just wrote a class that uses the jackson deserializer. Took about an hour or so... and this is one of the simpler cases. The class ended up looking like this:

@Configurable( dependencyCheck = true )
public class CategoryStudentSubmissionDeserializer extends JsonDeserializer<CategoryStudentSubmission> {

	@Autowired
	private ActiveQuestionDao activeQuestionDao;

	@Autowired
	private CategoryDao categoryDao;

	@Autowired
	private CategoryAnswerDao categoryAnswerDao;

	@Override
	public CategoryStudentSubmission deserialize( JsonParser parser, DeserializationContext context ) throws IOException, JsonProcessingException {
		CategoryStudentSubmission submission = new CategoryStudentSubmission();

		while( parser.nextToken() != JsonToken.END_OBJECT ) {
			if( parser.getCurrentToken() == JsonToken.FIELD_NAME ) {
				String fieldName = parser.getCurrentName();
				
				if( fieldName.equals( "activeQuestion" ) ) {
					submission.setActiveQuestion( parseActiveQuestion( parser, submission ) );
				} else if( fieldName.equals( "studentCategoryAnswers" ) ) {
					while( parser.nextToken() != JsonToken.END_ARRAY ) {
						submission.addAnswer( parseStudentCategoryAnswer( parser ) );
					}
				} else {
					throw new IOException("Unrecognized field '" + fieldName + "'" );
				}
			}
		}

		return submission;
	}

	private StudentCategoryAnswer parseStudentCategoryAnswer( JsonParser parser ) throws IOException {
		StudentCategoryAnswer studentCategoryAnswer = new StudentCategoryAnswer();

		while( parser.nextToken() != JsonToken.END_OBJECT ) {
			if( parser.getCurrentToken() == JsonToken.FIELD_NAME ) {
				String fieldName = parser.getCurrentName();

				if( fieldName.equals( "category" ) ) {
					parser.nextToken();
					studentCategoryAnswer.setCategory(
						categoryDao.find( Integer.parseInt( parser.getText() ) )
					);
				} else if( fieldName.equals( "categoryAnswer" ) ) {
					parser.nextToken();
					studentCategoryAnswer.setCategoryAnswer(
						categoryAnswerDao.find( Integer.parseInt( parser.getText() ) ) 
					);
				}
			}
		}

		return studentCategoryAnswer;
	}

	private ActiveQuestion parseActiveQuestion( JsonParser parser, CategoryStudentSubmission studentSubmission ) throws IOException {
		parser.nextToken();
		return activeQuestionDao.find( Integer.parseInt( parser.getText() ) );
	}
}

Maybe I could have used a different method to parse it... I dunno. To me, this seems really bloated.

1. The jackson framework is very strict, so notice all the calls to Integer.parseInt( parser.getText() ). I was unable to use parser.getIntValue() instead... because it will not gracefully parse the string itself. (Wonderful). These kind of details really obfuscate the problem at hand, even though this entire endeavor is bloated to begin with.

In the real world, JSON data is most strings, because jQuery (or some other ajax library) pulls things out of attribute ids, which are also strings. We can't expect "numbers" to really be numbers. This is so similar to form data and http request parameters... it's pretty sick.

2. While the above thing isn't really your concern directly (although because suck a feature is not inside of Spring, we have to code stuff like this), the real problem is getting our Spring beans into the Deserializer. I was lucky that I already had @Configurable working with other beans of mine... but to someone who doesn't have this setup properly... it could be a massive barrier to entry to do this sort of thing. I mean, they could spend hours getting AspectJ working correctly within maven and their IDE. It's not trivial. Man, I can imagine the cursing and swearing that will take place just to get JSON Data mapped.

I think wiring in conversion services and property editors makes so much sense. This kind of code could be slimmed down to just an annotation call... or even less - the same code that we would write for simple, non-mapped JSON data.

I am totally scared to implement deserializers for my other JSON payloads. Very, very scared because they are even more complicated........

[spring-projects-issues]

Arjen Poutsma commented

Ok, we will take a look at using the new ConversionService for JSON in the 3.1 timeline.

[spring-projects-issues]

Kasra Rasaee commented

I will submit my solution soon. Give me a few days to work out the kinks.

[spring-projects-issues]

Deejay commented

Just like to add my voice to support the fixing of this issue.

In short I'd like to be able to submit a JSON request, have the default BeanDeserializer attempt to parse, and in the event of failure fall back to looking up the object's class in a ConversionService. This will save having to litter the domain classes with JsonDeserialize annotations.

I've got a series of implementations of org.springframework.core.convert.converter.Converter that are used for taking Strings from dropdown boxes upon form submission, parsing the value as an Integer and then creating an instance of said object using the parsed Integer as an ID. This works fine when working with normal form binding.

I've changed my Converter implementations to also extend JsonDeserializer. I used the example code that refers anything with Spring Formatting annotations to ConversionService, and altered it to use my Converter/JsonDeserializer implementations. The problem with this is that it always tries to parse the JSON as an Integer ID {1}, even when it's a complex object representation {"complaint":{"claim":"1"},"version":{"expression":"mld","actionTaken":"mld"}}. You can't fall back, as there isn't scope to create a new JsonParser.

I'd copy/paste code, but can't get this thing to format it properly :/

[spring-projects-issues]

Kasra Rasaee commented

I implemented my own JsonSerlializer/Deserializer (tokenizer blah blah) that supports the dot-notation path as well as uses the Editors/Converters to do various types of mapping. I know this is not a "standard json" format, but its much simpler than building an "insane/complex" json graph just to represent your object model.

Example { "person.firstName":"bob", "person.address.street1":"123 Headless Chicken Ave." }

Split by "." and recrusively walk the target object by get/set methods.

In order to register the my custom jsonHttpMessageConverter I did the following

@Component
public class JsonConverterConfigurer implements BeanPostProcessor {
 // bunch of configuration beans... ctor, , etc.

  public Object postProcessAfterInitialization(Object bean, String beanName)
    throws BeansException {

    /**
    * This is more of a hack since Springs
    * org.springframework.transaction.config
    * .AnnotationDrivenBeanDefinitionParser does not allow us to inject a
    * custom HttpMessageConverter. Look at SPR-7091 on jira.springframework.org
    * !!!
    */
    if (bean instanceof AnnotationMethodHandlerAdapter) {
  
      AnnotationMethodHandlerAdapter adapter = (AnnotationMethodHandlerAdapter) bean;
      HttpMessageConverter<?>[] converters = adapter.getMessageConverters();
      List<HttpMessageConverter<?>> cs = new ArrayList<HttpMessageConverter<?>>();

      for (HttpMessageConverter<?> converter : converters) {
        cs.add(converter);
      }

      // add in the custom json http message converter
      cs.add(this.jsonHttpMessageConverter);

      HttpMessageConverter<?>[] css = new HttpMessageConverter<?>[cs.size()];
      css = cs.toArray(css);
      adapter.setMessageConverters(css);
    }
    return bean;
  }

  // converter setter getter, etc etc.
}

[spring-projects-issues]

Kasra Rasaee commented

I know some folks may not necessary like the idea of replacing Jackson, but we've built our own in house JsonSerializer/Deserializer that is incredibly fast as it does the parsing/compiling right when its streaming the data- its easy to use, supports UTF8 :-), uses property editors and the conversion service if available, and dot-notation path binding. Simple example -- { "details.first":"joe", "details.last":"smith" } which can be represented in your domain model, as
MyModel (@RequestBody)
-> get/set Details (type:PersonDetail)
-> get/set First (type:string)
-> get/set Last (type:string)

I would recommend looking at https://code.google.com/u/krasaee

Mapping of generic list objects goes along the lines of

@JsonProperty(MyModel.class)
List<MyModel> getModels() { ... }

void setModels(List<MyModel> models) { .... }

If you need details, I can dig up some code for you.