SerializeJson Sucks II

I want to thank those of you who read my rant and added your opinions, questions and corrections (Thanks Nic loosely typed of course :) )

Letting of steam certainly helps and getting feedback helps focus too.

Thanks Kai, I will attempt to clarify my problem with SerializeJSON.

Russ S. I will definitely look at those projects as serializejson for my project is not usable.

The Problem (As I see It)

The problem with CF being loosely typed is that there is no way for me to force a string. If you are working with user generated content, lets say a blog or a cms, and the user decides to name a page as "9999999999999999E+10", this value will be stored as such in the database, because the database field is also a string.

When I read this data out of the database and save it in a structure or variable coldfusion interprets that value as a number but leaves it untouched e.g. doesn't change its format. Returning the structure as JSON the page title is transformed from the previous value to 9.999999999999999E25. As I mentioned in my previous post the two numbers are mathematically identical, the strings however are completely different!

The user is expecting to see "9999999999999999E+10" therefore not expected behavior === bug. Ok, with such a user entry you may be considering that to be an edge case, and I would tend to agree that the likelyhood that a user chooses that exact number as the title for a page is remote, but the problem remains.

Queries are no help

A simple test with CF's built in queries proves that serializeJson ignores the column type and outputs the VarChar as Numeric.


<cfset q = queryNew("title","VarChar")>
<cfset queryAddRow(q,1)>
<cfset querysetcell(q,"title","123456789012345678901",1)>
<cfoutput>#serializeJson(q)#</cfoutput>

{"COLUMNS":["TITLE"],"DATA":[[123456789012345678901]]}

Although the number has not been changed it is still being sent back to the calling JS function as a number. Once JS parses the JSON our original string, which is now a number, is naturally interpreted as one by JS, which results in a value of : 123456789012345680000 !== "123456789012345678901"

The Right way?

As for the right way to handle this with a loosely typed language such as CF it's no easy task, some assumptions have to be made. I would prefer as little assumptions as possible be made about my data. I can live with dates and boolean assumptions (but then CF needs to send back true/false for all functions that return boolean instead of some with yes/no).

But the problem with numbers is when is a number a number and when is it a string? The database knows the data types that are stored. But as we saw above if I serialize a query it seems to ignore the data types.

To prevent the problems with when is a number a number and when is it a string, the only "real" solution, in my opinion, is return numbers as strings.

12 Comments to "SerializeJson Sucks II"- Add Yours
Nathan Mische's Gravatar Gary, you make some very valid points. ColdFusion's implicit type conversion can be a real problem, especially when trying to serialize to JSON. While ColdFusion's SerializeJSON function has definitely improved in the latest CF 9.0.1 hot fix, it can still be problematic for certain data values. To help with these problem data values I created the JSONUtil project (http://jsonutil.riaforge.org/). JSONUtil can look to query meta data to determine the serialization type, so it solves the query issue you list in this post. If you are not serializing queries you can use JavaCast to cast values to specific types and JSONUtil will serialize the value based on the Java type. I encourage you, or anyone else struggling with type conversion in JSON serialization, to take a look at this project. Feedback is always welcome.
# Posted By Nathan Mische | 11/24/10 5:39 AM
Matt Woodward's Gravatar Numbers *can* be returned as numbers; they're just not doing it that way in CF. SerializeJSON works perfectly in OpenBD (not sure about Railo), so I'd just submit all of your complaints as bugs to Adobe and drum up a lot of votes for them. SerializeJSON has been terrible in CF since it was first introduced and desperately needs to be fixed.
# Posted By Matt Woodward | 11/24/10 7:00 AM
Nathan Mische's Gravatar SerializeJSON does not work perfectly in OpenBD. Take the following:

<cfset x = 0010 />
<cfset p = { x = x } />
<cfset s = SerializeJSON(p)/>
<cfoutput>#s#</cfoutput>

This results in '{"x":0010}' which is not valid JSON. Pass that to JavaScript and it will interpret the value as 8 because it will assume octal notation, which CF doesn't support. This is why octal and hexadecimal formats are not supported in the JSON spec.
# Posted By Nathan Mische | 11/24/10 7:20 AM
Gary Gilbert's Gravatar @Nathan,

Thanks I'll take a look at your project again, the problem is that sometimes the structures are pretty complex and it requires remapping the structures to java which is really just too much work :)

@Matt i would be interested to know what your definition of perfect is. Numbers in CF are being returned as numbers. The "problem" is that cfset mystruct.a = 1234 and cfset mystruct.b ="1234" would produce exactly the same results, they are both numbers and would be returned as such by serializejson.

Railo on the other hand treats the above two differently.

In CF the json result would look like:
{"B":1234,"A":1234}
In Railo:
{"B":"1234","A":1234}

Railo also inspects the metadata of a query to determine the correct datatype of the columns and outputs them correctly as strings or numbers. Here an example using a query with column a defined as bigint, b as varchar.
In CF:
{"COLUMNS":["A","B"],"DATA":[[123456789012345,12345678901234]]}
in Railo:
{"COLUMNS":["A","B"],"DATA":[[123456789012345,"12345678901234"]]}

Caveat: Railo's implementation of querynew is somewhat "different" and in my opinion still needs quite a bit of work, additionally implicit conversions of bigints are also a bit buggy :)

How does openBD differ here?
# Posted By Gary Gilbert | 11/24/10 7:51 AM
Matt Woodward's Gravatar OK, guess I should have said it works perfectly in my usage of it.

If you ran into that in your use did you report it to us so we could fix it?
# Posted By Matt Woodward | 11/24/10 7:51 AM
Nathan Mische's Gravatar @Matt, I don't use OpenBD so I have not reported this issue. I just found it hard to believe that any JSON serialization routine can be perfect given CFML's loose typing. Just to confirm this I downloaded and installed OpenBD and the first test I tried failed. So, I think there is room for improvement in every CFML engine.
# Posted By Nathan Mische | 11/24/10 8:17 AM
Matt Woodward's Gravatar In the example you give with 1234 and "1234", OpenBD treats that first instance as a number and the second one as a string, e.g.:
<cfset foo = {a = 1234, b = "1234"} />
<cfset bar = SerializeJSON(foo) />
<cfdump var="#bar#" />

Outputs:
{a:1234,b:"1234"}

The big place you run into issues if it *doesn't* do that is (as you're pointing out) with large numbers, e.g. in CF 8 if you have something like a credit card number (just as an example), and you have that as a string, CF will convert that to a number and you get exponential notation for your credit card number (at least when last I tried it). So at least if SerializeJSON respects the original variable type (i.e. "1234" remains a string) you don't run into this problem.
# Posted By Matt Woodward | 11/24/10 8:18 AM
Nathan Mische's Gravatar @Matt, I have reported this as issue 289: http://code.google.com/p/openbluedragon/issues/det...
# Posted By Nathan Mische | 11/24/10 8:22 AM
Matt Woodward's Gravatar Sure, not saying there isn't room for improvement, but at least as compared to CF 8 OpenBD, again in my usage, does exactly what I expect it to do, whereas CF 8 does a lot of really goofy things that make it rather unusable IMO. Maybe they fixed some of this in CF 9.

I'll file a ticket for the octal issue. We can't fix what we don't know about, and no one had reported that to date.

Big question there is what the expected behavior is. Check for numbers with leading zeros and convert them to strings, or rely on the user to make it a string if they know it's going to cause issues on the JavaScript side of things? I wonder what people would prefer here, having the engine auto-convert hex/octal numbers to strings so they work in JavaScript? Or if hex/octal isn't supported in JSON anyway, what's the "right" thing to do here?
# Posted By Matt Woodward | 11/24/10 8:23 AM
Matt Woodward's Gravatar @Nathan--thanks for the ticket! Greatly appreciated.

Still wonder what's the "right" thing to do in this case ... I can see a couple of different valid arguments here.
# Posted By Matt Woodward | 11/24/10 8:24 AM
Matt Woodward's Gravatar Just as a point of comparison, if you try to enter as a value for a field in CouchDB's web-based tool, it just strips off the leading zeros and makes it 10.

If you try to insert the following JSON into CouchDB via curl:
{"foo":0010}

It throws an invalid JSON error. If you put quotes around the 0010, it works.

So I guess the question remains: what makes logical sense to do here? Auto-convert potential problematic numeric data into strings, or throw an invalid JSON error so people at least know there's invalid JSON?

I agree that leaving it as is and having 0010 get changed to 8 isn't the way to go, but I'm not yet convinced slapping quotes around it automatically is the way to go either. I suppose at least that way the JSON would be valid, but is that what people would want to have happen in these cases?
# Posted By Matt Woodward | 11/24/10 8:43 AM
# Posted By Matt Woodward | 11/24/10 8:58 AM

Powered By Railo

Subscribe

Subscribe via RSS
Follow garyrgilbert on Twitter Follow me on Twitter
Or, Receive daily updates via email.

Tags

adobe air ajax apple cf community cfml coldfusion examples ext flash flex google javascript jquery max2007 max2008 misc open source programming railo software technology ui

Recent Entries

No recent entries.

Blogroll

An Architect's View
CFSilence
Rey Bango
TalkingTree

Wish List

My Amazon.com Wish List