change reference_data.py to use tf.gfile

[robieta]

Reference tests are failing on TAP because they use open(). Due to some quirks with gfile (in py2 it defaults to bytes and in py3 it defaults to string, but it is illegal to pass "rt" or "rb") it is necessary to only interact with files as bytes. This leads to some awkward encode/decode calls with json, but those calls keep the code identical between py2 and py3.

[qlzh727]

I am curious about why read/write to JSON file as binary format. What's the content of JSON file? TF version should just be text.

[robieta]

My contention would be that we should adopt one absolutely safe way to dump JSON, and use it everywhere. Which unfortunately is serializing and then writing to the file. I understand the position that sometimes it's overkill, but that way the code is robust and you never have to think about whether the code will ever see ASCII.

[karmel]

Please add lots of comments explaining why this is necessary to avoid the next helpful engineer from thinking, gee, this could be simpler if we...

[qlzh727]

As chatted offline, I think for now its better to skip the encode/decode for utf8 since most of the data you dump to json are just text data. Using "wb" and "rb" confuses people.

[robieta]

Alright, everything should be good now. The tests were also throwing warnings due to a small change in batch_norm, so I updated them.

[karmel]

"Superficial change in batch_norm" in theory shouldn't require an update here, right? I thought we were going to be robust to that? I worry that we're about to introduce flaky tests.

[robieta]

We are robust to that. If you don't change the reference files it still passes, you just get warnings that the op graph is different than what is expected. So once we confirm that the change is innocuous we can make the change at our convenience.