JsonToProtoMessage java.time.LocalDateTime error

[Phivan]

1. BigQuery StorageWrite
2. Mac OS 11.5.2
3. Java version: 1.8.0_252
4. bigquerystorage version(s): 2.3.0 and 2.2.1

Steps to reproduce

1. Create a bigquery table with only one field of type DATETIME and with name "d"
2. Use JsonStreamWriter with this JSON Array [{"d":"2021-09-27T20:51:10.752"}] built using LocalDateTime.now()
3. run

Stack trace

java.lang.IllegalArgumentException: JSONObject does not have a int64 field at root.d.
	at com.google.cloud.bigquery.storage.v1beta2.JsonToProtoMessage.fillField(JsonToProtoMessage.java:217)
	at com.google.cloud.bigquery.storage.v1beta2.JsonToProtoMessage.convertJsonToProtoMessageImpl(JsonToProtoMessage.java:94)
	at com.google.cloud.bigquery.storage.v1beta2.JsonToProtoMessage.convertJsonToProtoMessage(JsonToProtoMessage.java:61)
	at com.google.cloud.bigquery.storage.v1beta2.JsonStreamWriter.append(JsonStreamWriter.java:108)
	at com.google.cloud.bigquery.storage.v1beta2.JsonStreamWriter.append(JsonStreamWriter.java:89)
	at app.booking.services.BigQueryServiceV2.$anonfun$writeData$1(BigQueryServiceV2.scala:79)
	at scala.util.Try$.apply(Try.scala:213)
	at app.booking.services.BigQueryServiceV2.writeData(BigQueryServiceV2.scala:67)
	at app.booking.services.BigQueryServiceV2.test(BigQueryServiceV2.scala:53)
	at controllers.tmp.TmpCtrl.$anonfun$testBQV2$1(TmpCtrl.scala:1227)
	at scala.Function1.$anonfun$andThen$1(Function1.scala:57)
	at play.api.mvc.ActionBuilderImpl.invokeBlock(Action.scala:441)
	at play.api.mvc.ActionBuilderImpl.invokeBlock(Action.scala:439)
	at play.api.mvc.ActionBuilder$$anon$9.apply(Action.scala:379)
	at play.api.mvc.Action.$anonfun$apply$4(Action.scala:82)
	at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$4(Accumulator.scala:168)
	at scala.util.Try$.apply(Try.scala:213)
	at play.api.libs.streams.StrictAccumulator.$anonfun$mapFuture$3(Accumulator.scala:168)
	at scala.Function1.$anonfun$andThen$1(Function1.scala:57)
	at play.api.libs.streams.StrictAccumulator.run(Accumulator.scala:199)
	at play.core.server.AkkaHttpServer.$anonfun$runAction$4(AkkaHttpServer.scala:417)
	at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
	at akka.http.scaladsl.util.FastFuture$.$anonfun$transformWith$3(FastFuture.scala:51)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
[error] p.a.h.DefaultHttpErrorHandler - Error while handling error
java.lang.ClassCastException: sbt.internal.inc.MappedVirtualFile cannot be cast to java.io.File
	at play.sbt.run.PlayReload$.$anonfun$sourceMap$1(PlayReload.scala:71)
	at scala.collection.MapLike$MappedValues.get(MapLike.scala:260)
	at play.runsupport.Reloader.$anonfun$findSource$1(Reloader.scala:547)
	at scala.Option.flatMap(Option.scala:271)
	at play.runsupport.Reloader.findSource(Reloader.scala:546)
	at play.core.server.DevServerStart$$anon$1$$anon$2.sourceOf(DevServerStart.scala:171)
	at play.core.SourceMapper.$anonfun$sourceFor$1(ApplicationProvider.scala:21)
	at play.core.SourceMapper.$anonfun$sourceFor$1$adapted(ApplicationProvider.scala:21)
	at scala.collection.IndexedSeqOptimized.$anonfun$find$1(IndexedSeqOptimized.scala:53)
	at scala.collection.IndexedSeqOptimized.$anonfun$find$1$adapted(IndexedSeqOptimized.scala:53)

External references such as API reference guides

https://cloud.google.com/bigquery/docs/write-api

Any additional information below

It seems that JsonToProtoMessage.java:173 is catching the datetime. Because it cannot be cast to Integer or Long, it throw an error : JSONObject does not have a int64 field at root.d.
case INT64: if (val instanceof Integer) { protoMsg.setField(fieldDescriptor, new Long((Integer) val)); return; } else if (val instanceof Long) { protoMsg.setField(fieldDescriptor, (Long) val); return; } break;

Am i doing it wrong ?

Thanks for your help. (and sorry for my English)

[yirutang]

Yes, we have an issue. In the meantime, could you use the CivilTimeEncoder library to convert LocalDateTime into int64?

[yirutang]

Example: json.put("test_time", CivilTimeEncoder.encodePacked64TimeMicros(LocalTime.of(1, 0, 1)));

[yirutang]

For string you can use LocalTime.parse.

[Phivan]

@yirutang It works thank you !

[adaykin]

I'm getting this same error in 2.10.1. The error message when trying to insert an OffsetDateTime is:

Exception: JSONObject does not have a int64 field at root.file_modification_datetime.

Stack:
java.lang.IllegalArgumentException: JSONObject does not have a int64 field at root.file_modification_datetime.
at com.google.cloud.bigquery.storage.v1beta2.JsonToProtoMessage.fillField(JsonToProtoMessage.java:306)
at com.google.cloud.bigquery.storage.v1beta2.JsonToProtoMessage.convertJsonToProtoMessageImpl(JsonToProtoMessage.java:138)
at com.google.cloud.bigquery.storage.v1beta2.JsonToProtoMessage.convertJsonToProtoMessage(JsonToProtoMessage.java:86)
at com.google.cloud.bigquery.storage.v1beta2.JsonStreamWriter.append(JsonStreamWriter.java:110)
at com.google.cloud.bigquery.storage.v1beta2.JsonStreamWriter.append(JsonStreamWriter.java:90)

Can this be reopened?