nimon AIX7-90: missing timestamps in influx output

The command parameters: nimon -I -s 30 -i <myinflux> -p <myport> **
should connect InfluxDB but not send timestamps.
This is deliberate and by design.
This is perfectly legal line protocol as InfluxDB documents it will add the timestamp itself for the time at which the data was received.
See https://docs.influxdata.com/influxdb/v1/write_protocols/line_protocol_tutorial/
It states </myport></myinflux>"The timestamp is optional in line protocol. If you do not specify a timestamp for your data point InfluxDB uses the server’s local nanosecond timestamp in UTC."
This might be a fraction of a second due to socket delays but this means it uses a consistent time and timezone regardless of the time and time zone of the server running nimon.
This is a good thing.
It seems this feature (optional timestamps) is missing with VictoriaMetrices.

The second parameters:** nimon -I -s 30 -i <myinflux> -p <myport> -m /tmp -ff**
You are requesting the data gets send to InfluxDB AND to a local file with timestamps.
This is an unexpected combination. I will have to check the code to see what is does - that might depend on order of the parameters.
That is never going to work. 
I will add code so nimon will stop with a error message.

Third parameter set: nimon -I -s 30 -i <myinflux> -p <myport> -m /tmp -ff -w</myport></myinflux>
The -w only effects the socket data i.e. being sent to Telegraf or Prometheus by not sending the POST or length details.

**The lack of timestamps is expected with a single -f flag, but my reading of the manual is that there should still be timestamps if file output is disabled completely (no -f flag).**
No idea where you got that idea.
The version 90 documentation states -ff and -f -f are the same thing. The first -f = file output.

Bottom line is nimon working as expected but could be enhanced
A) With this unexpected combination of parameters, stop nimon with an error message.
B) Improve the documentation to make it clearer the njmon/nimon output goes to one of
1. standard out
2. into a file or
3. down a socket.

Plus
C) Get the VictoriaMetrics team to support Line Protocol optional timestamp.

I am interested in VictoriaMetrics as an alternative to InfluxDB but the documentation is rather rough. But a bug report is not the place to discuss it.

Quick and dirty hack psubend() and psectionend()
Change
        if(file_output >= 2) 
            output_char += sprintf(&output[output_char], " %ld000000000\n", (long)time(0));
        else
            output_char += sprintf(&output[output_char], "   \n");

To JUST one line:
    output_char += sprintf(&output[output_char], " %ld000000000\n", (long)time(0));

then recompile the code with the make file.

Thanks for the extensive reply, I'll open a bug report at Victoriametrics with a request to support inferring the timestamp from the current time.

You are requesting the data gets send to InfluxDB AND to a local file with timestamps.
This is an unexpected combination. I will have to check the code to see what is does - that might depend on order of the parameters.
That is never going to work.

I figured this is a somewhat unexpected/hacky combination of parameters. It does "work" though, meaning that adding the "-m /tmp -ff -w" changes what nimon sends to the influx database. Tcpdump confirms that with those parameters set, a timestamp is added to the write request that is otherwise absent. I now successfully receive data and have built some new graphs in Grafana.

Might I suggest to decouple the "add a timestamp" option from the "output to a file" option? Meaning that "-f" enables file output (which should be mutually exclusive with -i as you said), and a new argument (-S for timeStamp?) enables timestamps regardless of whether the output is sent to a file or to a remote server. The "-ff" option could then be removed, or become an alias for "-f -S".

For the paper trail, reported to victoriametrics here: https://github.com/VictoriaMetrics/VictoriaMetrics/issues/10049

According to the folks at victoriametrics it should support writing data without a timestamp. However, nimon appears to be putting two empty spaces after the write string which throws off the timestamp detection and leads to victoriametrics treating the whitespace as the timestamp. Looking at the code I see that this is indeed the case on line 495:

output_char += sprintf(&output[output_char], "   \n");

We thus have one bug/issue and possibly one feature request:
Problem: Extra whitespaces confuse victoriametrics and lead to interpreting whitespace as timestamp, solution would be to remove the whitespace. Would this have side-effects?
Feature request: Separate the timestamp toggle from the file output toggle. This would easily work around the problem.
Plus a bug for victoriametrics:
Problem: Victoriametrics should probably add some condition to check that what it thinks is a timestamp is not in fact an empty string.

Hi Nowa,

Removing the three spaces - rejected:
Influxdata created Line Protocol and implemented it for InfluxDB.
nimon output via a socket works perfectly with InfluxDB and has since 2018.
I found via experiments that InfluxDB requires whitespace after the statistics (as a terminator for the statistics) and that three spaces work best. From my memory, other numbers of spaces were rejected by InfluxDB.
If nimon output tested with InfluxDB is confusing Victoriametrics, then please get Victoriametrics fixed.

In the short term the "hack" above gets you timestamps and avoids working by accident.

Accepted:
I will code extra checks so that nimon stops with unsupported parameter combinations.

New feature request:
I will consider add a new feature for adding timestamps to the socket output and output to a pipe for the next version.
With a new option flag.
But this has a bad side-effect: If source nimon virtual machines are not all in the same time zone or have different times, it can add a lot of confusion when later graphing data. If you take stats from different servers then the peak times may not match at all.
If in a well run professional IT site, all VMs have the same time zone and time - then there is no problem.
I have found that is quite rare.

So this became a Feature Request rather than a bug for njmon.

Implemented in version 91 for AIX and Linux.
A beta is ready for some testing to confirm the feature.

When in nimon mode i.e saving Line Protocol format:
Use the new -S flag to switch on TimeStamps in the output to InfluxDB (or other tools via the same API), a socket or pipe or when saving to a file.

https://sourceforge.net/projects/nmon/files/njmon_linux_91beta.zip/download

https://sourceforge.net/projects/nmon/files/njmon_AIX_91beta.zip/download

Also implemented the NJMON_INFLUX_API_PREFIX shell variable - see other Feature Request.

Awesome, I tested the beta version and found that this works as expected:

NJMON_INFLUX_API_PREFIX="/metrics/short/" /usr/lbin/nimon_beta -I -s 30 -i <my host> -p <my port> -k -n -S

I also sent a pull request to victoriametrics to prevent it from complaining about the extra whitespaces, this is now included in the latest version: https://github.com/VictoriaMetrics/VictoriaMetrics/pull/10096

Thanks!