Mini-language driven parser for structured numerical (or other) data in free text
Current Development Version:
Most Recent Stable Release:
Info:
Do you have structured numerical data stored as text?
Does the idea of writing regex to parse it fill you with loathing?
pent can help!
Say you have data in a text file that looks like this:
$vibrational_frequencies
18
0 0.000000
1 0.000000
2 0.000000
3 0.000000
4 0.000000
5 0.000000
6 194.490162
7 198.587114
8 389.931897
9 402.713910
10 538.244274
11 542.017838
12 548.246738
13 800.613516
14 1203.096114
15 1342.200360
16 1349.543713
17 1885.157022
What's the most efficient way to get that list of floats
extracted into a numpy array?
There's clearly structure here, but how to exploit it?
It would work to import the text into a spreadsheet, split columns appropriately, re-export just the one column to CSV, and import to Python from there, but that's just exhausting drudgery if there are dozens of files involved.
Automating the parsing via a line-by-line string search would work fine
(this is how cclib implements its data imports), but a new line-by-line
method is needed for every new kind of dataset,
and any time the formatting of a given dataset changes.
It's not too hard to write regex that will parse it, but because of the mechanics of regex group captures you have to write two patterns: one to capture the entire block, including the header (to ensure other, similarly-formatted data isn't also captured); and then one to iterate line-by-line over just the data block to extract the individual values. And, of course, one has to actually write (and proofread, and maintain) the regex.
pent provides a better way.
The data above comes from this file,
C2F4_01.hess. With pent, the data can be pulled into numpy in just a couple
of lines, without writing any regex at all:
>>> data = pathlib.Path("pent", "test", "C2F4_01.hess").read_text()
>>> prs = pent.Parser(
... head=("@.$vibrational_frequencies", "#.+i"),
... body=("#.+i #!..f")
... )
>>> arr = np.array(prs.capture_body(data), dtype=float)
>>> print(arr)
[[[ 0. ]
[ 0. ]
[ 0. ]
[ 0. ]
[ 0. ]
[ 0. ]
[ 194.490162]
[ 198.587114]
[ 389.931897]
[ 402.71391 ]
[ 538.244274]
[ 542.017838]
[ 548.246738]
[ 800.613516]
[1203.096114]
[1342.20036 ]
[1349.543713]
[1885.157022]]]The result comes out as a length-one list of 2-D matrices, since the search pattern occurs only once in the data file. The single 2-D matrix is laid out as a column vector, because the data runs down the column in the file.
pent can handle larger, more deeply nested data as well.
Take this 18x18 matrix
within C2F4_01.hess, for example.
Here, it's necessary to pass a Parser as the body of another Parser:
>>> prs_hess = pent.Parser(
... head=("@.$hessian", "#.+i"),
... body=pent.Parser(
... head="#++i",
... body="#.+i #!+.f"
... )
... )
>>> result = prs_hess.capture_body(data)
>>> arr = np.column_stack([np.array(_, dtype=float) for _ in result[0]])
>>> print(arr[:3, :7])
[[ 0.468819 -0.006771 0.020586 -0.38269 0.017874 -0.05449 -0.044552]
[-0.006719 0.022602 -0.016183 0.010997 -0.033397 0.014422 -0.01501 ]
[ 0.020559 -0.016184 0.066859 -0.033601 0.014417 -0.072836 0.045825]]The need for the generator expression, the [0] index into result,
and the composition via np.column_stack arises
due to the manner in which pent returns data from a nested match like this.
See the documentation,
in particular this example,
for more information.
The grammar of the pent mini-language is designed to be flexible enough that
it should handle essentially all well-formed structured data, and even some data
that's not especially well formed. Some datasets will require post-processing of the
data structures generated by pent before they can be pulled into
numpy (see, e.g., this test,
parsing this data block).
Beta releases available on PyPI: pip install pent
Full documentation is hosted at Read The Docs.
Source on GitHub. Bug reports,
feature requests, and Parser construction help requests
are welcomed at the
Issues page there.
Copyright (c) Brian Skinn 2018-2019
License: The MIT License. See LICENSE.txt for full license terms.