Processing AQG raw data¶

This part shows basic functionalities of reading and processing AQG data. This workflow is related to the AQGRawData-class. After processing is finished, the raw data is converted to a AQGDataset. Furher usage of this can be found in the documentation on "Handling processed AQG data" (next section).

Reading AQG raw data¶

You can load data from a folder, containing the raw measurement data:

In [1]:

from gravitools.aqg import read_aqg_raw_dataset
from gravitools.corrections import read_eop_data

path_to_raw_dataset = "../data/20240620_163341/"
# please note: reading EOP data explicitly is only necessary for this documentation (due to missing local file system)
read_eop_data('../data/finals2000A.all.csv')
raw = read_aqg_raw_dataset(path_to_raw_dataset)
raw

from gravitools.aqg import read_aqg_raw_dataset from gravitools.corrections import read_eop_data path_to_raw_dataset = "../data/20240620_163341/" # please note: reading EOP data explicitly is only necessary for this documentation (due to missing local file system) read_eop_data('../data/finals2000A.all.csv') raw = read_aqg_raw_dataset(path_to_raw_dataset) raw

Out[1]:

<AQGRawData: 20240620_163341>

Otherwise data can also be loaded directly from a compressed file (of the raw measurement data):

In [2]:

from gravitools.aqg import read_aqg_raw_dataset

path_to_raw_dataset = "../data/20240620_163341.zip"
raw = read_aqg_raw_dataset(path_to_raw_dataset)
raw

from gravitools.aqg import read_aqg_raw_dataset path_to_raw_dataset = "../data/20240620_163341.zip" raw = read_aqg_raw_dataset(path_to_raw_dataset) raw

Out[2]:

<AQGRawData: 20240620_163341>

AQGRawData has an attribute name, which is a unique string that identifies the dataset. By default, its value is infered from the source filename.

In [3]:

raw.name

raw.name

Out[3]:

'20240620_163341'

AQGRawData wraps a pandas.DataFrame, which can also be accessed directly.

In [4]:

raw.df.iloc[0]

raw.df.iloc[0]

Out[4]:

gps_timestamp                                 0.0
gps_state                                       0
ref_time_mode                                   S
g_raw                                  9808372019
dg_tilt                                        -1
dg_pressure                                   -21
dg_earth_tide                                 637
dg_polar                                        0
atmospheric_pressure                       949.56
external_temperature                          NaN
sensor_head_temperature                      37.2
vacuum_chamber_temperature                   48.4
tiltmeter_temperature                        38.1
x_tilt                                    0.00001
y_tilt                                  -0.000001
sat_abs_offset_correction                 -890000
sat_abs_offset_correction_applied               1
raw_quartz_frequency                 99999995.981
cooler_phi                                 -0.076
cooler_eta                                  0.098
rep_phi                                     1.544
rep_eta                                       NaN
atoms_number                                68634
laser_temperature                             NaN
is_outlier                                  False
Name: 2024-06-20 16:33:41.790999889+00:00, dtype: object

Metadata from the .info-file of the AQG raw data is mapped to the AQGRawData.metadata attribute, which is a simple dictionary.

In [5]:

raw.metadata["software"]

raw.metadata["software"]

Out[5]:

'1.7.4'

Perform standard data processing¶

In [6]:

raw.process()

raw.process()

Out[6]:

<AQGRawData: 20240620_163341>

You can also configure this procedure as needed

In [7]:

raw.process(pressure_admittance=-2.9)

raw.process(pressure_admittance=-2.9)

Out[7]:

<AQGRawData: 20240620_163341>

Since the configuration parameters are passed as a dictionary, they can of course also be read from a file format that maps to a dictionary, such as JSON or YAML. You can thus load a configuration file and pass it to the processing command. For how config files are setup and used, please see section Processing configuration.

In [8]:

# load config
import yaml
with open("../data/config_file.yml", encoding="utf-8") as f:
    config = yaml.load(f.read(), Loader=yaml.CLoader)
# processes with loaded configuration parameters
raw.process(**config)

# load config import yaml with open("../data/config_file.yml", encoding="utf-8") as f: config = yaml.load(f.read(), Loader=yaml.CLoader) # processes with loaded configuration parameters raw.process(**config)

Out[8]:

<AQGRawData: 20240620_163341>

Finalize the dataset¶

The class AQGRawData is a wrapper to a pandas.DataFrame. It includes the metadata and provides methods for manipulating the data frame. It is intended for processing the data set. Once this stage of the workflow is complete, convert the raw data to an AQGDataset instance, which represents a finalized processed dataset.

In [9]:

dataset = raw.to_dataset()
dataset

dataset = raw.to_dataset() dataset

Out[9]:

<AQGDataset: 20240620_163341>

Save to file¶

AQGDataset.to_nc() exports the dataset as NETCDF4, a binary data format. The resulting file contains the full signal of all data columns and all relevant metadata. It is also transparently compressed.

In [10]:

dataset.to_nc("../data/test.nc")

dataset.to_nc("../data/test.nc")

Custom data processing¶

You do not have to run AQGRawData.process(). It is merely a convenience method to run a standard sequence of steps and accepts configuration parameters for these.

For instance, you can do the workflow without any processing whatsoever and keep the original corrections.

In [11]:

raw = read_aqg_raw_dataset(path_to_raw_dataset)
dataset = raw.to_dataset()

raw = read_aqg_raw_dataset(path_to_raw_dataset) dataset = raw.to_dataset()

Process an AQG raw dataset from the command line¶

gravitools provides a simple command line interface (CLI) to read an AQG raw dataset, perform standard processing (AQGRawData.process()), and save it as an .nc-file.

The following example reads a zip-archive of AQG raw data and saves it as .\20240620_163341_000.nc.

$ python -m gravitools.aqg -o . 20240620_163341_000.zip

You can pass configuration parameters via a YAML-formatted file.

$ python -m gravitools.aqg -c config_file.yml -o . 20240620_163341_000.zip