Skip to content

gravitools.aqg.rawdata

Functions and classes to handle the AQG's raw data files

AQGRawData(df, metadata, name)

Dataframe of AQG rawdata

This is a wrapper class for a pandas.DataFrame and a dictionary of metadata.

Attributes:

  • df (DataFrame) –

    Full raw data

  • metadata (dict) –

    All metadata. Created from the .info file.

  • name (str) –

    Dataset identifier.

  • _modified (bool) –

    True, when columns where modified and the corrected signal has to be recalculated.

  • log_entries (list[str]) –

    List of log messages explaining processing steps taken.

variable_corrections property

List of correction columns in the dataframe

const_corrections property

Constant gravity corrections in the metadata

coordinate property

Location latitude and longitude, in degree

altitude property

Location altitude, in meters

meter property

Instrument identifier

point property

Measurement point identifier

orientation property

Measurement setup orientation

vgg property

Vertical gravity gradient

pressure_admittance property

Atmospheric pressure admittance factor

start_time property

Time when the first data was recorded

tilt_offset property

Instrument tiltmeter offset angles

dg_syst property

Const. systematic bias correction

from_files(info_path, raw_csv_paths, csv_paths, name, source=None) classmethod

Read an AQGRawData instance from rawdata files

Parameters:

  • info_path (str | Path | Path) –

    Path to AQG .info file.

  • raw_csv_paths (list[str | Path | Path]) –

    Paths to AQG _raw.csv files with one row per drop.

  • csv_paths (list[str | Path | Path]) –

    Paths to AQG .csv files of preprocessed data.

  • name (str) –

    Dataset identifier.

  • source (str | Path, default: None ) –

    Data source identifier. If unspecified, info_path is taken.

Returns:

log(text)

Add a message to the processing log

replace_metadata(*, default_orientation=None, orientation=None, pressure_admittance=None, syst_uncertainty=None, vgg=None, tilt_offset=None, **kwargs)

Replace metadata parameters

Parameters:

  • orientation (int, default: None ) –

    Sensor head setup orientation, in degree.

  • default_orientation (int, default: None ) –

    Default sensor head setup orientation, if it is not yet explicitly specified.

  • pressure_admittance (float, default: None ) –

    Atmospheric pressure admittance factor, in nm/s²/hPa.

  • vgg (float | str, default: None ) –

    Gravity gradient, in nm/s²/m.

  • tilt_offset ((tuple, str), default: None ) –

    Instrument tiltmeter offset angles this dataset was recorded with. String is split by commas.

  • **kwargs (dict, default: {} ) –

    Additional metadata parameters. Parameters starting with prefix dg_, such as dg_syst, are interpreted as constant gravity corrections.

process(*, change_tilt_offset=None, recalculate_dg_pressure=True, recalculate_dg_polar=True, detect_outliers=None, outlier_ranges=None, station_height_difference=None, time_period=None, **metadata)

Apply a standard processing procedure to the dataset

Parameters:

  • change_tilt_offset (tuple[float, float], default: None ) –

    Recalculate tilt angles and tilt correction for new tiltmeter offset.

  • recalculate_dg_pressure (bool, default: True ) –

    Whether to call recalculate_dg_pressure().

  • recalculate_dg_polar (bool, default: True ) –

    Whether to call recalculate_dg_polar().

  • detect_outliers (dict | False, default: None ) –

    Parameters for outlier detection. Pass a dictionary of parameters to override defaults of detect_outliers(). Pass None to use default parameters. Set False to deactivate.

  • outlier_ranges (list, default: None ) –

    List of time ranges to mark as outliers. See mark_outlier_ranges().

  • station_height_difference (float, default: None ) –

    Offset to apply to the measurement reference height.

  • time_period (tuple[str | datetime, str | datetime], default: None ) –

    Tuple of start and end timestamps according to which the time series will be cut.

  • **metadata (dict, default: {} ) –

    Parameters to pass on to replace_metadata().

Returns:

mark_outlier_ranges(time_ranges)

Mark a list of time ranges as outliers

Parameters:

  • time_ranges (list[tuple | str]) –

    List of time ranges. Elements can be a tuple of two timestamps, or a string of format START .. END, where START and END are inputs to pd.to_datetime().

limit_time_period(t_from=None, t_to=None)

Limit the data time period to a certain interval

Parameters:

  • t_from (datetime | str, default: None ) –

    Start time ...

  • t_to (datetime | str, default: None ) –

    ... and end time to which the data should be adjusted to. If provided as a string, this has to be in a format that is automatically parsable by standard conversion functions.

recalculate_tilt(tilt_offset)

Recalculate the tilt angles for a new tilt offset

Parameters:

  • tilt_offset (tuple[float, float]) –

    New tilt offset angles, in radian.

In order to reclculate tilt angles with a new tilt offset, the original tilt offset has to be known. This is currently not part of the AQG .info files, but can be found in the global AQG configruation file settings.ini.

Set the original tilt offset manually by specifying it in the metadata:

AQGRawData.metadata["tilt_offset"] = 1.234e-3, 0.123e-3  # radians

set_dg_earth_tide(dg_earth_tide, name)

Replace the earth tide gravity correction time series

The time series will be linearly interpolated to determine a correction for each drop.

Parameters:

  • dg_earth_tide (Series) –

    Time series of gravity corrections.

  • name (str) –

    Tide model identifier name.

set_pressure_admittance(admittance)

Change the admittance factor used for atmospheric pressure correction

Parameters:

  • admittance (float | str) –

    New admittance factor. Pass a float value in units of nm/s²/hPa. Or pass a string value with adequate units, that can be parsed by parse_pressure_admittance().

shift_reference_height(height_offset)

Apply an offset to the measurement reference height

calculate_dg_pressure()

Recalculate the gravity correction due to atmospheric pressure

calculate_dg_polar()

Recalculate the gravity correction due to polar motion

calculate_dg_tilt()

Recalculate the instrument tilt gravity correction

mark_outlier_range(t_start, t_end)

Mark a segment of drops as outliers

set_const_correction(name, value)

Set a constant bias correction

apply_corrections()

Calculate corrected gravity signal by applying all corrections

detect_outliers(g_true=None, g_std=None, threshold=2000, sigma_threshold=5, neighbors=20, min_gap='60s')

Identify and mark outliers in the gravity signal

Drops that deviate from the mean g_true by more than g_std * sigma_threshold are considered outliers.

Parameters:

  • g_true (float, default: None ) –

    Gravity value to take as reference for calculation of outlier deviation. Pass None to have this value calculated automatically.

  • g_std (float, default: None ) –

    Gravity signal standard deviation to use for sigma-criterion. Pass None for automatic calculation.

  • threshold (float, default: 2000 ) –

    Absolute threshold for maximum deviation.

  • sigma_threshold (float, default: 5 ) –

    Multiplicator for standard deviation g_std defining outlier threshold.

  • neighbors (int, default: 20 ) –

    Number of drops neighboring an outlier to also mark as outliers.

  • min_gap (str, default: '60s' ) –

    Minimum time gap between two outliers to be considered as separate events. Specify as pandas.Timedelta argument, e. g. '60s'.

to_dataset()

Create a finalized dataset of processing results

Converts the pandas.DataFrame to an xarray.Dataset with metadata.

Returns:

read_aqg_raw_dataset(path, name=None)

Read an AQG raw dataset from a directory or zip-archive

Parameters:

  • path (str | Path | Path) –

    Path to directory or zip-archive containing AQG raw data.

  • name (str, default: None ) –

    Dataset identifier. If unspecified, the name is infered from the parent directory of the data files.

Returns:

process_aqg_raw_dataset(path, *, name=None, config=None)

Read and process an AQG raw dataset from a directory or zip-archive

Applies standard processing procesdure. See process().

Parameters:

  • path (str | Path | Path) –

    Path to directory or zip-archive containing AQG raw data.

  • name (str, default: None ) –

    Dataset identifier. If unspecified, the name is infered from the parent directory of the data files.

  • config (dict, default: None ) –

    Processing configuration parameters. The relevant parameters are selected using combine_dataset_config.

Returns: