em module is related for a few meter exploration in the case of groundwater exploration. Module provides some basics processing steps for EMAP data filtering and remove noises.

class watex.methods.em.EM(survey_name=None, verbose=0)[source]#

Bases: IsEdi

Create EM object as a collection of EDI-file.

Collect edifiles and create an EM object. It sets the properties from audio-magnetotelluric. The two(2) components XY and YX will be set and calculated.Can read MT data instead, however the full handling transfer function like Tipper and Spectra is not completed. Use other MT softwares for a long periods data.

Parameters:

survey_name (str) – location name where the date where collected . If surveyname is None can chech on edifiles.

ediObjs_#

array of the collection of edifiles read_sucessfully

Type:

Array-like of shape (N,)

data_#

array of all edifiles feed in the EM modules whatever sucessuffuly read or not.

Type:

Array-like of shape (N, )

edinames_#

array of all edi-names sucessfully read

Type:

array-like of shape (N,)

edifiles_#

array of all edifiles if given.

Type:

array of shape (N, )

freqs_#

Array of the frequency range from EDIs

Type:

array-like of shape (N, )

refreq_#

Reference refrequency for data correction. Note the reference frequency is the highest frequency with clean data.

Type:

float,

Properties#
------------
longitude#

longitude coordinate values collected from EDIs

Type:

array-like, shape (N,)

latitude#

Latitude coordinate values collected from EDIs

Type:

array-like, shape (N, )

elevation#

Elevation coordinates collected from EDIs

Type:

array-like, shape (N,)

property elevation#
exportedis(ediObjs, new_Z, savepath=None, **kws)[source]#

Export EDI files from multiples EDI or z objects

Export new EDI file from the former object with a given new impedance tensors. The export is assumed a new output EDI resulting from multiples corrections applications.

Parameters:

  • ediObjs (list of string watex.edi.Edi) – Full path to Edi file/object or object from class:EM objects.

  • new_Z (list of ndarray (nfreq, 2, 2)) – A collection of Ndarray of impedance tensors Z. The tensor Z is 3D array composed of number of frequency nfreq`and four components (``xx`, xy, yx, and yy) in 2X2 matrices. The tensor Z is a complex number.

  • savepath (str, Optional) – Path to save a new EDI file. If None, outputs to _outputEDI_ folder.

Return type:

ediObj from watex.edi.Edi

See also

exportedi

Export single EDI from

fit(data)[source]#

Assert and make EM object from a collection EDIs.

Parameters:

data (str, or list or pycsamt.core.edi.Edi object) – Full path to EDI files or collection of EDI-objects

Returns:

self

Return type:

EM object from a collection EDIs

Examples

>>> from watex.methods.em import EM
>>> emObjs = EM().fit (r'data/edis')
>>> emObjs.ediObjs_
...
static get_z_from(edi_obj_list, /)[source]#

Get z object from Edi object. :param z_or_edis_obj_list: A collection of EDI- or Impedances tensors objects. :type z_or_edis_obj_list: list of watex.edi.Edi or watex.externals.z.Z :param .. versionadded:: v0.1.9:

Returns:

Z – List of impedance tensor Objects.

Return type:

list of watex.externals.z.Z

getfullfrequency(to_log10=False)[source]#

Get the frequency with clean data.

The full or plain frequency is array frequency with no missing data during the data collection. Note that when using Natural Source Audio-Magnetotellurics, some data are missing due to the weak of missing frequency at certain band especially in the attenuation band.

Parameters:

to_log10 (bool, default=False,) – export frequency to base 10 logarithm

Returns:

f – frequency with clean data. Out of attenuation band if survey is completed with Natural Source Audio-Magnetotellurics.

Return type:

Arraylike 1d of shape(N, )

See also

watex.utils.exmath.get_full_frequency

Get the complete frequency with no missing signals.

Example

>>> import watex as wx
>>> edi_sample = wx.fetch_data ('edis', return_data=True, samples = 12 )
>>> wx.EM().fit(edi_sample).getfullfrequency(to_log10 =True )
array([4.76937733, 4.71707639, 4.66477553, 4.61247466, 4.56017382,
       4.50787287, 4.45557204, 4.40327104, 4.35097021, 4.29866928,
       4.24636832, 4.19406761, 4.14176668, 4.08946565, 4.03716465,
       ...
       2.67734228, 2.62504479, 2.57274385, 2.52044423, 2.46814047,
       2.41584107, 2.36353677, 2.31124512, 2.25892448, 2.20663701,
       2.15433266, 2.10202186, 2.04972182, 1.99743007])
getreferencefrequency(to_log10=False)[source]#

Get the reference frequency from collection Edis objects.

The highest frequency with clean data should be selected as the reference frequency

Parameters:

  • data (list of pycsamt.core.edi.Edi or mtpy.core.edi.Edi objects) – Collections of EDI-objects from `pycsamt`_

  • to_log10 (bool,) – outputs the reference frequency into base 10 logarithm in Hz.

Returns:

rf – the reference frequency at the clean data in Hz

Return type:

float

Examples

>>> from watex.methods.em import EM
>>> edipath ='data/3edis'
>>> ref = EM().getreferencefrequency(edipath, to_log10=True)
>>> ref
... 4.845098040014257 # in Hz

References

http://www.zonge.com/legacy/PDF_DatPro/Astatic.pdf

property inspect#

Inspect object whether is fitted or not

is_valid(obj)[source]#

Assert that the given argument is an EDI -object from modules EDI or EDI from pycsamt and MTpy packages. A TypeError will occurs otherwise.

Parameters:

obj (str, pycsamt.core.edi.Edi or mtpy.core.edi.Edi) – Full path EDI file or `pycsamt`_ or `MTpy`_ objects.

Returns:

obj – Identical object after asserting.

Return type:

str, pycsamt.core.edi.Edi or mtpy.core.edi.Edi

property latitude#
property longitude#
make2d(out='resxy', *, kind='complex', **kws)[source]#

Out 2D resistivity, phase-error and tensor matrix from a collection of EDI-objects.

Matrix depends of the number of frequency times number of sites. The function asserts whether all data from all frequencies are available. The missing values should be filled by NaN.

Parameters:

  • data (Path-like object or list of pycsamt.core.edi objects) – Collections of EDI-objects from `pycsamt`_ or full path to EDI files.

  • out (str) – kind of data to output. Be sure to provide the component to retrieve the attribute from the collection object. Except the error and frequency attribute, the missing component to the attribute will raise an error. for instance resxy for xy component. Default is resxy.

  • kind (bool or str) – focuses on the tensor output. Note that the tensor is a complex number of ndarray (nfreq, 2,2 ). If set to``modulus`, the modulus of the complex tensor should be outputted. If real or``imag``, it returns only the specific one. Default is complex.

  • kws (dict) – Additional keywords arguments from :func:`~.getfullfrequency `.

Returns:

mat2d – the matrix of number of frequency and number of Edi-collectes which correspond to the number of the stations/sites.

Return type:

np.ndarray(nfreq, nstations)

Examples

>>> from watex.methods.em import EM
>>> edipath ='data/edis'
>>> emObjs= EM().fit(edipath)
>>> phyx = EM().make2d ('phaseyx')
>>> phyx
... array([[ 26.42546593,  32.71066454,  30.9222746 ],
       [ 44.25990541,  40.77911136,  41.0339148 ],
       ...
       [ 37.66594686,  33.03375863,  35.75420802],
       [         nan,          nan,  44.04498791]])
>>> phyx.shape
... (55, 3)
>>> # get the real number of the yy componet of tensor z
>>> zyy_r = make2d (ediObjs, 'zyx', kind ='real')
... array([[ 4165.6   ,  8665.64  ,  5285.47  ],
       [ 7072.81  , 11663.1   ,  6900.33  ],
       ...
       [   90.7099,   119.505 ,   122.343 ],
       [       nan,        nan,    88.0624]])
>>> # get the resistivity error of component 'xy'
>>> resxy_err = EM.make2d ('resxy_err')
>>> resxy_err
... array([[0.01329037, 0.02942557, 0.0176034 ],
       [0.0335909 , 0.05238863, 0.03111475],
       ...
       [3.33359942, 4.14684926, 4.38562271],
       [       nan,        nan, 4.35605603]])
>>> phyx.shape ,zyy_r.shape, resxy_err.shape
... ((55, 3), (55, 3), (55, 3))
rewrite(*, by='name', prefix=None, dataid=None, savepath=None, how='py', correct_ll=True, make_coords=False, reflong=None, reflat=None, step='1km', edi_prefix=None, export=True, **kws)[source]#

Rewrite Edis, correct station coordinates and dipole length.

Can rename the dataid, customize sites and correct the positioning latitudes and longitudes.

Parameters:

  • dataid (list) – list of ids to rename the existing EDI-dataid from Head.dataid. If given, it should match the length of the collections of ediObjs. A ValueError will occurs if the length of ids provided is out of the range of the number of EDis objects

  • by (str) – Rename according to the inner module Id. Can be name, id, number. Default is name. If survey_name is given, the whole survey name should be overwritten. Conversly, the argument ix outputs the number of formating stations excluding the survey name.

  • prefix (str) – Prefix the number of the site. It could be the abbreviation of the survey area.

  • correct_ll (bool,) – Write the scaled positions( longitude and latitude). Default is True.

  • make_coords (bool) – Useful to hide the real coordinates of the sites by generating a ‘fake’ coordinates for a specific purposes. When setting to True be sure to provide the reflong and reflat values otherwise and error will occurs.

  • reflong (float or string) – Reference longitude in degree decimal or in DD:MM:SS for the site considered as the origin of the lamdmark.

  • reflat (float or string) – Reference latitude in degree decimal or in DD:MM:SS for the reference site considered as the landmark origin.

  • step (float or str) – Offset or the distance of seperation between different sites in meters. If the value is given as string type, except the km, it should be considered as a m value. Only meters and kilometers are accepables. Default value of seperation between the site is 1km.

  • savepath (str) – Full path of the save directory. If not given, EDIs should be outputed in the created directory.

  • how (str) – The way to index the stations. Default is the Python indexing i.e. the counting starts by 0. Any other value will start counting the site from 1.

  • export (bool,) – Export new edi-files

  • kws (dict) – Additionnal keyword arguments from ~Edi.write_edifile and watex.utils.coreutils.makeCoords().

Returns:

EM – Returns self for easy method chaining.

Return type:

EM instance

Examples

>>> from watex.methods.em import EM
>>> edipath = r'data/edis'
>>> savepath =  r'/Users/Daniel/Desktop/ediout'
>>> emObjs = EM().fit(edipath)
>>> emObjs.rewrite_edis(by='id', edi_prefix ='b1',
                        savepath =savepath)
>>> #
>>> # second example to write 7 samples of edi from
>>> # Edi objects inner datasets
>>> #
>>> import watex as wx
>>> edi_sample = wx.fetch_data ('edis', key ='edi',
                                samples =7, return_data =True )
>>> emobj = wx.EM ().fit(edi_sample)
>>> emobj.rewrite(by='station', prefix='PS')
property stnames#
tslicer(freqs=None, z=None, component='xy')[source]#

Returns tensor 2d from components

Parameters:

  • freqs (arraylike) – full frequency that composed the tensor. If None, use the components in

  • Z (ArrayLike 3D) – Tensor is composed of 3D array of shape (n_freqs, 2, 2)

  • component (str,) – components along side to retrieve . Can be [‘xx’|’xy’|’yx’|’yy’]

  • versionadded: (..) – v0.2.0:

Returns:

z or slice – Returns 2D tensor or dictionnary of components index slicers.

Return type:

Arralike 2D tensor, or dict

class watex.methods.em.Processing(window_size=5, component='xy', mode='same', method='slinear', out='srho', c=2, **kws)[source]#

Bases: EM

Base processing of EM object

Fast process EMAP and AMT data. Tools are used for data sanitizing, removing noises and filtering.

Parameters:

  • data (Path-like object or list of :class:watex.edi.Edi` or pycsamt.core.edi.Edi objects) – Collections of EDI-objects

  • freqs (array-like, shape (N)) – Frequency array. It should be the complete frequency used during the survey area. It can be get using the :func:`getfullfrequency ` No need if ediObjs is provided.

  • window_size (int) – the length of the window. Must be greater than 1 and preferably an odd integer number. Default is 5

  • component (str) – field tensors direction. It can be xx, xy,``yx``, yy. If arr2d` is provided, no need to give an argument. It become useful when a collection of EDI-objects is provided. If don’t specify, the resistivity and phase value at component xy should be fetched for correction by default. Change the component value to get the appropriate data for correction. Default is xy.

  • mode (str) – mode of the border trimming. Should be ‘valid’ or ‘same’.’valid’ is used for regular trimimg whereas the ‘same’ is used for appending the first and last value of resistivity. Any other argument except ‘valid’ should be considered as ‘same’ argument. Default is same.

  • method (str, default slinear) – Interpolation technique to use. Can also be nearest. Refer to the documentation of interpolate2d().

  • out (str) – Value to export. Can be sfactor, tensor for corrections factor and impedance tensor. Any other values will export the static corrected resistivity.

  • c (int,) – A window-width expansion factor that must be input to the filter adaptation process to control the roll-off characteristics of the applied Hanning window. It is recommended to select c between 1 and 4. Default is 2.

Examples

>>> import matplotlib.pyplot as plt
>>> from watex.methods.em import Processing
>>> edipath = 'data/edis'
>>> p = Processing().fit(edipath)
>>> p.window_size =2
>>> p.component ='yx'
>>> rc= p.tma()
>>> # get the resistivy value of the third frequency  at all stations
>>> p.res2d_[3, :]
... array([ 447.05423001, 1016.54352954, 1415.90992189,  536.54293994,
       1307.84456036,   65.44806698,   86.66817791,  241.76592273,
       ...
        248.29077039,  247.71452712,   17.03888414])
>>>  # get the resistivity value corrected at the third frequency
>>> rc [3, :]
... array([ 447.05423001,  763.92416768,  929.33837349,  881.49992091,
        404.93382163,  190.58264151,  160.71917654,  163.30034875,
        394.2727092 ,  679.71542811,  953.2796567 , 1212.42883944,
        ...
        164.58282866,   96.60082159,   17.03888414])
>>> plt.semilogy (np.arange (p.res2d_.shape[1] ), p.res2d_[3, :], '--',
                  np.arange (p.res2d_.shape[1] ), rc[3, :], 'ok--')

References

ama(smooth=True, drop_outliers=True, return_phi=False)[source]#

Use an adaptive-moving-average filter to estimate average apparent resistivities at a single static-correction-reference frequency..

The AMA filter estimates static-corrected apparent resistivities at a single reference frequency by calculating a profile of average impedances along the length of the line. Sounding curves are then shifted so that they intersect the averaged profile.

Parameters:

  • smooth (bool, default=True,) – Smooth the tensor data along the frequencies.

  • drop_outliers (bool, default=True) – Suppress outliers in the data when smoothing data along the frequencies axis. Note that drop_outliers does not take effect if smooth is False.

  • return_phi (bool, default=False,) – return corrected phase. Mostly the phase does not need to correct since it is not affected by the static shift effect. However, it can be as smooth phase curve when smooth=True

  • versionadded: (..) – 0.2.1: Polish the tensor data along the frequency axis remove noises and deal with the static shift effect when interferences noises are strong enough.

Returns:

  • rc (np.ndarray, shape (N, M)) – EMAP apparent resistivity static shift corrected or static correction factor or impedance tensor.

  • rc, phi_c (Tuple of shape (N, N)) – EMAP apparent resistivity and phase corrected.

Example

>>> import watex as wx
>>> import matplotlib.pyplot as plt
>>> edi_data = wx.fetch_data ('edis', as_frame =True, key ='edi')
>>> p = wx.EMProcessing (out='z').fit(edi_data.edi)
>>> z_corrected = p.ama () # output z in complex dtype
>>> plt.plot (np.arange (len(p.ediObjs_)) , np.abs(
    [ ediobj.Z.z[:, 0, 1][7]  for ediobj in p.ediObjs_]) , '-ok',
    np.arange(len(p.ediObjs_)), np.abs( z_corrected[7,: ]) , 'or-')

References

[2]

Torres-Verdin and Bostick, 1992, Principles of spatial surface electric field filtering in magnetotellurics: electromagnetic array profiling (EMAP), Geophysics, v57, p603-622.https://doi.org/10.1190/1.2400625

static controlFrequencyBuffer(freq, buffer=None)[source]#

Assert buffer and find the nearest value if the value of the buffer is not in frequency ranges .

Parameters:

  • freq – array-like of frequencies

  • buffer – list of maximum and minimum frequency. It should contains only two values. If None, the max and min frequencies are selected

Returns:

Buffer frequency range

Example:

>>> import numpy as np
>>> from watex.methods.em import Processing
>>> freq_ = np.linspace(7e7, 1e0, 20) # 20 frequencies as reference
>>> buffer = Processing.controlFrequencyBuffer(freq_, buffer =[5.70e7, 2e1])
>>> freq_
... array([7.00000000e+07, 6.63157895e+07, 6.26315791e+07, 5.89473686e+07,
       5.52631581e+07, 5.15789476e+07, 4.78947372e+07, 4.42105267e+07,
       4.05263162e+07, 3.68421057e+07, 3.31578953e+07, 2.94736848e+07,
       2.57894743e+07, 2.21052638e+07, 1.84210534e+07, 1.47368429e+07,
       1.10526324e+07, 7.36842195e+06, 3.68421147e+06, 1.00000000e+00])
>>> buffer
... array([5.52631581e+07, 1.00000000e+00])
drop_frequencies(freqs=None, tol=None, interpolate=False, rotate=0.0, export=False, **kws)[source]#

Drop useless frequencies in the EDI data.

Due to the terrain constraints, topographic and interferences noises some frequencies are not meaningful to be kept in the data. The function allows to explicitely remove the bad frequencies after analyses and interpolated the remains. If bad frequencies are not known which is common in real world, the tolerance parameter tol can be set to automatically detect with 50% smoothness in data selection.

New in version v0.2.0.

Parameters:

  • tol (float,) – the tolerance parameter. The value indicates the rate from which the data can be consider as meaningful. Preferably it should be less than 1 and greater than 0. At this value. If None, the list of frequencies to drop must be provided. If the tol parameter is set to auto, the selection of useless frequencies is tolerate to 50%.

  • freqs (list , Optional) –

    The list of frequencies to remove in the: term:EDI`objects. If ``None`, the tol parameter must be provided, otherwise an error will raise. If the

    return the interpolated frequency if set to True.

  • Interpolate (bool, default=False,) – Interpolate the frequencies after bad frequencies removal.

  • rotate (float, default=0.) –

    Rotate Z array by angle alpha in degrees. All angles are referenced to geographic North, positive in clockwise direction. (Mathematically negative!). In non-rotated state, X refs to North and Y to East direction.

    Note that if rotate is given, it is only used in interpolation i.e interpolation is set to True.

  • export (bool , default =False,) – Output new sanitized EDIs.

Returns:

Zcol – return the quality control value and interpolated frequency if return_freq is set to True otherwise return the index of useless data.

Return type:

or (float, array-like, shape (N, ))

Examples

>>> import watex as wx
>>> sedis = wx.fetch_data ('huayuan', samples = 12 ,
                           return_data =True , key='raw')
>>> p = wx.EMProcessing ().fit(sedis)
>>> ff = [ len(ediobj.Z._freq)  for ediobj in p.ediObjs_]
>>> ff
[53, 52, 53, 55, 54, 55, 56, 51, 51, 53, 55, 53]
>>> p.ediObjs_[0].Z.z[:, 0, 1][:7]
array([ 4165.6 +2070.13j,  7072.81+6892.41j,  8725.84+5874.15j,
       14771.8 -2831.28j, 21243.7 -6802.36j,  6381.48+3411.65j,
        5927.85+5074.27j])
>>> Zcol = p.drop_frequencies (tol =.2 )
>>> Zcol [0].z[:, 0, 1 ][:7]
array([ 4165.6 +2070.13j,  7072.81+6892.41j,  8725.84+5874.15j,
       14771.8 -2831.28j, 21243.7 -6802.36j,  6381.48+3411.65j,
        5927.85+5074.27j])
>>> [ len(z.freq) for z in Zcol ]
[53, 52, 52, 53, 53, 53, 53, 50, 49, 53, 53, 52]
>>> p.verbose =True
>>> Zcol = p.drop_frequencies (tol =.2 , interpolate= True )
Frequencies:     1- 81920.0    2- 48.5294    3- 5.625  Hz have been dropped.
>>> [ len(z.freq) for z in Zcol ] # all are interpolated to 53 frequencies
[53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53, 53]
>>> Zcol = p.drop_frequencies (tol =.2 , interpolate= True , export =True )
>>> # drop a specific frequencies
>>> # let visualize the 7 frequencies of our ediObjs
>>> p.freqs_ [:7]
array([81920., 70000., 58800., 49500., 41600., 35000., 29400.])
>>> # let try to drop 49500 and 29400 frequencies explicitly.
>>> Zcol = p.drop_frequencies (freqs = [49500 , 29400] )
>>> # let check whether this frequencies still available in the data
>>> Zcol [5].freq[:7]
array([81920., 70000., 58800., 41600., 35000., 24700., 20800.])
>>> # frequencies do not need to match exactly the value in frequeny
>>> # range. Here is an example
>>> Zcol = p.drop_frequencies (freqs = [49800 , 29700] )
Frequencies:     1- 49500.0    2- 29400.0  Hz have been dropped.
>>> # explicitly it drops the 49500 and 29400 Hz the closest.
flma(smooth=True, drop_outliers=True, return_phi=False)[source]#

A fixed-length-moving-average filter to estimate average apparent resistivities at a single static-correction-reference frequency.

The FLMA filter estimates static-corrected apparent resistivities at a single reference frequency by calculating a profile of average impedances along the length of the line. Sounding curves are then shifted so that they intersect the averaged profile.

Parameters:

  • smooth (bool, default=True,) – Smooth the tensor data along the frequencies.

  • drop_outliers (bool, default=True) – Suppress outliers in the data when smoothing data along the frequencies axis. Note that drop_outliers does not take effect if smooth is False.

  • return_phi (bool, default=False,) – return corrected phase in degrees. Mostly the phase does not need to correct since it is not affected by the static shift effect. However, it can be as smooth phase curve when smooth=True

  • versionadded: (..) – 0.2.1: Polish the tensor data along the frequency axis remove noises and deal with the static shift effect when interferences noises are strong enough.

Returns:

  • rc (np.ndarray, shape (N, M)) – EMAP apparent resistivity static shift corrected or static correction factor or impedance tensor.

  • rc, phi_c (Tuple of shape (N, N)) – EMAP apparent resistivity and phase corrected.

Example

>>> import watex as wx
>>> import matplotlib.pyplot as plt
>>> edi_data = wx.fetch_data ('edis', as_frame =True, key ='edi')
>>> p = wx.EMProcessing (out='z').fit(edi_data.edi)
>>> z_corrected = p.flma () # output z in complex dtype
>>> plt.plot (np.arange (len(p.ediObjs_)) , np.abs(
    [ ediobj.Z.z[:, 0, 1][7]  for ediobj in p.ediObjs_]) , '-ok',
    np.arange(len(p.ediObjs_)), np.abs( z_corrected[7,: ]) , 'or-')

References

static freqInterpolation(y, /, buffer=None, kind='freq')[source]#

Interpolate frequency in frequeny buffer range.

Parameters:

  • y – array-like, shape(N, ) - Can be a frequency array or periods note that the frequency is not in log10 Hz.

  • buffer – list of maximum and minimum frequency. It should contains only two values. If None, the max and min frequencies are used

  • kind – str type of given data. Can be ‘period’ if the value is given as periods or ‘frequency’ otherwise. Any other value should be considered as a frequency values.

Returns:

array_like, shape (N2, ) New interpolated frequency with N2 size

Example:
>>> from watex.methods.em import Processing
>>> pobj = Processing().fit('data/edis')
>>> f = getfullfrequency (pobj.ediObjs_)
>>> buffer = [5.86000e+04, 1.6300e+01]
>>> f
... array([7.00000e+04, 5.88000e+04, 4.95000e+04, 4.16000e+04, 3.50000e+04,
       2.94000e+04, 2.47000e+04, 2.08000e+04, 1.75000e+04, 1.47000e+04,
       ...
       2.75000e+01, 2.25000e+01, 1.87500e+01, 1.62500e+01, 1.37500e+01,
       1.12500e+01, 9.37500e+00, 8.12500e+00, 6.87500e+00, 5.62500e+00])
>>> new_f = freqInterpolation(f, buffer = buffer)
>>> new_f
... array([5.88000000e+04, 4.93928459e+04, 4.14907012e+04, 3.48527859e+04,
       2.92768416e+04, 2.45929681e+04, 2.06584471e+04, 1.73533927e+04,
       ...
       2.74153120e+01, 2.30292565e+01, 1.93449068e+01, 1.62500000e+01])
getValidTensors(tol=0.5, **kws)[source]#

Select valid tensors from tolerance threshold and write EDI if applicable.

Function analyzes the data and keep the good ones. The goodness of the data depends on the threshold rate. For instance 50% means to consider an impedance tensor ‘z’ valid if the quality control shows at least that score at each frequency of all stations.

Parameters:

  • data (Path-like object or list of pycsamt.core.edi.Edi) – collections of EDI-objects from `pycsamt`_ . data params is passed to fit() method.

  • tol (float,) – tolerance parameter. The value indicates the rate from which the data can be consider as a valid. The valid data selection should be soft when the tolerance parameter is close to ‘1’ and hard otherwise. As the tol value decreases, the selection becomes severe. Default is .5 means 50 %

  • kws (dict ,) – Additional keywords arguments for EDI file exporting

Returns:

Zc

Return type:

watex.externals.z.Z impedance tensor objects.

Examples

>>> from watex.methods.em import Processing
>>> pObj = Processing ().fit('data/edis')
>>> f= pObj.freqs_
>>> len(f)
... 55
>>> zObjs_hard = pObj.getValidTensors (tol= 0.3 ) # None doesn't export EDI-file
>>> len(zObjs_hard[0]._freq) # suppress 3 tensor data
... 52
>>> zObjs_soft  = pObj.getValidTensors(p.ediObjs_, tol = 0.6 , option ='write')
>>> len(zObjs_soft[0]._freq)  # suppress only two
... 53
static interpolate_z(z_or_edis_obj_list, /, **kws)[source]#

Interpolate z and return new interpolated z objects

Interpolated frequencies is useful to have all frequencies into the same scale.

New in version 0.2.0.

Parameters:

  • z_or_edis_obj_list (list of watex.edi.Edi or watex.externals.z.Z) – A collection of EDI- or Impedances tensors objects.

  • kws (dict,) – Additional keywords to export EDI or rotate EDI/Z. - "option": export EDI if set to write. - rotate : float, a rotate angle for Z if value is given.

Returns:

Z – List interpolated impedance tensor Objects or None if option is set to write.

Return type:

list of watex.externals.z.Z

Examples

>>> import watex as wx
>>> sedis = wx.fetch_data ('huayuan', samples = 12 ,
                                 return_data =True , key='raw')
>>> p = wx.EMProcessing ().fit(sedis)
>>> ff = [ len(ediobj.Z._freq)  for ediobj in p.ediObjs_]
[53, 52, 53, 55, 54, 55, 56, 51, 51, 53, 55, 53]
>>> Zcol = p.interpolate_z (sedis)
>>> ffi = [ len(z.freq) for z in Zcol ]
[56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56]
>>> # visualize seven Z values at the first site component xy
>>> p.ediObjs_[0].Z.z[:, 0, 1][:7]
array([ 4165.6 +2070.13j,  7072.81+6892.41j,  8725.84+5874.15j,
       14771.8 -2831.28j, 21243.7 -6802.36j,  6381.48+3411.65j,
        5927.85+5074.27j])
>>> Zcol [0].z[:, 0, 1 ][:7]
array([ 4165.6 +2070.13j,  4165.6 +2070.13j,  7072.81+6892.41j,
        8725.84+5874.15j, 14771.8 -2831.28j, 21243.7 -6802.36j,
        6381.48+3411.65j])
qc(tol=0.5, *, return_freq=False, return_ratio=False, to_log10=True)[source]#

Check the quality control of the collected EDIs.

Analyse the data in the EDI collection and return the quality control value. It indicates how percentage are the data to be representative.

Parameters:

  • tol (float,) – the tolerance parameter. The value indicates the rate from which the data can be consider as meaningful. Preferably it should be less than 1 and greater than 0. At this value. Default is .5 means 50 %

  • return_freq (bool, Default =False) – return the interpolated frequency if set to True.

  • return_ratio (bool, default=False,) –

    return only the ratio of the representation of the data.

    New in version 0.1.5.

  • to_log10 (bool, default=False) – convert the interpolated frequency into a log10.

Returns:

return the quality control value and interpolated frequency if return_freq is set to True otherwise return the index of useless data.

Return type:

Tuple (float , index ) or (float, array-like, shape (N, ))

Examples

>>> from watex.methods.em import Processing
>>> pobj = Processing().fit('data/edis')
>>> f = pobj.getfullfrequency ()
>>> # len(f)
>>> # ... 55 # 55 frequencies
>>> c,_ = pobj.qc ( tol = .4 ) # mean 60% to consider the data as
>>> # representatives
>>> c  # the representative rate in the whole EDI- collection
>>> # ... 0.95 # the whole data at all stations is safe to 95%.
>>> # now check the interpolated frequency
>>> c, freq_new  = pobj.qc ( tol=.6 , return_freq =True)
skew(method='swift', return_skewness=False, suppress_outliers=False)[source]#

The conventional asymmetry parameter based on the Z magnitude.

The EM signal is influenced by several factors such as the dimensionality of the propagation medium and the physical anomalies, which can distort the EM field both locally and regionally. The distortion of Z was determined from the quantification of its asymmetry and the deviation from the conditions that define its dimensionality. The parameters used for this purpose are all rotational invariant because the Z components involved in its definition are independent of the orientation system used. The conventional asymmetry parameter based on the Z magnitude is the skew defined by Swift (1967) as follows:

\[skew_{swift}= |\frac{Z_{xx} + Z_{yy}}{ Z_{xy} - Z_{yx}}|\]

When the \(skew_{swift}\) is close to 0., we assume a 1D or 2D model when the \(skew_{swift}\) is greater than >=0.2, we assume 3D local anomaly (Bahr, 1991; Reddy et al., 1977). It is generally considered that an electrical structure of \(skew < 0.4\) can be treated as a 2D medium.

Furthermore, Bahr (1988) proposed the phase sensitive skew which calculates the skew taking into account the distortions produced in Z over 2D structures by shallow conductive anomalies and is defined as follows:

\[ \begin{align}\begin{aligned}skew_{Bahr} & = & \sqrt{ \frac{|[D_1, S_2] -[S_1, D_2]|}{|D_2|}} \quad \text{where}\\S_1 & = & Z_{xx} + Z_{yy} \quad ; \quad S_2 = Z_{xy} + Z_{yx}\\D_1 & = & Z_{xx} - Z_{yy} \quad ; \quad D_2 = Z_{xy} - Z_{yx}\end{aligned}\end{align} \]

Note that The phase differences between two complex numbers \(C_1\) and \(C_2\) and the corresponding amplitude products are now abbreviated by the commutators:

\[ \begin{align}\begin{aligned}\[C_1, C_2] & = & \text{Im} C_2*C_1^*\\\[C_1, C_2] & = & \text{Re} C_1 * \text{Im}C_2 - R_e(C_2)* \text{Im}C_1\end{aligned}\end{align} \]

Indeed, \(skew_{Bahr}\) measures the deviation from the symmetry condition through the phase differences between each pair of tensor elements,considering that phases are less sensitive to surface distortions(i.e. galvanic distortion). The \(skew_{Bahr}\) threshold is set at 0.3 and higher values mean 3D structures (Bahr, 1991).

Parameters:

  • data (str of path-like or list of pycsamt.core.edi.Edi) – EDI data or EDI object with full impedance tensor Z.

  • method (str) – Kind of correction. Can be swift for the remove distorsion proposed by Swift in 1967. The value close to 0. assume the 1D and 2D structures and 3D otherwise. Conversly to bahr for the remove distorsion proposed by Bahr in 1991. The latter threshold is set to 0.3. Above this value the structures is 3D.

  • return_skewness (str,) – Typically returns the type of skewness. 'skew' or mu for skew and rotation- all invariant values respectively. Any other value return both skew and rotational invariant.

  • suppress_outliers (bool, default=False,) –

    Remove the outliers (if applicable in the data ) before normalizing.

    New in version 0.1.6.

Returns:

skw, mu –

  • Array of skew at each frequency

  • rotational invariant mu at each frequency that measures of phase differences in the impedance tensor.

Return type:

Tuple of ndarray-like , shape (N, M )

See also

watex.utils.plot_skew

For phase sensistive skew visualization - naive plot.

watex.view.TPlot.plotSkew

For consistent plot of phase sensitive skew visualization. Allow customize plots.

watex.view.TPlot.plot_phase_tensors

Plot skew as ellipsis visualization by turning the tensor parameter to skew.

Examples

>>> from watex.methods.em import Processing
>>> edipath = 'data/edis'
>>> p = Processing().fit(edipath)
>>> sk,_ = p.skew()
>>> sk[0:, ]
... array([0.45475527, 0.7876896 , 0.44986397])

References

Bahr, K., 1991. Geological noise in magnetotelluric data: a classification

of distortion types. Physics of the Earth and Planetary Interiors 66 (1–2), 24–38.

Barcelona, H., Favetto, A., Peri, V.G., Pomposiello, C., Ungarelli, C., 2013.

The potential of audiomagnetotellurics in the study of geothermal fields: A case study from the northern segment of the La Candelaria Range, northwestern Argentina. J. Appl. Geophys. 88, 83–93. https://doi.org/10.1016/j.jappgeo.2012.10.004

Swift, C., 1967. A magnetotelluric investigation of an electrical conductivity

anomaly in the southwestern United States. Ph.D. Thesis, MIT Press. Cambridge.

tma(smooth=True, drop_outliers=True, return_phi=False)[source]#

A trimmed-moving-average filter to estimate average apparent resistivities at a single static-correction-reference frequency.

The TMA filter option estimates near-surface resistivities by averaging apparent resistivities along line at the selected static-correction reference frequency. The highest frequency with clean data should be selected as the reference frequency.

Parameters:

  • smooth (bool, default=True,) – Smooth the tensor data along the frequencies.

  • drop_outliers (bool, default=True) – Suppress outliers in the data when smoothing data along the frequencies axis. Note that drop_outliers does not take effect if smooth is False.

  • return_phi (bool, default=False,) – return corrected phase in degrees. Mostly the phase does not need to correct since it is not affected by the static shift effect. However, it can be as smooth phase curve when smooth=True

  • versionadded: (..) – 0.2.1: Polish the tensor data along the frequency axis remove noises and deal with the static shift effect when interferences noises are strong enough.

Returns:

  • rc (np.ndarray, shape (N, M)) – EMAP apparent resistivity static shift corrected or static correction factor or impedance tensor.

  • rc, phi_c (Tuple of shape (N, N)) – EMAP apparent resistivity and phase corrected.

Example

>>> import watex as wx
>>> import matplotlib.pyplot as plt
>>> edi_data = wx.fetch_data ('edis', as_frame =True, key ='edi')
>>> p = wx.EMProcessing (out='z').fit(edi_data.edi)
>>> z_corrected = p.tma () # output z in complex dtype
>>> plt.plot (np.arange (len(p.ediObjs_)) , np.abs(
    [ ediobj.Z.z[:, 0, 1][7]  for ediobj in p.ediObjs_]) , '-ok',
    np.arange(len(p.ediObjs_)), np.abs( z_corrected[7,: ]) , 'or-')

References

zrestore(*, tensor=None, component=None, buffer=None, method='pd', **kws)[source]#

Fix the weak and missing signal at the ‘dead-band`- and recover the missing impedance tensor values.

The function uses the complete frequency (frequency with clean data) collected thoughout the survey to recover by inter/extrapolating the missing or weak frequencies thereby restoring the impedance tensors at that ‘dead-band’. Note that the ‘dead- band’ also known as ‘attenuation -band’ is where the AMT signal is weak or generally abscent.

Parameters:

  • tensor (str, optional, ["resistivity"|"phase"|"z"|"frequency"]) – Name of the tensor. If the name of tensor is given, function returns the tensor valuein two-dimensionals composed of (n_freq , n_sites) where n_freq=number of frequency and n_sations number of sites. Note that if the tensor is passed as boolean values True, the resistivity tensor is exported by default and the ``component``should be the component passed to Processing at initialization.

  • buffer (list [max, min] frequency in Hz) – list of maximum and minimum frequencies. It must contain only two values. If None, the max and min of the clean frequencies are selected. Moreover the [min, max] frequency should not compulsory to fit the frequency range in the data. The given frequency can be interpolated to match the best closest frequencies in the data.

  • method (str, optional , default='pd') – Method of Z interpolation. Use base for scipy interpolation, mean or bff for scaling methods and pd for pandas interpolation methods. Note that the first method is fast and efficient when the number of NaN in the array if relatively few. It is less accurate to use the base interpolation when the data is composed of many missing values. Alternatively, the scaled method(the second one) is proposed to be the alternative way more efficient. Indeed, when mean argument is set, function replaces the NaN values by the nonzeros in the raw array and then uses the mean to fit the data. The result of fitting creates a smooth curve where the index of each NaN in the raw array is replaced by its corresponding values in the fit results. The same approach is used for bff method. Conversely, rather than averaging the nonzeros values, it uses the backward and forward strategy to fill the NaN before scaling. mean and bff are more efficient when the data are composed of a lot of missing values. When the interpolation method is set to pd, function uses the pandas interpolation but ended the interpolation with forward/backward NaN filling since the interpolation with pandas does not deal with all NaN at the begining or at the end of the array.

  • fill_value (array-like, str, optional, default='extrapolate',) – If a ndarray (or float), this value will be used to fill in for requested points outside of the data range. If not provided, then the default is NaN. The array-like must broadcast properly to the dimensions of the non-interpolation axes. If two-element in tuple, then the first element is used as a fill value for x_new < x[0] and the second element is used for x_new > x[-1]. Anything that is not a 2-element tuple (e.g., list or ndarray,regardless of shape) is taken to be a single array-like argument meant to be used for both bounds as below, above = fill_value, fill_value. Using a two-element tuple or ndarray requires bounds_error=False.

  • kws (dict) – Additional keyword arguments from interpolate1d().

Returns:

  • Array-like of watex.external.z.Z objects

  • Array collection of new Z impedances objects with dead-band tensor

  • recovered. watex.externals.z..Z are ndarray (nfreq, 2, 2).

  • 2x2 matrices for components xx, xy and yx, yy. If tensor given,

  • it returns a collection of 2D tensor of each stations.

class watex.methods.em.ZC(window_size=5, c=2, **kws)[source]#

Bases: EM

Impedance tensor multiple EDI correction class.

Applied filters in a collections of EDI objects.

New in version v0.2.0.

Parameters:

  • data (Path-like object or list of :class:watex.edi.Edi` or pycsamt.core.edi.Edi objects) – Collections of EDI-objects

  • window_size (int) – the length of the window. Must be greater than 1 and preferably an odd integer number. Default is 5

  • c (int, default=2) – A window-width expansion factor that must be input to the filter adaptation process to control the roll-off characteristics of the applied Hanning window. It is recommended to select c between 1 and 4 [1].

References

[1]

Torres-Verdin and Bostick, 1992, Principles of spatial surface electric field filtering in magnetotellurics: electromagnetic array profiling(EMAP), Geophysics, v57, p603-622.https://doi.org/10.1190/1.2400625

Examples

>>> import watex
>>> from watex.methods import ZC
>>> edi_sample = watex.fetch_data ('edis', samples =17, return_data =True)
>>> zo = ZC ().fit(edi_sample)
>>> zo.ediObjs_[0].Z.resistivity[:, 0, 1][:10] # for xy components
array([ 427.43690401,  524.87391142,  732.85475419, 1554.3189371 ,
       3078.87621649, 1550.62680093,  482.64709443,  605.3153687 ,
        499.49191936,  468.88692879])
>>> zss = zo.remove_static_shift(ss_fx =0.7 , ss_fy =0.85 )
>>> zss[0].resistivity[:, 0, 1][:10] # corrected xy components
array([ 278.96395263,  319.11187959,  366.43170231,  672.24446295,
       1344.20120487,  691.49270688,  260.25625996,  360.02452498,
        305.97381587,  273.46251961])
get_ss_correction_factors(r=1000.0, nfreq=21, skipfreq=5, tol=0.12)[source]#

Compute the static shift correction factor from a station using a spatial median filter.

This will find those station within the given radius (meters). Then it will find the median static shift for the x and y modes and remove it, given that it is larger than the shift tolerance away from 1.

Parameters:

  • r (float, default=1000.) – radius to look for nearby stations, in meters.

  • nfreq (int, default=21) – number of frequencies calculate the median static shift. This is assuming the first frequency is the highest frequency. Cause usually highest frequencies are sampling a 1D earth.

  • skipfreq** (int, default=5) – number of frequencies to skip from the highest frequency. Sometimes the highest frequencies are not reliable due to noise or low signal in the AMT deadband. This allows you to skip those frequencies.

tol: float, default=0.12

Tolerance on the median static shift correction. If the data is noisy the correction factor can be biased away from 1. Therefore the shift_tol is used to stop that bias. If 1-tol < correction < 1+tol then the correction factor is set to 1

Returns:

(sx_x, ss_y) – static shift corrections factor for x and y modes

Return type:

(float, float)

Examples

>>> import watex
>>> from watex.methods import ZC
>>> edi_sample = watex.fetch_data ('edis', samples =17 ,
                                   return_data =True )
>>> zo = ZC ().fit(edi_sample).get_ss_correction_factors ()
Out[16]: (1.5522030221266643, 0.742682340427651)
remove_distortion(distortion, /, error=None, out=False, **kws)[source]#

Remove distortion D form an observed impedance tensor Z.

Allow to obtain the uperturbed “correct” \(Z_{0}\) expressed as:

\[Z = D * Z_{0}\]
Parameters:

  • distortion_tensor (np.ndarray(2, 2, dtype=real)) – Real distortion tensor as a 2x2

  • error (np.ndarray(2, 2, dtype=real), Optional) – Propagation of errors/uncertainties included

  • out (bool , default =False,) – Output new filtered EDI. Otherwise return Z collections objects of corrected Tensors.

Returns:

d, new_z, new_z_err –

  • input distortion tensor

  • impedance tensor with distorion removed

  • impedance tensor error after distortion is removed

If export=True, export to new EDI and return None.

Return type:

NDArray ( 2 x2 , dtype =real )

Examples

>>> import watex
>>> from watex.methods import ZC
>>> edi_sample = watex.fetch_data ('edis', samples =17 , return_data =True )
>>> zo = ZC ().fit(edi_sample)
>>> zo.ediObjs_[0].Z.z[:, 0, 1][:7]
array([10002.46 +9747.34j , 11679.44 +8714.329j, 15896.45 +3186.737j,
       21763.01 -4539.405j, 28209.36 -8494.808j, 19538.68 -2400.844j,
        8908.448+5251.157j])
        >>> distortion = np.array([[1.2, .5],[.35, 2.1]])
>>> zc = zo.remove_distortion (distortion)
>>> zc[0].z[:, 0, 1] [:7]
        array([ 9724.52643923+9439.96503198j, 11159.25927505+8431.1101919j ,
        14785.52643923+3145.38324094j, 19864.708742  -4265.80166311j,
        25632.53518124-8304.88093817j, 17889.15373134-2484.60144989j,
         8413.19671642+4925.46660981j])
remove_ss_emap(fltr='ama', out=False, smooth=True, drop_outliers=True, **kws)[source]#

Filter Z to remove the static schift using the EMAP moving average filters.

Three available filters:

  • ‘ama’: Adaptative moving average

  • ‘tma’: Trimming moving-average

  • ‘flma’: Fixed-length dipole moving average

Could export new Edi if the keyword argument export is set to True

Parameters:

  • fltr (str , default='ama') –

    Type of filter to apply. Default is Adaptative moving-average of Torres-verdin [1]. Can be [‘ama’|’tma’|’flma’]

    The AMA filter estimates static-corrected apparent resistivities at a single reference frequency by calculating a profile of average impedances along the length of the line. Sounding curves are then shifted so that they intersect the averaged profile.

  • out (bool , default =False,) – Output new filtered EDI. Otherwise return Z collections objects of corrected Tensors.

  • smooth (bool, default=False,) – Smooth the tensor data along the frequencies.

  • versionadded (..) –

  • drop_outliers (bool, default=True) – Suppress outliers in the data when smoothing data along the frequencies axis. Note that drop_outliers does not take effect if smooth is False.

  • versionadded – Polish the tensor data along the frequency axis remove noises and deal with the static shift effect when interferences noises are strong enough.

Returns:

Z – Return None by default (when export is set to False)

Return type:

list of watex.externals.z objects or None

References

[1]

Torres-Verdin and Bostick, 1992, Principles of spatial surface electric field filtering in magnetotellurics: electromagnetic array profiling(EMAP), Geophysics, v57, p603-622.https://doi.org/10.1190/1.2400625

See also

remove_static_shift

Remove static shift using the spatial filter median and write a new edifile.

Examples

>>> import watex
>>> from watex.methods import ZC
>>> edi_sample = watex.fetch_data ('edis', samples =17 , return_data =True )
>>> zo = ZC ().fit(edi_sample)
>>> zo.ediObjs_[0].Z.z[:, 0, 1][:7]
array([10002.46 +9747.34j , 11679.44 +8714.329j, 15896.45 +3186.737j,
       21763.01 -4539.405j, 28209.36 -8494.808j, 19538.68 -2400.844j,
        8908.448+5251.157j])
>>> zc = zo.remove_ss_emap()
>>> zc[0].z[:, 0, 1] [:7]
array([12120.08320804+6939.9874753j , 13030.91462606+6522.58481295j,
       15432.0206124 +4970.42806287j, 21899.60942244+3826.47476912j,
       29109.17100085+4537.17072741j, 19252.07839732+4108.71578943j,
        9473.20464326+4146.50327315j])
remove_static_shift(ss_fx=None, ss_fy=None, out=False, rotate=0.0, **kws)[source]#

Remove the static shift from correction factor from x and y.

The correction factors ss_fx and ss_fy are used for the resistivity in the x and y components for static shift removal.

Factors can be determined by using the get_ss_correction_factors() If None, factors are found using the spatial median filter. Assume the original observed tensor Z is built by a static shift \(S\) and an unperturbated “correct” :math:`Z_{0} :

\[Z = S * Z_{0}\]

therefore the correct Z will be:

\[Z_{0} = S^{(-1)} * Z\]
Parameters:

  • ss_fx (float, Optional) – static shift factor to be applied to x components (ie z[:, 0, :]). This is assumed to be in resistivity scale. If None should be automatically computed using the spatial median filter.

  • ss_fy (float, optional) – static shift factor to be applied to y components (ie z[:, 1, :]). This is assumed to be in resistivity scale. If None , should be computed using the spatial filter median.

  • rotate (float, default=0.) – Rotate Z array by angle alpha in degrees. All angles are referenced to geographic North, positive in clockwise direction. (Mathematically negative!). In non-rotated state, X refs to North and Y to East direction.

  • out (bool , default =False,) – Output new filtered EDI. Otherwise return Z collections objects of corrected Tensors.

  • ss (dict,) – Additional kweyword arguments passed to get_ss_correction_factors()

Returns:

( static shift matrix,coorected_z) – If export is True export to new edis and returns nothing.

Return type:

np.ndarray ((2, 2)) or watex.externals.z.Z

Note

The factors are in resistivity scale, so the entries of the matrix “S” need to be given by their square-roots. Furhermore, ss_fx and ss_fy must be supplied for a manual correction. If one argument of the aforementionned parameters are missing, the auto factor computation could be triggered and reset the given previous given factor.

Examples

>>> import watex
>>> from watex.methods import ZC
>>> edi_sample = watex.fetch_data ('edis', samples =17 , return_data =True )
>>> zo = ZC ().fit(edi_sample)
>>> zo.ediObjs_[0].Z.z[:, 0, 1][:7]
array([10002.46 +9747.34j , 11679.44 +8714.329j, 15896.45 +3186.737j,
       21763.01 -4539.405j, 28209.36 -8494.808j, 19538.68 -2400.844j,
        8908.448+5251.157j])
>>> zc = zo.remove_static_shift ()
>>> zc[0].z[:, 0, 1] [:7]
array([ 8028.46578676+7823.69394148j,  9374.49231974+6994.54856416j,
       12759.27171475+2557.831671j  , 17468.06097719-3643.54946031j,
       22642.21817697-6818.35022516j, 15682.70444455-1927.03534064j,
        7150.35801004+4214.83658174j])