autorino.download package

Submodules

autorino.download.dwl_cls module

class autorino.download.dwl_cls.DownloadGnss(out_dir, tmp_dir, log_dir, inp_dir, inp_file_regex, epoch_range, access, site=None, session=None, options=None, metadata=None)[source]

Bases: StepGnss

A class to handle the downloading of GNSS data.

Inherits from

arocmn.StepGnss

param out_dir:

The output directory where the downloaded files will be stored.

type out_dir:

str

param tmp_dir:

The temporary directory used during the download process.

type tmp_dir:

str

param log_dir:

The directory where log files will be stored.

type log_dir:

str

param epoch_range:

The range of epochs for which data will be downloaded.

type epoch_range:

EpochRange

param access:

A dictionary containing access information such as protocol, hostname, login, and password.

type access:

dict

param site:

The site information. Default is None.

type site:

dict, optional

param session:

The session information. Default is None.

type session:

dict, optional

param options:

Additional options for the download process. Default is None.

type options:

dict, optional

param metadata:
The metadata to be included in the converted RINEX files. Possible inputs are:

  • list of string (sitelog file paths),

  • single string (single sitelog file path)

  • single string (directory containing the sitelogs)

  • list of MetaData objects

  • single MetaData object.

Defaults to None.

type metadata:

str or list, optional

__init__(out_dir, tmp_dir, log_dir, inp_dir, inp_file_regex, epoch_range, access, site=None, session=None, options=None, metadata=None)[source]

Initialize the DownloadGnss object.

Parameters:

  • out_dir (str) – The output directory where the downloaded files will be stored.

  • tmp_dir (str) – The temporary directory used during the download process.

  • log_dir (str) – The directory where log files will be stored.

  • epoch_range (EpochRange) – The range of epochs for which data will be downloaded.

  • access (dict) – A dictionary containing access information such as protocol, hostname, login, and password.

  • site (dict, optional) – The site information. Default is None.

  • session (dict, optional) – The session information. Default is None.

  • options (dict, optional) – Additional options for the download process. Default is None.

  • metadata (str or list, optional) –

    The metadata to be included in the converted RINEX files. Possible inputs are:

    • list of string (sitelog file paths),

    • single string (single sitelog file path)

    • single string (directory containing the sitelogs)

    • list of MetaData objects

    • single MetaData object.

    Defaults to None.

ask_local_raw()[source]

Guess the paths and name of the local raw files based on the EpochRange and table’s fpath_inp basename i.e. the theoretical remote file name generated with guess_remot_raw().

see also method guess_remot_raw(),

ask_remote_raw()[source]

Retrieve the list of remote files from the server.

This method guesses the remote directories and then lists the files in those directories based on the protocol specified in the access information.

Returns:

A list of remote file paths.

Return type:

list

download(verbose=False, force=False, remote_find_method='ask', invalidate_small_local_files=True, timeout=60, max_try=4, sleep_time=5, ping_max_try=4, ping_timeout=20, ping_disable=False)[source]

Frontend method to download files from a GNSS receiver

Parameters:

  • verbose (bool, optional) – If True, prints detailed information during the download process. Default is False.

  • force (bool, optional) – If True, forces the download even if the files already exist locally. Default is False.

  • remote_find_method (str, optional) – Method to find remote files. Can be ‘ask’ to list files from the server (for FTP only) or ‘guess’ to guess file paths. Default is ‘ask’.

  • invalidate_small_local_files (bool, optional) – If True, invalidates small local files to ensure they are re-downloaded. Default is True.

  • timeout (int, optional) – Timeout in seconds for the download operations. Default is 60.

  • max_try (int, optional) – Maximum number of retry attempts for the download operations. Default is 4.

  • sleep_time (int, optional) – Sleep time in seconds between retry attempts. Default is 5.

  • ping_max_try (int, optional) – Maximum number of retry attempts for pinging the remote server. Default is 4.

  • ping_timeout (int, optional) – Timeout in seconds for pinging the remote server. Default is 20.

  • ping_disable (bool, optional) – If True, skips the pinging of the remote server. Default is False.

Return type:

None

fetch_remote_files(force=False, timeout=60, max_try=4, sleep_time=5)[source]

will download locally the files which have been identified by the guess_remote_files method

exploits the fname_remote column of the DownloadGnss.table attribute

This fetch_remote_files method is for the download stricly speaking. Ìn operation, use the download method which does a broader preliminary actions.

guess_local_raw()[source]

Guess the paths and name of the local raw files based on the EpochRange and inp_file_regex attributes of the DownloadGnss object

see also method guess_remot_raw(),

guess_remot_raw()[source]

Guess the paths and name of the remote files based on the Session and EpochRange attributes of the DownloadGnss

see also method guess_local_raw()

guess_remote_dirs()[source]

this method is specific for ask_remote_raw guessing the directories is different from guessing the files: * no hostname * no filename (obviously)

property inp_basename
property inp_dirname
mono_fetch(irow, force=False, timeout=60, max_try=4, sleep_time=5)[source]
ping_remote(ping_max_try=4, ping_timeout=20)[source]

Ping the remote server to check if it is reachable.

set_ftp_obj(timeout=15, max_try=4, sleep_time=5)[source]

Create the FTP object for the FTP protocol.

autorino.download.dwl_fcts module

Created on Mon Mar 27 09:16:34 2023

@author: psakic

exception autorino.download.dwl_fcts.AutorinoDownloadError[source]

Bases: AutorinoError

exception autorino.download.dwl_fcts.AutorinoError[source]

Bases: Exception

class autorino.download.dwl_fcts.TqdmToLogger(logger_inp, level=None)[source]

Bases: StringIO

Output stream for TQDM which will output to logger module instead of the StdOut.

buf = ''
flush()[source]

Flush write buffers, if applicable.

This is not implemented for read-only and non-blocking streams.

level = None
logger = None
write(buf)[source]

Write string to file.

Returns the number of characters written, which is always equal to the length of the string.

autorino.download.dwl_fcts.check_file_size(file_path, min_size=1000)[source]

Check the size of a file and log a warning if it is too small to be useful.

Parameters:

  • file_path (str) – The path to the file to check.

  • min_size (int, optional) – The minimum file size in bytes. Default is 1000 bytes.

Returns:

  • ok_size (bool) – True if the file size is acceptable, otherwise False.

  • file_size (int) – The size of the file in bytes

autorino.download.dwl_fcts.download_ftp(url, output_dir, username=None, password=None, timeout=30, max_try=4, sleep_time=5, ftp_obj_inp=None)[source]

Download a file from an FTP server with retry logic and progress bar.

Parameters:

  • url (str) – The URL of the file to download.

  • output_dir (str) – The directory where the downloaded file will be saved.

  • username (str) – The username for FTP login. Overrided by the ftp_obj_inp parameter.

  • password (str) – The password for FTP login. Overrided by the ftp_obj_inp parameter.

  • timeout (int, optional) – The timeout for FTP connection in seconds. Default is 15 seconds.

  • max_try (int, optional) – The maximum number of retry attempts in case of failure. Default is 3.

  • sleep_time (int, optional) – The sleep time between retry attempts in seconds. Default is 5 seconds.

  • ftp_obj_inp (ftplib.FTP, optional) – An existing FTP object to use for the connection. Overrides the username and password parameters. Default is None.

Returns:

The path to the downloaded file, or an empty string if the download failed. If something goes wrong, the function will raise an AutorinoDownloadError (no empty string or None).

Return type:

str

Raises:

AutorinoDownloadError – If the download fails after the maximum number of retry attempts.

autorino.download.dwl_fcts.download_http(url, output_dir, timeout=120, max_try=4, sleep_time=5)[source]

Download a file from an HTTP server with retry logic and progress bar.

Parameters:

  • url (str) – The URL of the file to download.

  • output_dir (str) – The directory where the downloaded file will be saved.

  • timeout (int, optional) – The timeout for the HTTP connection in seconds. Default is 120 seconds.

  • max_try (int, optional) – The maximum number of retry attempts in case of failure. Default is 4.

  • sleep_time (int, optional) – The sleep time between retry attempts in seconds. Default is 5 seconds.

Returns:

The path to the downloaded file, or an empty string if the download failed.

Return type:

str

Raises:

AutorinoDownloadError – If the download fails after the maximum number of retry attempts.

autorino.download.dwl_fcts.ftp_create_obj(hostname_inp, username=None, password=None, timeout=15, max_try=3, sleep_time=5)[source]

Create an FTP object, and retry in case of a timeout.

Parameters:

  • hostname_inp (str) – The hostname of the FTP server.

  • username (str, optional) – The username for FTP login. Default is None.

  • password (str, optional) – The password for FTP login. Default is None.

  • timeout (int, optional) – The timeout for FTP connection in seconds. Default is 15 seconds.

  • max_try (int, optional) – The maximum number of retry attempts in case of failure. Default is 3.

  • sleep_time (int, optional) – The sleep time between retry attempts in seconds. Default is 5 seconds.

Returns:

The FTP object if the connection is successful, otherwise None.

Return type:

ftplib.FTP or None

Raises:

TimeoutError – If the connection fails after the maximum number of retry attempts.

autorino.download.dwl_fcts.join_url(protocol_inp, hostname_inp, dir_inp, fname_inp)[source]

a wrapper to classical join to format correctly the URL

autorino.download.dwl_fcts.list_remote_ftp(hostname, remote_dir, username=None, password=None, timeout=15, max_try=3, ftp_obj_inp=None)[source]

List files in a remote FTP directory.

Parameters:

  • hostname (str) – The hostname of the FTP server.

  • remote_dir (str) – The remote directory to list files from.

  • username (str, optional) – The username for FTP login. Default is None.

  • password (str, optional) – The password for FTP login. Default is None.

  • timeout (int, optional) – The timeout for FTP connection in seconds. Default is 15 seconds.

  • max_try (int, optional) – The maximum number of retry attempts in case of failure. Default is 3.

  • ftp_obj_inp (ftplib.FTP, optional) – An existing FTP object to use for the connection. Default is None.

Returns:

A list of file paths in the remote directory.

Return type:

list

Raises:

AutorinoDownloadError – If the FTP connection fails or if username/password are missing.

autorino.download.dwl_fcts.list_remote_http(hostname, remote_dir)[source]
autorino.download.dwl_fcts.ping(host, ping_timeout=20)[source]

Executes the ping command and captures the output.

This function sends a single ICMP echo request to the specified host and captures the round-trip time (RTT) from the output.

Parameters:

  • host (str) – The hostname or IP address to ping.

  • ping_timeout (int, optional) – The timeout for the ping command in seconds. Default is 10 seconds.

Returns:

The round-trip time (RTT) in seconds if the ping is successful, otherwise None.

Return type:

float or None

autorino.download.dwl_fcts.size_remote_file_http(url)[source]