dqsegdb Package¶
dqsegdb Package¶
apicalls Module¶
-
dqsegdb.apicalls.InsertMultipleDQXMLFileThreaded(filenames, logger, server='http://slwebtest.virgo.infn.it', hackDec11=False, debug=True, threads=1, testing_options={})[source]¶ Inserts multiple dqxml files of data into the DQSEGDB.
Input: - filenames is a list of string filenames for DQXML files. - hackDec11 is deprecated (always should be false): This was used to differentiate function against different server APIs before we used numbering an responses to make decisions. - testing_options is a dictionary including (optionally):offset(int),synchronize(time in ‘HH:MM’ format (string))
Output: returns True if it completes sucessfully
-
dqsegdb.apicalls.coalesceResultDictionary(result_dict)[source]¶ Takes a dictionary as returned by QueryTimes or QueryTimeless and converts the lists of tuples into actual segment lists (and coalesces them).
- Parameters
- result_dictdict
This is the input result dictionary from the other api calls
- out_result_dictdict
This is the output result dictionary with actual segment lists (and coalesced results).
-
dqsegdb.apicalls.dqsegdbCascadedQuery(protocol, server, ifo, name, include_list_string, startTime, endTime, debug=False, warnings=True)[source]¶ Queries server for needed flag_versions to generate the result of a cascaded query (was called a versionless query in S6).
Returns a python dictionary representing the calculated result “versionless” flag and also the python dictionaries (in a list) for the flag_versions necessary to construct the result.
- Parameters
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- ifostring
Ex: ‘L1’
- name: `string`
Ex: ‘DMT-SCIENCE’
- versionstring or int
Ex: ‘1’
- include_list_stringstring
Ex: “metadata,known,active”
- startTimeint
Ex: 999999999
- endTimeint
Ex: 999999999
- debugbool
Ex: False
- warningsbool
show warnings for HTTPError (as well as raising exception), default: True
-
dqsegdb.apicalls.dqsegdbCheckVersion(protocol, server, ifo, name, version, warnings=True)[source]¶ Checks for existence of a given version of a flag in the db. Returns true if version exists
- Parameters
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- ifostring
Ex: ‘L1’
- name: `string`
Ex: ‘DMT-SCIENCE’
- versionstring or int
Ex: ‘1’
- warningsbool
show warnings for HTTPError (as well as raising exception), default: True
-
dqsegdb.apicalls.dqsegdbFindEndTime(flag_dict)[source]¶ Determines max end_time from known times in flag_dict
- Parameters
- flag_dict: `dictionary`
Input dictionary, converted from json using json.loads in previous call
- Returns max_end_time: `int`
-
dqsegdb.apicalls.dqsegdbMaxVersion(protocol, server, ifo, name)[source]¶ Checks for existence of a flag in the db, returns maximum version if the flag exists exists, 0 if the flag does not exist.
- Parameters
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- ifostring
Ex: ‘L1’
- name: `string`
Ex: ‘DMT-SCIENCE’
-
dqsegdb.apicalls.dqsegdbQueryTimeless(protocol, server, ifo, name, version, include_list_string, warnings=True)[source]¶ Issue query to server for ifo:name:version without start and end time Returns the python loaded JSON response converted into a dictionary and queryurl! Returns ———- [dictionary,string(url)]
- Parameters
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- ifostring
Ex: ‘L1’
- name: `string`
Ex: ‘DMT-SCIENCE’
- versionstring or int
Ex: ‘1’
- include_list_stringstring
Ex: “metadata,known,active”
- warningsbool
show warnings for HTTPError (as well as raising exception), default: True
-
dqsegdb.apicalls.dqsegdbQueryTimes(protocol, server, ifo, name, version, include_list_string, startTime, endTime, warnings=True)[source]¶ Issue query to server for ifo:name:version with start and end time Returns the python loaded JSON response!
- Parameters
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- ifostring
Ex: ‘L1’
- name: `string`
Ex: ‘DMT-SCIENCE’
- versionstring or int
Ex: ‘1’
- include_list_stringstring
Ex: “metadata,known,active”
- startTimeint
Ex: 999999999
- endTimeint
Ex: 999999999
- warningsbool
show warnings for HTTPError (as well as raising exception), default: True
-
dqsegdb.apicalls.dqsegdbQueryTimesCompatible(protocol, server, ifo, name, version, include_list_string, startTime, endTime, warnings=True)[source]¶ Issue query to server for ifo:name:version with start and end time This is the version that reproduces S6 style query results when the query is empty Returns the python loaded JSON response!
- Parameters
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- ifostring
Ex: ‘L1’
- name: `string`
Ex: ‘DMT-SCIENCE’
- versionstring or int
Ex: ‘1’
- include_list_stringstring
Ex: “metadata,known,active”
- startTimeint
Ex: 999999999
- endTimeint
Ex: 999999999
- warningsbool
show warnings for HTTPError (as well as raising exception), default: True
-
dqsegdb.apicalls.dtd_uri_callback(uri)[source]¶ S6 helper function for XML file writing and parsing using a dtd.
-
dqsegdb.apicalls.parseKnown(jsonResult)[source]¶ Accepts jsonResult from reportKnown call and parses into expected format for ligolw_segment_query client to generate xml.
-
dqsegdb.apicalls.patchWithFailCases(i, url, debug=True, inlogger=None, testing_options={})[source]¶ Attempts to patch data to a url where the data is in a dictionary format that can be directly dumped to json that the dqsegdb server expects. Correctly fails to making a new version or flag in the database as needed.
-
dqsegdb.apicalls.queryAPIVersion(protocol, server, verbose, warnings=True)[source]¶ Construct url and issue query to get the reported list of all IFOs provided by dqsegd and the API version of the server.
- Parameters
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- warningsbool
show warnings for HTTPError (as well as raising exception), default: True
-
dqsegdb.apicalls.reportActive(protocol, server, includeSegments, verbose, gps_start_time, gps_end_time, warnings=True)[source]¶ Construct url and issue query to get the reported list of all active segments for all flags in the time window provided. From the API Doc: Get a JSON string resource containing the active segments for all flags between t1 and t2. Note that this returns exactly what /dq/IFO/FLAG/VERSION does, except for ALL flags over the query period instead of one flag. The clients must assume that they may get empty active lists for flags that are unactive between times t1 and t2.
- Parameters
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- includeSegmentsboolean
Ex: True
- verboseboolean
Ex: True
- gps_start_time: `int`
Ex: 999999999
- gps_end_time: `int`
Ex: 999999999
- warningsbool
show warnings for HTTPError (as well as raising exception), default: True
-
dqsegdb.apicalls.reportFlags(protocol, server, verbose, warnings=True)[source]¶ Construct url and issue query to get the reported list of all flags provided by dqsegdb. From the API Doc: Get a JSON formatted string resource describing all the flags in the database. This provides an optimization by returning all flag names and all associated versions in a single call.
- Parameters
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- warningsbool
show warnings for HTTPError (as well as raising exception), default: True
-
dqsegdb.apicalls.reportKnown(protocol, server, includeSegments, verbose, gps_start_time, gps_end_time, warnings=True)[source]¶ Construct url and issue query to get the reported list of all known segments for all flags in the time window provided. From the API Doc: Get a JSON string resource containing the known segments for all flags between t1 and t2. Note that this returns exactly what /dq/IFO/FLAG/VERSION/known does, except for ALL flags over the query period instead of one flag. The clients must assume that they may get empty known lists for flags that are unknown between times t1 and t2.
- Parameters
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- includeSegmentsboolean
Ex: True
- verboseboolean
Ex: True
- gps_start_time: `int`
Ex: 999999999
- gps_end_time: `int`
Ex: 999999999
- warningsbool
show warnings for HTTPError (as well as raising exception), default: True
-
dqsegdb.apicalls.setupSegment_md(filename, xmlparser, lwtparser, debug)[source]¶ Helper function used to setup ligolw parser (S6 xml generation tool).
clientutils Module¶
-
class
dqsegdb.clientutils.ShowTypesResult[source]¶ Bases:
object-
ifos¶
-
name¶
-
segment_definer_comment¶
-
segment_summary_comment¶
-
segment_summary_end_time¶
-
segment_summary_start_time¶
-
version¶
-
-
class
dqsegdb.clientutils.ShowTypesResultTable(*args)[source]¶ Bases:
glue.ligolw.table.Table-
RowType¶ alias of
ShowTypesResult
-
tableName= 'show_types_result:table'¶
-
validcolumns= {'ifos': 'lstring', 'name': 'lstring', 'segment_definer_comment': 'lstring', 'segment_summary_comment': 'lstring', 'segment_summary_end_time': 'int_4s', 'segment_summary_start_time': 'int_4s', 'version': 'int_4s'}¶
-
-
dqsegdb.clientutils.add_segment_info_ns(doc, proc_id, segdefs, segments, segment_summaries)[source]¶
-
dqsegdb.clientutils.add_to_segment_summary_ns(xmldoc, proc_id, seg_def_id, sgmtlist, comment='')[source]¶
-
dqsegdb.clientutils.calculate_combined_result(includedJSON, excludedJSON, startTime, endTime, ifo)[source]¶ Calculate the result of the union of the active times for the included flag less the intersection of that result with the union of the excluded flags Inputs are 2 lists of python dictionaries representing the JSON (already have run json.loads() on the JSON), a start time, and end time, and the ifo name (it does not make sense to include/exclude across multiple ifos)
- Parameters
- startTimeint
Ex: 999999999
- endTimeint
Ex: 999999999
- ifostring
Ex: ‘L1’
-
dqsegdb.clientutils.calculate_versionless_result(jsonResults, startTime, endTime, ifo_input=None)[source]¶ Construct output segments lists from multiple JSON objects. The jsonResults input is a list of json ojbects and are expected to be in order of decreasing versions.
-
dqsegdb.clientutils.include_exclude_caller(includedList, excludedList, startTime, endTime, protocol, server, include_list_string)[source]¶ Function to query the dqsegdb for lists of included and excluded flags. Returns lists of JSON for the included and excluded flags and lists of URLs used to query the database.
- Parameters
- includedListlist
List of ifo,name,version tuples
- excludedListlist
List of ifo,name,version tuples
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- include_list_stringstring
Ex: “metadata,known,active”
- startTimeint
Ex: 999999999
- endTimeint
Ex: 999999999
-
dqsegdb.clientutils.run_query_segments(doc, process_id, engine, gps_start_time, gps_end_time, include_segments, exclude_segments, result_name)[source]¶
-
dqsegdb.clientutils.run_query_types(doc, proc_id, connection, engine, gps_start_time, gps_end_time, included_segments)[source]¶
-
dqsegdb.clientutils.run_show_types(doc, connection, engine, gps_start_time, gps_end_time, included_segments_string, excluded_segments_string)[source]¶
jsonhelper Module¶
-
class
dqsegdb.jsonhelper.FlagVersion(ifo, name, version)[source]¶ Bases:
objectClass to set up a flag version object for parsing into JSON
-
appendActive(self, active_segments)[source]¶ Appends a segmentlist of active segments to the existing active segments for the object, and coalesces.
-
-
class
dqsegdb.jsonhelper.InsertFlagVersion(ifo, name, version)[source]¶ Bases:
dqsegdb.jsonhelper.PatchFlagVersionClass to set up a flag version object for parsing into JSON
Class to extend basic flag version class to include insert_history
Adds metadata for initial inserts
-
class
dqsegdb.jsonhelper.InsertFlagVersionOld(ifo, name, version)[source]¶ Bases:
dqsegdb.jsonhelper.PatchFlagVersionClass to set up a flag version object for parsing into JSON
Class to extend basic flag version class to include insert_history
Adds metadata for initial inserts
-
class
dqsegdb.jsonhelper.PatchFlagVersion(ifo, name, version, hackDec11=False)[source]¶ Bases:
dqsegdb.jsonhelper.FlagVersionClass to set up a flag version object for parsing into JSON
Class to extend basic flag version class to include insert_history
-
dqsegdb.jsonhelper.buildFlagDict(ifo, name, version, known_segments, active_segments)[source]¶ Helper function to build a flag_version dictionary for JSON production.
known_segments and active_segments are assumed to be segmentlist objects Note that this currently does not generate a false “query_metadata” block
-
dqsegdb.jsonhelper.convertJSONtoXML(json_response)[source]¶ Incomplete!! Converts a standard JSON response from the dqsegdb server into a DQXML format.
-
dqsegdb.jsonhelper.convert_json_list_to_segmentlist(jsonlist)[source]¶ Helper function used to convert json list of lists type object to a segmentlist object
urifunctions Module¶
-
dqsegdb.urifunctions.constructFlagQueryURL(protocol, server, ifo)[source]¶ Simple url construction method for dqsegdb server flag queries.
- Parameters
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- ifostring
Ex: ‘L1’
-
dqsegdb.urifunctions.constructSegmentQueryURL(protocol, server, ifo, name, version, include_list_string)[source]¶ Simple url construction method for dqsegdb server flag:version queries not including restrictions on time ranges.
- Parameters
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- ifostring
Ex: ‘L1’
- name: `string`
Ex: ‘DMT-SCIENCE’
- versionstring or int
Ex: ‘1’
- include_list_stringstring
Ex: “metadata,known,active”
-
dqsegdb.urifunctions.constructSegmentQueryURLTimeWindow(protocol, server, ifo, name, version, include_list_string, startTime, endTime)[source]¶ Simple url construction method for dqsegdb server flag:version queries including restrictions on time ranges.
- Parameters
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- ifostring
Ex: ‘L1’
- name: `string`
Ex: ‘DMT-SCIENCE’
- versionstring or int
Ex: ‘1’
- include_list_stringstring
Ex: “metadata,known,active”
- startTimeint
Ex: 999999999
- endTimeint
Ex: 999999999
-
dqsegdb.urifunctions.constructVersionQueryURL(protocol, server, ifo, name)[source]¶ Simple url construction method for dqsegdb server version queries.
- Parameters
- protocolstring
Ex: ‘https’
- serverstring
Ex: ‘dqsegdb5.phy.syr.edu’
- ifostring
Ex: ‘L1’
- name: `string`
Ex: ‘DMT-SCIENCE’
-
dqsegdb.urifunctions.getDataUrllib2(url, timeout=900, logger=None, warnings=True, **urlopen_kw)[source]¶ Return response from server
- Parameters
- urlstr
remote URL to request (HTTP or HTTPS)
- timeoutfloat
time (seconds) to wait for reponse
- loggerlogging.Logger
logger to print to
- **urlopen_kw
other keywords are passed to
urllib.request.urlopen()
- Returns
- responsestr
the text reponse from the server
-
dqsegdb.urifunctions.patchDataUrllib2(url, payload, timeout=900, logger=None, **urlopen_kw)[source]¶ Wrapper method for urllib2 that supports PATCHs to a url.
- Parameters
- urlstring
- payloadstring
JSON formatted string