2021SC@SDUSC
stream_tools.py: implements the FileLikeQueue helper class. This helper class is designed to handle use case requests that are passed in PUT and should be streamed directly to the remote target. Usage: return an instance of this class to ` begin_write 'and deliver it to consumers at the same time:
def begin_write(self, contentType=None):
Queue = FileLikeQueue(max_size=1)
requests.post(..., data = queue)
Return queue
The file contains FileLikeQueue and StreamingFile.
FileLikeQueue class: behaves like a block queue for a file. read() and write() are usually called from different threads. This helper class is designed to handle the use case request of the incoming PUT, which should be streamed directly to the remote target:
def begin_write(self, contentType=None):
# create proxy buffer
Queue = FileLikeQueue(max_size=1)
# ... And use it as a source for consumers:
requests.post(..., data = queue)
# pass it as a target to the PUT handler
Return queue
Function read(self, size=0): reads a large byte from the queue. Size = 0: read the next block (any length) > 0: read a large block of 'size' bytes (less if the stream is closed) < 0: read all bytes as a single block (i.e. block until the stream is closed). This method will block until the requested size is available. However, if close() is called, '', it will be returned immediately.
def read(self, size=0):
"""Read a chunk of bytes from queue.
size = 0: Read next chunk (arbitrary length)
> 0: Read one chunk of `size` bytes (or less if stream was closed)
< 0: Read all bytes as single chunk (i.e. blocks until stream is closed)
This method blocks until the requested size become available.
However, if close() was called, '' is returned immediately.
"""
res = self.unread
self.unread = ""
# Get next chunk, cumulating requested size as needed
while res == "" or size < 0 or (size > 0 and len(res) < size):
try:
# Read pending data, blocking if neccessary
# (but handle the case that close() is called while waiting)
res += compat.to_native(self.queue.get(True, 0.1))
except compat.queue.Empty:
# There was no pending data: wait for more, unless close() was called
if self.is_closed:
break
# Deliver `size` bytes from buffer
if size > 0 and len(res) > size:
self.unread = res[size:]
res = res[:size]
# print("FileLikeQueue.read({}) => {} bytes".format(size, len(res)))
return resThe function write(self, chunk) puts a large byte (or an iteratable) into the queue. If the number of max_size blocks is reached, it may block.
def write(self, chunk):
"""Put a chunk of bytes (or an iterable) to the queue.
May block if max_size number of chunks is reached.
"""
if self.is_closed:
raise ValueError("Cannot write to closed object")
# print("FileLikeQueue.write(), n={}".format(len(chunk)))
# Add chunk to queue (blocks if queue is full)
if compat.is_basestring(chunk):
self.queue.put(chunk)
else: # if not a string, assume an iterable
for o in chunk:
self.queue.put(o)StreamingFile class: a file object that surrounds an iterator / data stream
Functions__ init__(self, data_stream): initialize an object with a data stream.
def __init__(self, data_stream):
"""Intialise the object with the data stream."""
self.data_stream = data_stream
self.buffer = ""Function read(self, size=None): reads bytes from the iterator
code
def read(self, size=None):
"""Read bytes from an iterator."""
while size is None or len(self.buffer) < size:
try:
self.buffer += next(self.data_stream)
except StopIteration:
break
sized_chunk = self.buffer[:size]
if size is None:
self.buffer = ""
else:
self.buffer = self.buffer[size:]
return sized_chunkwsgidav_app.py: WSGI container that handles HTTP requests. This object is passed to the WSGI server and represents our WsgiDAV application externally. During initialization: use the configuration dictionary to initialize the lock manager, property manager, and domain controller. Create a dictionary that is shared to the provider mapping. Initialize the middleware object and set the WSGI application stack. For each request: find the registered DAV provider for the current request. Add or modify information in WSGI ` ` environment `: mount point currently shared by environment ["SCRIPT_NAME"]. Environment ["PATH_INFO"] resource path, relative to mount path. The environment ["wsgidav.provider"] is registered to handle the current DAVProvider object requirements. Environment ["wsgidav.config"] configuration dictionary. Environment ["wsgidav.verbose"] debug level [0-3]. Record the HTTP request, and then pass the request to the first middleware.
Function resolve_provider(self, path): get the registered DAVProvider of the given path. Return: tuple: (shared, provider)
def resolve_provider(self, path):
"""Get the registered DAVProvider for a given path.
Returns:
tuple: (share, provider)
"""
# Find DAV provider that matches the share
share = None
lower_path = path.lower()
for r in self.sorted_share_list:
# @@: Case sensitivity should be an option of some sort here;
# os.path.normpath might give the preferred case for a filename.
if r == "/":
share = r
break
elif lower_path == r or lower_path.startswith(r + "/"):
share = r
break
if share is None:
return None, None
return share, self.provider_map.get(share)xml_tools.py: small package for different etree packages.
Function string_to_xml(text): converts an XML string to etree. XML Element
def string_to_xml(text):
"""Convert XML string into etree.Element."""
try:
return etree.XML(text)
except Exception:
# TODO:
# ExpatError: reference to invalid character number: line 1, column 62
# litmus fails, when xml is used instead of lxml
# 18. propget............... FAIL (PROPFIND on `/temp/litmus/prop2':
# Could not read status line: connection was closed by server)
# text = <ns0:high-unicode xmlns:ns0="http://example.com/neon/litmus/">��
# </ns0:high-unicode>
# t2 = text.encode("utf8")
# return etree.XML(t2)
_logger.error(
"Error parsing XML string. "
"If lxml is not available, and unicode is involved, then "
"installing lxml _may_ solve this issue."
)
_logger.error("XML source: {}".format(text))
raiseFunction xml_to_bytes(element, pretty_print=False): etree.tostring's wrapper, which handles unsupported pretty_ Print option and pre add an encoded header.
def xml_to_bytes(element, pretty_print=False):
"""Wrapper for etree.tostring, that takes care of unsupported pretty_print
option and prepends an encoding header."""
if use_lxml:
xml = etree.tostring(
element, encoding="UTF-8", xml_declaration=True, pretty_print=pretty_print
)
else:
xml = etree.tostring(element, encoding="UTF-8")
if not xml.startswith(b"<?xml "):
xml = b'<?xml version="1.0" encoding="utf-8" ?>\n' + xml
assert xml.startswith(b"<?xml ") # ET should prepend an encoding header
return xmlFunction make_multistatus_el():etree.Element, which handles unsupported nsmap options.
def make_multistatus_el():
"""Wrapper for etree.Element, that takes care of unsupported nsmap option."""
if use_lxml:
return etree.Element("{DAV:}multistatus", nsmap={"D": "DAV:"})
return etree.Element("{DAV:}multistatus")Function make_prop_el():etree.Element, which handles unsupported nsmap options
def make_prop_el():
"""Wrapper for etree.Element, that takes care of unsupported nsmap option."""
if use_lxml:
return etree.Element("{DAV:}prop", nsmap={"D": "DAV:"})
return etree.Element("{DAV:}prop")Function make_ sub_ element(parent, tag, nsmap=None):etree. Wrapper for subelement that handles unsupported nsmap options.
def make_sub_element(parent, tag, nsmap=None):
"""Wrapper for etree.SubElement, that takes care of unsupported nsmap option."""
if use_lxml:
return etree.SubElement(parent, tag, nsmap=nsmap)
return etree.SubElement(parent, tag)Function element_content_as_string(element):
Serialize etree Element.
Note: an element may contain multiple child elements or only text (that is, no child elements at all). Therefore, when passed back to etree.XML(), the generated string may throw an exception
def element_content_as_string(element):
"""Serialize etree.Element.
Note: element may contain more than one child or only text (i.e. no child
at all). Therefore the resulting string may raise an exception, when
passed back to etree.XML().
"""
if len(element) == 0:
return element.text or "" # Make sure, None is returned as ''
stream = compat.StringIO()
for childnode in element:
stream.write(xml_to_bytes(childnode, pretty_print=False) + "\n")
# print(xml_to_bytes(childnode, pretty_print=False), file=stream)
s = stream.getvalue()
stream.close()
return s