博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
Python和sendfile
阅读量:2520 次
发布时间:2019-05-11

本文共 10165 字,大约阅读时间需要 33 分钟。

is a UNIX system call which provides a “zero-copy” way of copying data from one file descriptor (a file) to another (a socket). Because this copying is done entirely within the kernel, sendfile(2) is more efficient than the combination of “file.read()” and “socket.send()”, which requires transferring data to and from user space.  This copying of the data twice imposes some performance and resource penalties which sendfile(2) syscall avoids; it also results in a single system call (and thus only one context switch), rather than the series of / system calls (each system call requiring a context switch) used internally for the data copying. A more exhaustive explanation of how sendfile(2) works is available ,  but long story short is that sending a file with sendfile() is usually  than using plain . Typical applications which can benefit from using sendfile() are FTP and HTTP servers.

是UNIX系统调用,它提供“零复制”方式将数据从一个文件描述符(一个文件)复制到另一个文件描述符(一个套接字)。 由于此复制完全在内核中完成,因此sendfile(2)比“ file.read()”和“ socket.send()”的组合更有效,后者需要在用户空间之间来回传输数据。 两次复制数据会造成一些性能和资源损失,这是sendcall(2)syscall避免的。 它还导致单个系统调用(因此只有一个上下文切换),而不是内部用于数据复制的一系列 / 系统调用(每个系统调用都需要上下文切换)。 关于sendfile(2)的工作原理的更详尽的解释可以 ,但是长话短说,使用sendfile()发送文件通常比使用普通 。 可以从使用sendfile()中受益的典型应用是FTP和HTTP服务器。

socket.sendfile() ( socket.sendfile())

I recently contributed a patch for Python’s socket module which adds a high-level  method (see full discussion at ). socket.sendfile() will transmit a file until EOF is reached by attempting to use , if available, else it falls back on using plain socket.send(). Internally, it takes care of handling socket timeouts and provides two optional parameters to move the file offset or to send only a limited amount of bytes. I came up with this idea because getting all of that right is a bit tricky, so a generic wrapper seemed to be convenient to have. socket.sendfile() will make its appearance in Python 3.5.

我最近为Python的套接字模块提供了一个补丁,该补丁添加了一个高级方法(请参阅完整讨论)。 socket.sendfile()将一直传输文件,直到尝试通过使用达到EOF 如果可用),否则将使用普通的socket.send()退回。 在内部,它负责处理套接字超时,并提供两个可选参数来移动文件偏移或仅发送有限数量的字节。 我想出了这个主意,是因为要正确地使用所有这些技巧有些棘手,因此使用通用包装器似乎很方便。 socket.sendfile()将在Python 3.5中显示。

sendfile和Python ( sendfile and Python)

sendfile(2) made its first appearance into the Python stdlib kind of late: Python 3.3. It was contributed by Ross Lagerwall and me in . Since the patch didn’t make it into python 2.X and I wanted I later decided to release it as a stand alone module working with older (2.5+) Python versions (see project). Starting with version 3.5, Python will hopefully start using sendfile() more extensively, in details:

sendfile(2)首次出现在Python stdlib中:Python 3.3。 它由Ross Lagerwall和我在 。 由于该修补程序并未进入python 2.X,并且我想我后来决定将其作为独立模块发布,与较旧的(2.5+)Python版本一起使用(请参阅项目)。 从3.5版开始,Python有望开始更广泛地使用sendfile(),详细信息:

  • asyncio – there are some plans for this even though no actual patch yet, see and .
  • asyncio –即使没有实际补丁,也有一些计划,请参阅和 。
. Now that socket.sendfile() is in place it seems natural to add support for it as well (see 。 现在socket.sendfile()就位了,添加它的支持似乎也很自然(请参阅 ). )。

反向移植到Python 2.6和2.7 ( Backport to Python 2.6 and 2.7)

For those of you who are interested in using socket.sendfile() with older Python 2.6 and 2.7 versions here’s a backport. It requires modules to be installed. Full code including tests is hosted .

对于那些对将Socket.sendfile()与旧版2.6和2.7版本一起使用感兴趣的人,这里提供了一个backport。 它需要安装模块。 完整的代码(包括测试)位于 。

#!/usr/bin/env python"""This is a backport of socket.sendfile() for Python 2.6 and 2.7.socket.sendfile() will be included in Python 3.5:http://bugs.python.org/issue17552Usage:>>> import socket>>> file = open("somefile.bin", "rb")>>> sock = socket.create_connection(("localhost", 8021))>>> sendfile(sock, file)42319283>>>"""import errnoimport ioimport osimport selectimport sockettry:    memoryview  # py 2.7 onlyexcept NameError:    memoryview = lambda x: xif os.name == 'posix':    import sendfile as pysendfile  # requires "pip install pysendfile"else:    pysendfile = None_RETRY = frozenset((errno.EAGAIN, errno.EALREADY, errno.EWOULDBLOCK,                    errno.EINPROGRESS))class _GiveupOnSendfile(Exception):    passif pysendfile is not None:    def _sendfile_use_sendfile(sock, file, offset=0, count=None):        _check_sendfile_params(sock, file, offset, count)        sockno = sock.fileno()        try:            fileno = file.fileno()        except (AttributeError, io.UnsupportedOperation) as err:            raise _GiveupOnSendfile(err)  # not a regular file        try:            fsize = os.fstat(fileno).st_size        except OSError:            raise _GiveupOnSendfile(err)  # not a regular file        if not fsize:            return 0  # empty file        blocksize = fsize if not count else count        timeout = sock.gettimeout()        if timeout == 0:            raise ValueError("non-blocking sockets are not supported")        # poll/select have the advantage of not requiring any        # extra file descriptor, contrarily to epoll/kqueue        # (also, they require a single syscall).        if hasattr(select, 'poll'):            if timeout is not None:                timeout *= 1000            pollster = select.poll()            pollster.register(sockno, select.POLLOUT)            def wait_for_fd():                if pollster.poll(timeout) == []:                    raise socket._socket.timeout('timed out')        else:            # call select() once in order to solicit ValueError in            # case we run out of fds            try:                select.select([], [sockno], [], 0)            except ValueError:                raise _GiveupOnSendfile(err)            def wait_for_fd():                fds = select.select([], [sockno], [], timeout)                if fds == ([], [], []):                    raise socket._socket.timeout('timed out')        total_sent = 0        # localize variable access to minimize overhead        os_sendfile = pysendfile.sendfile        try:            while True:                if timeout:                    wait_for_fd()                if count:                    blocksize = count - total_sent                    if blocksize <= 0:                        break                try:                    sent = os_sendfile(sockno, fileno, offset, blocksize)                except OSError as err:                    if err.errno in _RETRY:                        # Block until the socket is ready to send some                        # data; avoids hogging CPU resources.                        wait_for_fd()                    else:                        if total_sent == 0:                            # We can get here for different reasons, the main                            # one being 'file' is not a regular mmap(2)-like                            # file, in which case we'll fall back on using                            # plain send().                            raise _GiveupOnSendfile(err)                        raise err                else:                    if sent == 0:                        break  # EOF                    offset += sent                    total_sent += sent            return total_sent        finally:            if total_sent > 0 and hasattr(file, 'seek'):                file.seek(offset)else:    def _sendfile_use_sendfile(sock, file, offset=0, count=None):        raise _GiveupOnSendfile(            "sendfile() not available on this platform")def _sendfile_use_send(sock, file, offset=0, count=None):    _check_sendfile_params(sock, file, offset, count)    if sock.gettimeout() == 0:        raise ValueError("non-blocking sockets are not supported")    if offset:        file.seek(offset)    blocksize = min(count, 8192) if count else 8192    total_sent = 0    # localize variable access to minimize overhead    file_read = file.read    sock_send = sock.send    try:        while True:            if count:                blocksize = min(count - total_sent, blocksize)                if blocksize <= 0:                    break            data = memoryview(file_read(blocksize))            if not data:                break  # EOF            while True:                try:                    sent = sock_send(data)                except OSError as err:                    if err.errno in _RETRY:                        continue                    raise                else:                    total_sent += sent                    if sent < len(data):                        data = data[sent:]                    else:                        break        return total_sent    finally:        if total_sent > 0 and hasattr(file, 'seek'):            file.seek(offset + total_sent)def _check_sendfile_params(sock, file, offset, count):    if 'b' not in getattr(file, 'mode', 'b'):        raise ValueError("file should be opened in binary mode")    if not sock.type & socket.SOCK_STREAM:        raise ValueError("only SOCK_STREAM type sockets are supported")    if count is not None:        if not isinstance(count, int):            raise TypeError(                "count must be a positive integer (got %s)" % repr(count))        if count <= 0:            raise ValueError(                "count must be a positive integer (got %s)" % repr(count))def sendfile(sock, file, offset=0, count=None):    """sendfile(sock, file[, offset[, count]]) -> sent    Send a *file* over a connected socket *sock* until EOF is    reached by using high-performance sendfile(2) and return the    total number of bytes which were sent.    *file* must be a regular file object opened in binary mode.    If sendfile() is not available (e.g. Windows) or file is    not a regular file socket.send() will be used instead.    *offset* tells from where to start reading the file.    If specified, *count* is the total number of bytes to transmit    as opposed to sending the file until EOF is reached.    File position is updated on return or also in case of error in    which case file.tell() can be used to figure out the number of    bytes which were sent.    The socket must be of SOCK_STREAM type.    Non-blocking sockets are not supported.    """    try:        return _sendfile_use_sendfile(sock, file, offset, count)    except _GiveupOnSendfile:        return _sendfile_use_send(sock, file, offset, count)

翻译自:

转载地址:http://jiqwd.baihongyu.com/

你可能感兴趣的文章
[经典算法] 排列组合-N元素集合的M元素子集
查看>>
Codeforces 279D The Minimum Number of Variables 状压dp
查看>>
打分排序系统漫谈2 - 点赞量?点赞率?! 置信区间!
查看>>
valgrind检测linux程序内存泄露
查看>>
MSP430(F149)学习笔记——红外接收
查看>>
cef3的各个接口你知道几个
查看>>
Hadoop以及组件介绍
查看>>
1020 Tree Traversals (25)(25 point(s))
查看>>
第一次作业
查看>>
“==”运算符与equals()
查看>>
单工、半双工和全双工的定义
查看>>
Hdu【线段树】基础题.cpp
查看>>
时钟系统
查看>>
BiTree
查看>>
5个基于HTML5的加载动画推荐
查看>>
水平权限漏洞的修复方案
查看>>
静态链接与动态链接的区别
查看>>
Android 关于悬浮窗权限的问题
查看>>
如何使用mysql
查看>>
linux下wc命令详解
查看>>