Skip to content

feat(receive): add maxsize argument to bound memory usage #463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Tieske wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat(receive): add maxsize argument to bound memory usage #463

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Changelog

## Unreleased

* Add `maxsize` argument to `receive` to bound the memory a single call may accumulate, returning `"oversized"` instead of growing without limit – @Tieske

## [v3.1.0](https://github.com/lunarmodules/luasocket/releases/v3.1.0) — 2022-07-27

* Add support for TCP Defer Accept – @Zash
Expand Down
58 changes: 55 additions & 3 deletions docs/tcp.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -351,7 +351,7 @@ <h2 id="tcp">TCP</h2>
<!-- receive ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -->

<p class="name" id="receive">
client:<b>receive(</b>[pattern [, prefix]]<b>)</b>
client:<b>receive(</b>[pattern [, prefix [, maxsize]]]<b>)</b>
</p>

<p class="description">
Expand Down Expand Up @@ -380,14 +380,23 @@ <h2 id="tcp">TCP</h2>
of any received data before return.
</p>

<p class="parameters">
<tt>Maxsize</tt> is an optional positive integer bounding the number of
payload bytes the call may accumulate, <em>including</em> <tt>prefix</tt>.
Omitted or <tt><b>nil</b></tt> means unlimited.
</p>

<p class="return">
If successful, the method returns the received pattern. In case of error,
the method returns <tt><b>nil</b></tt> followed by an error
message, followed by a (possibly empty) string containing
the partial that was received. The error message can be
the string '<tt>closed</tt>' in case the connection was
closed before the transmission was completed or the string
'<tt>timeout</tt>' in case there was a timeout during the operation.
closed before the transmission was completed, the string
'<tt>timeout</tt>' in case there was a timeout during the operation, or,
when <tt>maxsize</tt> was given, the string '<tt>oversized</tt>' in case
the pattern did not complete within <tt>maxsize</tt> bytes -- in which case
the third return value holds exactly <tt>maxsize</tt> bytes.
</p>

<p class="note">
Expand All @@ -399,6 +408,49 @@ <h2 id="tcp">TCP</h2>
too.
</p>

<p class="note">
<b>Note on <tt>maxsize</tt></b>: passing a <tt>maxsize</tt> that is smaller
than 1, a <tt>prefix</tt> whose length is greater than or equal to
<tt>maxsize</tt>, or, for a numeric <tt>pattern</tt>, a byte count greater
than <tt>maxsize</tt>, all raise a Lua error rather than returning
<tt><b>nil</b></tt> plus a message -- these are caller logic errors, and
they are detected before any byte is read from the socket. To drain and
discard an oversized line while keeping memory bounded and the stream
aligned:
</p>

<pre class="example">
local data, err, part
repeat
data, err, part = client:receive("*l", "", 4096)
until err ~= "oversized"
</pre>

<p class="note">
To instead retry and eventually get the whole thing, carry the partial
forward as <tt>prefix</tt> and grow <tt>maxsize</tt>:
</p>

<pre class="example">
local data, err, part = client:receive("*l", nil, 4096)
if err == "oversized" then
data, err, part = client:receive("*l", part, 65536) -- larger cap, or this raises
end
</pre>

<p class="note">
Retrying with <tt>prefix</tt> set to the previous partial result and an
<em>unchanged</em> <tt>maxsize</tt> raises the length-check error above by
design -- otherwise it would be a zero-progress spin: no I/O, no timeout,
no error, just CPU. A <tt>timeout</tt> partial is always strictly shorter
than <tt>maxsize</tt>, so it is always safe to feed straight back as
<tt>prefix</tt> with the same <tt>maxsize</tt>. Finally, note that
<tt>maxsize</tt> bounds the payload <em>returned</em>, not necessarily the
bytes taken off the wire: for the <tt>*l</tt> pattern the discarded CR
characters and the line terminator mean more bytes may have been consumed
than the returned length suggests.
</p>

<!-- send +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -->

<p class="name" id="send">
Expand Down
108 changes: 88 additions & 20 deletions src/buffer.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,24 @@
* Internal function prototypes
\*=========================================================================*/
static int recvraw(p_buffer buf, size_t wanted, luaL_Buffer *b);
static int recvline(p_buffer buf, luaL_Buffer *b);
static int recvall(p_buffer buf, luaL_Buffer *b);
static int recvline(p_buffer buf, luaL_Buffer *b, size_t budget);
static int recvall(p_buffer buf, luaL_Buffer *b, size_t budget);
static int buffer_get(p_buffer buf, const char **data, size_t *count);
static void buffer_skip(p_buffer buf, size_t count);
static int sendraw(p_buffer buf, const char *data, size_t count, size_t *sent);

/* Internal completion code for buffer_meth_receive. err is not confined to
* the IO_* enum: socket_recv/socket_send (usocket.c/wsocket.c) propagate raw
* platform errors (POSIX errno, Windows WSA codes) straight through, and
* those are always positive, so a positive sentinel here could collide with
* a genuine transport error (e.g. errno 1 == EPERM) and get misreported as
* "oversized". Chosen negative and outside {IO_DONE, IO_TIMEOUT, IO_CLOSED,
* IO_UNKNOWN} (0, -1, -2, -3) so it can never collide with anything err
* legitimately takes.
* MUST be handled before buf->io->error() is called -- it is not a transport
* error. */
#define BUF_OVERSIZED (-1000)

/* min and max macros */
#ifndef MIN
#define MIN(x, y) ((x) < (y) ? x : y)
Expand Down Expand Up @@ -105,8 +117,33 @@ int buffer_meth_send(lua_State *L, p_buffer buf) {
int buffer_meth_receive(lua_State *L, p_buffer buf) {
int err = IO_DONE, top;
luaL_Buffer b;
size_t size;
size_t size, wanted = 0, maxsize = 0;
size_t budget = 0; /* 0 == unlimited */
int numeric = lua_isnumber(L, 2);
const char *part = luaL_optlstring(L, 3, "", &size);

/* ---- validation: must precede timeout_markstart() and any I/O ---- */
if (numeric) {
double n = lua_tonumber(L, 2);
luaL_argcheck(L, n >= 0, 2, "invalid receive pattern");
wanted = (size_t) n;
} else {
const char *p = luaL_optstring(L, 2, "*l");
luaL_argcheck(L, p[0] == '*' && (p[1] == 'l' || p[1] == 'a'),
2, "invalid receive pattern");
}
if (!lua_isnoneornil(L, 4)) {
double m = luaL_checknumber(L, 4);
luaL_argcheck(L, m >= 1, 4, "maxsize must be a positive number");
maxsize = (size_t) m;
luaL_argcheck(L, size < maxsize, 4,
"prefix length >= maxsize (drain with prefix=\"\" or raise maxsize)");
if (numeric)
luaL_argcheck(L, wanted <= maxsize, 4,
"maxsize smaller than requested byte count");
budget = maxsize - size;
}

timeout_markstart(buf->tm);
/* make sure we don't confuse buffer stuff with arguments */
lua_settop(L, 3);
Expand All @@ -116,24 +153,28 @@ int buffer_meth_receive(lua_State *L, p_buffer buf) {
luaL_buffinit(L, &b);
luaL_addlstring(&b, part, size);
/* receive new patterns */
if (!lua_isnumber(L, 2)) {
if (!numeric) {
const char *p= luaL_optstring(L, 2, "*l");
if (p[0] == '*' && p[1] == 'l') err = recvline(buf, &b);
else if (p[0] == '*' && p[1] == 'a') err = recvall(buf, &b);
else luaL_argcheck(L, 0, 2, "invalid receive pattern");
if (p[0] == '*' && p[1] == 'l') err = recvline(buf, &b, budget);
else err = recvall(buf, &b, budget);
/* get a fixed number of bytes (minus what was already partially
* received) */
} else {
double n = lua_tonumber(L, 2);
size_t wanted = (size_t) n;
luaL_argcheck(L, n >= 0, 2, "invalid receive pattern");
if (size == 0 || wanted > size)
err = recvraw(buf, wanted-size, &b);
}
/* check if there was an error */
if (err != IO_DONE) {
/* we can't push anyting in the stack before pushing the
* contents of the buffer. this is the reason for the complication */
/* luaL_pushresult(&b) must come first (its accumulator lives on the
* stack), but the partial it produces belongs in slot 3, not 1 -- so
* both error branches push buffer/error/buffer-copy/nil, then
* lua_replace the nil into slot 1. */
if (err == BUF_OVERSIZED) {
luaL_pushresult(&b);
lua_pushliteral(L, "oversized");
lua_pushvalue(L, -2);
lua_pushnil(L);
lua_replace(L, -4);
} else if (err != IO_DONE) {
luaL_pushresult(&b);
lua_pushstring(L, buf->io->error(buf->io->ctx, err));
lua_pushvalue(L, -2);
Expand Down Expand Up @@ -201,36 +242,61 @@ static int recvraw(p_buffer buf, size_t wanted, luaL_Buffer *b) {

/*-------------------------------------------------------------------------*\
* Reads everything until the connection is closed (buffered)
* budget == 0 means unlimited; otherwise the number of payload bytes still
* allowed. Completion (connection closed) beats the cap: filling the cap
* exactly and then seeing EOF means the whole stream was received.
\*-------------------------------------------------------------------------*/
static int recvall(p_buffer buf, luaL_Buffer *b) {
static int recvall(p_buffer buf, luaL_Buffer *b, size_t budget) {
int err = IO_DONE;
size_t total = 0;
while (err == IO_DONE) {
const char *data; size_t count;
err = buffer_get(buf, &data, &count);
if (budget && count > budget - total) { /* strictly more than fits */
count = budget - total;
luaL_addlstring(b, data, count);
buffer_skip(buf, count);
return BUF_OVERSIZED;
}
total += count;
luaL_addlstring(b, data, count);
buffer_skip(buf, count);
}
if (err == IO_CLOSED) {
if (err == IO_CLOSED) { /* completion beats the cap */
if (total > 0) return IO_DONE;
else return IO_CLOSED;
} else return err;
}
if (budget && total == budget) return BUF_OVERSIZED;
return err;
}

/*-------------------------------------------------------------------------*\
* Reads a line terminated by a CR LF pair or just by a LF. The CR and LF
* are not returned by the function and are discarded from the buffer
* budget == 0 means unlimited; otherwise the number of payload bytes still
* allowed. The cap test sits before consuming a byte, so a line of exactly
* budget payload bytes succeeds while budget+1 reports oversized. A timeout
* or close with the payload exactly at the cap and no terminator yet also
* resolves to oversized, never to timeout/closed.
\*-------------------------------------------------------------------------*/
static int recvline(p_buffer buf, luaL_Buffer *b) {
static int recvline(p_buffer buf, luaL_Buffer *b, size_t budget) {
int err = IO_DONE;
size_t total = 0;
while (err == IO_DONE) {
size_t count, pos; const char *data;
err = buffer_get(buf, &data, &count);
pos = 0;
while (pos < count && data[pos] != '\n') {
/* we ignore all \r's */
if (data[pos] != '\r') luaL_addchar(b, data[pos]);
/* we ignore all \r's -- they are consumed but never counted */
if (data[pos] != '\r') {
if (budget && total == budget) {
/* leave the offending byte in the buffer for the next call */
buffer_skip(buf, pos);
return BUF_OVERSIZED;
}
luaL_addchar(b, data[pos]);
total++;
}
pos++;
}
if (pos < count) { /* found '\n' */
Expand All @@ -239,7 +305,9 @@ static int recvline(p_buffer buf, luaL_Buffer *b) {
} else /* reached the end of the buffer */
buffer_skip(buf, pos);
}
return err;
if (err == IO_DONE) return IO_DONE; /* '\n' found: success, regardless of total */
if (budget && total == budget) return BUF_OVERSIZED; /* stalled/closed exactly at the cap: I1 */
return err; /* real timeout/closed, below the cap */
}

/*-------------------------------------------------------------------------*\
Expand Down
Loading
Loading