On Fri, Sep 13, 2024 at 09:49:27AM +0200, Daniel Stenberg wrote:
> On Fri, 13 Sep 2024, Patrick Steinhardt wrote:
>
> > In a nutshell:
>
> Thanks, this is helpful.
>
> > - We then clone a repository from Apache with http.postbuffer=65536,
> > which makes us use a small buffer when POSTing data via curl. We
> > typically use 1MB buffers, and when changing it back to 1MB instead
> > of 65kB the test works just fine.
>
> Is this a git buffer size or is this a value you tell libcurl in an option
> to set a buffer size?
I'm not all that familiar with the "remote-curl.c" remote helper in Git,
so let me try to figure out things as we go.
- The code that sets up the POST buffer is `stateless_connect()`. The
buffer is allocated by ourselves.
- We then execute `post_rpc()` in a loop until we see EOF.
- `post_rpc()` itself is doing all the work to set up the curl handle,
mostly via calls to `curl_easy_setopt()`.
- In there we hit the `large_request` code path. We set up
CURLOPT_READFUNCTION and CURLOPT_SEEKFUNCTION. The callback that
uses our buffer is the one set up via CURLOPT_READFUNCTION, which is
`rpc_out()`.
Whether or not we hit `large_request` depends on out POST buffer size.
We first try to read all the data we want to send into the buffer, and
if it fits we send it out in a single call to curl by setting up
CURLOPT_POSTFIELDS and CURLOPT_POSTFIELDSIZE_LARGE. If it doesn't fit
into the buffer, which is the case for in this testcase, we instead use
the callbacks to write data via curl.
> > I've appended two curl traces, the working one with 1MB buffers and the
> > failing one with 65kB buffers. I hope that helps.
>
> How are you feeding the data to libcurl? (callback or by setting the
> postfields option?) I noticed that in the working case log, the POST
> requests always have a content-length header while the failing case log
> shows that header lacking in the final POST request.
>
> Is that on purpose?
>
> libcurl should still handle it fine, it might just be a clue for me to
> narrow down my search.
I think so. We're using a "chunked" transfer encoding in the
`large_request` case and do not yet know how much data we are about to
send. We'll only figure that out as we go.
Patrick
