HTTP

This libary provides support for HTTP 1.0 and 1.1 connections, as well as for headers used in HTTP requests and responses.

1 Request

1.1 Why not just use net/url?

Racket’s net/url functions are very convenient for doing simple requests using HTTP 1.0. Use them when you can.

Sometimes you need to use HTTP 1.1. For example you may need the speed of making multiple requests to a server over one persistent connection. Or you may want to use the Expect: 100-continue request header—so that if a server is going to reject or redirect your put or post request, it can inform you before you send it a potentially large amount of data for the entity.

This library is intended to help you with those kinds of applications. It provides basic building blocks to let you do any kind of HTTP request flow, with help for some of the gory details.

Also it supplies some higher-level support. The call/requests functions are in the style of call/input-url. These take care of some standard setup and teardown, including ensuring the ports are closed even if an exception is raised.

It even provides variations of net/url’s functions like get-impure-port. You can use those when you want to use Expect: 100-continue but do not need to make multiple requests over one connection.

1.2 Connections and requests

When connection-pool-timeout is positive, disconnect will keep the connection open for that number of seconds, and connect may reuse it. Connections that timeout are automatically closed.

When connection-pool-timeout is zero, connect behaves like "raw" connect* and disconnect behaves like "raw" disconnect*.

• scheme, "http" (default if unspecified) or "https"

• host, such as "www.foo.com"

• port as specified by uri. Defaults to 80 if scheme is "http" or 443 if "https".

• When method is "PUT" or "POST", you should pay attention to the returned boolean? value:

  • #t: You should go ahead and transmit the entity (the data for the put or post request), after which you should call flush-output and call purify-port/log-debug to read the response headers.

  • #f: You should not transmit the entity data. Instead just call purify-port/log-debug to read the response headers. (The reason for this is to support the Expect: 100-continue request header, and the case where the server says not to continue because it will fail or redirect the request. This lets you avoid sending a large entity body that won’t be used.)

• For other methods (such as "GET" or "HEAD"), this will always return #t. You should always just go ahead and call purify-port/log-debug to read the response headers.

1.3 Reading response headers

The other functions in this library use log-debug, so using purify-port/log-debug will give a complete/consistent log output.

1.4 Reading response entities

1.4.1 Building blocks

Used by read-entity/bytes.

Examines the Content-Encoding header, if any, and returns a function (input-port? output-port? -> void?) which will copy the bytes and decode them. If there no content encoding specified, or an unsupported encoding, this simply returns copy-port.

1.4.2 Conveniences

How the xexpr is created depends on the Content-Type response header present in h:

• The xexpr is created using read-xml and xml->xexpr if the Content-Type is text/xml or application/xml, or if there is no Content-Type specified and the first bytes of the entity are "<?xml".

• The xexpr is created using read-html-as-xml and xml->xexpr if the Content-Type is text/html.

• If the Content-Type is text/plain then the xexpr is simply bytes?.

• If the Content-Type is application/x-www-form-urlencoded then the xexpr is `(data ,(form-url-encoded->alist (bytes->string/utf-8 bytes))).

• Otherwise the xexpr is `(entity () ,(format "~a" by)).

1.5 call/input-url style

Like call/input-url, these procedures help you make HTTP requests and ensure that the ports are closed even if there is an exception.

Your func function is called with two arguments, the input and output ports. The value you return from func is returned as the value of call/requests.

Tip: Remember to call flush-output after writing data to out, otherwise the server may not get the data and respond, while you wait to read from in.

(define-values (scheme host port) (uri->scheme&host&port uri))

(call/requests

scheme host port

(lambda (in out)

(define-values (path rh)

(uri&headers->path&header uri '("Expect: 100-continue")))

(define tx-data? (start-request in out method path rh "1.0"))

(when tx-data?

(display data out)

(flush-output out)) ;Important!

(define h (purify-port/log-debug in))

(read-entity/bytes in h)))

For a single request, you may prefer to use call/input-request or call/output-request.

Like call/requests, this guarantees the ports will be closed.

Your entity-reader function is called after the response header has been read (the header is passed to you as a string), giving you the ability to read the response entity from the supplied input-port?. For example you may read the response using read-entity/bytes or read-entity/xexpr.

If redirects is non-zero, up to that number of redirects will be followed automatically. Whether the response is a redirect is determined using redirect-uri. If the redirected location can be reached on the existing HTTP connection (as determined by close-connection? and same-connection?), then the existing connection will be used (which is faster than disconnecting and reconnecting.)

Like call/requests, this guarantees the ports will be closed.

The data and data-length arguments depend on whether you want to provide the data as bytes? or as a function that will write the data:

• If data is bytes?, you may set data-length to #f. A Content-Length header will be created for you using (bytes-length data).

• If you pass a writer function for data, then data-length should be the number of bytes that you will write. This allows a Content-Length header to be created for you. Otherwise, if you supply #f for data-length, there will be no Content-Length header (unless you supply your own in heads).

Your entity-reader function is called after the response header has been read (the header is passed to you as a string), giving you the ability to read the response entity from the supplied input-port?. For example you may read the response using read-entity/bytes or read-entity/xexpr.

If redirects is non-zero, up to that number of redirects will be followed automatically. Whether the response is a redirect is determined using redirect-uri. If the redirected location can be reached on the existing HTTP connection (as determined by close-connection? and same-connection?), then the existing connection will be used (which is faster than disconnecting and reconnecting.)

If you supply "1.1" for http-version and the request header Expect: 100-continue, then redirects or failures can be handled much more efficiently. If the server supports Expect: 100-continue, then it can respond with the redirect or failure before you transmit all of the entity data.

1.6 net/url impure port style

These variations add the ability to do HTTP 1.1, and support for you including an Expect: 100-continue request header in heads. This allows a PUT or POST request to fail or redirect, before data is sent to the server.

To make these functions do HTTP 1.1 requests, you should set the http-ver parameter to "1.1".

Because HTTP 1.1 defaults to persistent connections, it would be nice for you to include a Connection: close request header in heads.

1.7 Date string conversion

2 Head

2.1 Supplements to net/head

These stick with net/head’s respresentation of headers as a string?, and add a few functions that are "missing" or would be convenient.

Use case: You want to ensure the header is present, but not disrupt any existing value—for example, Date or Content-Length.

Some web services (hello Amazon) require this as part of "signing" a request for authentication, which means it has to be done exactly right.

2.2 Heads as a dict

Specifically, handle the case of duplicate headers. A real-world example of such duplicate headers is Set-Cookie. Other than association lists, most types of dicts don’t permit duplicate keys, so we can’t store duplicate headers using those. Instead, duplicate headers are stored in the dict under the same key, with the various values separated by dupe-sep.