# 2021-Oct-7: Running webservers behind a transparent reverse proxy **URL:** \ **Category:** BLUG\ **Created:** [September 28, 2021, 6:24am UTC](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318 "2021-09-28T06:24:04Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![logan](https://avatars.discourse-cdn.com/v4/letter/l/278dde/32.png) [@logan](https://discuss.lfnw.org/u/logan)\ **Post date:** [September 28, 2021, 6:24am UTC](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318/1 "2021-09-28T06:24:04Z")
Looks like the most requested choice for a topic for October 7 is setting up webservers behind transparent reverse proxies. This is partially a sequel to my recorded LFNW2020 presentation on implementing transparent reverse proxies, but if you haven’t seen that, don’t worry, I’ll give a brief recap. I think there will be enough time to go over how to run about 4 different webservers, which should be sufficient to apply the lessons to any webserver. Given there are more options for “other webservers” than there is time to cover, if you have a preference, please pick here. I’ll put the presentation together in priority order. _Poll ([view on site](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318/1))_ ---
**Author:** ![meonkeys](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.lfnw.org/meonkeys/32/94_2.png) [@meonkeys](https://discuss.lfnw.org/u/meonkeys)\ **Post date:** [October 1, 2021, 8:37pm UTC](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318/2 "2021-10-01T20:37:09Z")
Related thread: [Reverse proxies](https://discuss.lfnw.org/t/reverse-proxies/320) ---
**Author:** ![logan](https://avatars.discourse-cdn.com/v4/letter/l/278dde/32.png) [@logan](https://discuss.lfnw.org/u/logan)\ **Post date:** [October 2, 2021, 1:32am UTC](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318/3 "2021-10-02T01:32:36Z")
# Terminology As mentioned in the first post, this follows my presentation from a year ago on transparent reverse proxies. While I will begin this presentation by covering the basics again, there are several ambiguous and confusing terms involved. So this post will attempt to provide some clarity of terminology, and a written version of the background information. ## Open Files First set of confusing terms is `file handle` vs `file descriptor` vs `file description`. All three refer to an open `file`, which can be a logical file on a filesystem, a socket, a device, _everything_ is a `file`. ### File Handle File Handles are a userspace structure for referencing a file (open or not). The key here is **userspace**. As such, they are process-specific. ### File Descriptor File Descriptors are still process-specific, but they are the kernelspace counterpart to the File Handle. These are presented to userspace as _just_ an integer. File Descriptors can be copied, including between processes. The duplicate File Descriptor will be assigned a new integer. Multiple File Descriptors can point to the same File Description. ### File Description File Descriptions are the **internal** kernelspace datastructure for an open file. Each open file has _exactly one_ File Description. File Descriptions hold metadata about the opened file (such as read/write mode, pointer offset, underlying location, and so on). ## Topology Load-balanced or Virtual Hosted services are another area of confusing terminology. There is less standardization of terms here, but for the purposes of this presentation, I will use the following. ### Load Balancer A load balancer is a service that splits incoming connections between multiple service providers. This can be at the DNS level, at the application level, or at the edge server. Application and DNS level load balancing split the connections at Layer-4 (TCP/UDP), so the main connection (HTTP GET requests, et cetera) are routed to different logical service providers. When load balancing at the edge server level, _all_ the traffic hits the edge server. ### Edge Server The logical server, after resolving domain names, which accepts incoming connections. When multiple logical servers are involved, often _only_ the edge server listens on the public internet. Note that the Edge Server can run on the same logical machine (physical or virtual) as other logical servers. ### Upstream Server A logical server which handles a subset of incoming connections or requests. These are hidden from the “outside world” behind the edge server. A Reverse Proxy can intelligently select between upstream servers. ### Reverse Proxy A service running on the Edge Server which examines incoming connections and relays or routes them to different upstream servers based on the _name_ of the server the client requested. ### Transparent Reverse Proxy A Reverse Proxy which uses the SSL `Server Name Indication` Field or packet inspection to determine the _name_ of the requested Upstream Server _without_ affecting the state of the connection. The connection is then passed (usually by duplicating the File Descriptor for the socket) to the Upstream Server. At the application level, this process is **transparent**. ## Other Terms ### Server Name Indication (SNI) The name field of a TLS header, which indicates in plain-text the desired logical server name. ### Named Virtual Host A Named Virtual Host is a logical website (or other HTTP service) identified by a unique hostname. This is provided by the SNI field of a TLS header, or as the `HOST: ` line of the HTTP headers of a request. # Notes 1. In the description of Topology, HTTP is mentioned _only in passing_. Reverse Proxies are used for a wide variety of application-level protocols, and one of the major advantages of Transparent Reverse Proxies is they are application-protocol agnostic (provided the initial handshake has some way for the Reverse Proxy server to determine which Upstream Server to use). 2. Transparent Reverse Proxies are not always superior to HTTP Reverse Proxies. HTTP Reverse Proxies can cache results, are often supported out of the box, and often produce equivalent results. This is _not_ a dig at them, or at those who write or use them. 3. Load balancing (but not Named Virtual Hosting) can also be done by the kernel (starting in Linux 4.something). I’m going to ignore that for expediency. 4. I am assuming all logical services are running on the same logical computer on the same physical computer. It _is_ possible to use a Transparent Reverse Proxy when that is not true, but demonstrating it is more difficult, the setup is often trivialized, and some of the performance benefits of a Transparent Reverse Proxy are lost. ---
**Author:** ![meonkeys](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.lfnw.org/meonkeys/32/94_2.png) [@meonkeys](https://discuss.lfnw.org/u/meonkeys)\ **Post date:** [September 29, 2021, 9:36pm UTC](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318/4 "2021-09-29T21:36:38Z")
This is related to [2021-Oct-7: Running webservers behind a transparent reverse proxy](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318) . I’m unable to comment on that poll. I’ve been using traefik proxy in front of docker containers for running various web services like Nextcloud, Jellyfin, Wallabag, Pi-hole. Traefik proxy is interesting because it dynamically discovers new web services when you start them up (based on labels associated with containers) I really prefer this setup and I don’t miss manual config with apache/haproxy/nginx. Anyway, I just wanted to put a plug in for this talk including traefik or something like it. I gave a related workshop at LibrePlanet this year, see [Adam Monsen / LibrePlanet 2021 Private Cloud Workshop · GitLab](https://gitlab.com/meonkeys/lp2021-private-cloud-workshop) ---
**Author:** ![logan](https://avatars.discourse-cdn.com/v4/letter/l/278dde/32.png) [@logan](https://discuss.lfnw.org/u/logan)\ **Post date:** [September 30, 2021, 12:19am UTC](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318/5 "2021-09-30T00:19:25Z")
Odd that you cannot reply there… Are you a member of the BLUG group? If not, you’ll need to join before you can post on BLUG threads. I believe there’s an invite link here somewhere. Traefik looks like a neat project. I have considered trying to do something similar, but UI programming is not something I particularly enjoy. It is worth noting that Traefik is _not_ transparent, it must pass connection metadata (remote IP/port, and so on) out-of-band. It likely will have scaling issues once you get to where you need to be using `sendfile(2)`, but for a dead-simple-to-use reverse proxy, it has promise. You mention docker, I should probably include a bit on how to use dockerized services behind the proxy too. I don’t normally use docker, it tries to be fancier than a container manager should, which leads to many of its users not understanding what it’s doing, but I think I can make an exception here. (Normally I use firejail to manage starting new kernel namespaces). ---
**Author:** ![bear454](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.lfnw.org/bear454/32/5_2.png) [@bear454](https://discuss.lfnw.org/u/bear454)\ **Post date:** [September 30, 2021, 4:30pm UTC](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318/6 "2021-09-30T16:30:50Z")
@meonkeys - would you like to join the [BLUG group](https://discuss.lfnw.org/g/blug)? ---
**Author:** ![meonkeys](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.lfnw.org/meonkeys/32/94_2.png) [@meonkeys](https://discuss.lfnw.org/u/meonkeys)\ **Post date:** [October 1, 2021, 8:34pm UTC](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318/7 "2021-10-01T20:34:48Z")
> [@logan](#): > > Are you a member of the BLUG group? Maybe not. I’ll reply to @bear454 below. > [@logan](#): > > UI programming is not something I particularly enjoy UI programming? > [@logan](#): > > It is worth noting that Traefik is _not_ transparent Oh huh, and this is different than using maybe HAProxy or nginx / apache as a reverse proxy? I’d love to learn more about this. To be clear, I’m not using Traefik for performance reasons, more for convenience–so I can easily serve many container-based web services with different domain names from the same IP address (and, because I’m cheap/lazy, the same physical server). > [@logan](#): > > Normally I use firejail to manage starting new kernel namespaces Cool, I’d like to learn more about that too. I’ve heard of a several different ways to containerize and just went with Docker because it seemed like the popular option (and there are tons of prebuilt images available). ---
**Author:** ![meonkeys](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.lfnw.org/meonkeys/32/94_2.png) [@meonkeys](https://discuss.lfnw.org/u/meonkeys)\ **Post date:** [October 1, 2021, 8:36pm UTC](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318/8 "2021-10-01T20:36:08Z")
> [@bear454](#): > > @meonkeys - would you like to join the [BLUG group](https://discuss.lfnw.org/g/blug)? Sure! I think I got it, I just clicked Join on [Blug members - LinuxFest Northwest](https://discuss.lfnw.org/g/blug) . Ah yep, that was why I couldn’t reply to the poll. Thanks @logan and @bear454 ! ---
**Author:** ![logan](https://avatars.discourse-cdn.com/v4/letter/l/278dde/32.png) [@logan](https://discuss.lfnw.org/u/logan)\ **Post date:** [October 1, 2021, 11:10pm UTC](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318/9 "2021-10-01T23:10:13Z")
> [@meonkeys](#): > > UI programming? One of the most compelling features of Traefik is its own web user interface for monitoring things. While I appreciate good UIs, and can make them, given sufficient time, I lack the talent to make them quickly. > [@meonkeys](#): > > Cool, I’d like to learn more about that too. I’ve heard of a several different ways to containerize and just went with Docker because it seemed like the popular option (and there are tons of prebuilt images available). I am something for a “black sheep”. I’ve been doing asynchronous programming since the days when Java was **the** language for enterprise software and multiplexing connections meant just `fork()` a few extra times. Docker is popular because Docker is popular. Docker is fairly terrible from a security perspective (when’s the last time you looked inside a docker container to see what it’s really doing?), but it reduces the friction on deploying software to “the cloud” (computers you don’t control), so it is popular. _Containerization_ is good, and is likely here to stay. Docker will likely be replaced eventually (or morph to something different). > [@meonkeys](#): > > and this is different than using maybe HAProxy or nginx / apache as a reverse proxy Yes. The key word is _transparent_. HAProxy, nginx, and similar are HTTP proxies, using the HTTP proxy protocol. This is a Layer-5 proxy protocol, and has side-effects at layer 5. For instance, the SSL certificates must be handled by the proxy itself. The proxied server cannot directly see the remote IP address (that must be included in the HTTP-Proxy metadata instead). As a related issue, socket options on the outgoing connection cannot be changed by the proxied server. The transparent remote proxy I use is a Layer-4 proxy protocol. It slightly complicates establishing the layer-4 (TCP) link, but is perfectly transparent once the connection is established. The SSL certs are handled by the proxied server, remote client information is available through the usual syscalls, and the data generally need never pass through the proxy server process once the initial connection is established (if the proxied server is physically separate, having the proxy server pass the data is one option). I realize this probably just raises a few more questions. I’m working on a post for the other thread with some terminology and background information. Essentially the material I covered last year for LFNW2020. ---
**Author:** ![bear454](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.lfnw.org/bear454/32/5_2.png) [@bear454](https://discuss.lfnw.org/u/bear454)\ **Post date:** [October 4, 2021, 10:40pm UTC](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318/10 "2021-10-04T22:40:45Z")
[https://www.blug.org/event/virtualblug-meeting-10-7-logan-perkins-running-webservers-behind-a-transparent-reverse-proxy/](https://www.blug.org/event/virtualblug-meeting-10-7-logan-perkins-running-webservers-behind-a-transparent-reverse-proxy/) > **[Logan Perkins – Running webservers behind a transparent reverse proxy](https://www.meetup.com/Bellingham-Linux-User-Group-BLUG/events/281215450/?isFirstPublish=true)** > > Thu, Oct 7, 2021, 7:00 PM: Meeting Includes…BLUG/LFNW & Linux News, Q&A, and a presentation by Logan Perkins on running webservers behind a transparent reverse proxy.Logan’s Presentation:This is parti ---
**Author:** ![logan](https://avatars.discourse-cdn.com/v4/letter/l/278dde/32.png) [@logan](https://discuss.lfnw.org/u/logan)\ **Post date:** [October 8, 2021, 3:21am UTC](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318/11 "2021-10-08T03:21:40Z")
As these things typically go, it is at the end that the right choice is apparent. I just went to put the code on github, and found it’s already there. Complete with some pieces that would have been useful for the presentation. Anyway, the sources are [here](https://github.com/perkinslr/PyReverseProxy), complete with a patch to let nginx work, a working LD\_PRELOAD library, and a shim to support TCP upstream hosts. The slides are available [here](http://slides.lfnw.lp-programming.com/) ---
**Author:** ![bear454](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.lfnw.org/bear454/32/5_2.png) [@bear454](https://discuss.lfnw.org/u/bear454)\ **Post date:** [October 9, 2021, 3:15am UTC](https://discuss.lfnw.org/t/2021-oct-7-running-webservers-behind-a-transparent-reverse-proxy/318/12 "2021-10-09T03:15:52Z")
Thanks for a great presentation, @logan !