Site-to-Site VPN: Manual IPSec

Despite residing in the heart of Silicon Valley here in California, I have exactly one ISP offering speeds greater than 25 Mbps - Xfinity. For the last few years, I had been running shop with a 100 Mbps down / 5 Mbps up cable plan (which Xfinity has graciously upgraded recently to 400 Mbps / 20 Mbps). As mentioned in the previous section, I do run a relatively heavy network, thanks to the lab infrastructure in place for evaluating systems and storage devices in addition to serving the needs of a typical family of four.

Back in India, there is a lot more competition among ISPs to serve consumers. There are multiple fiber-to-the-home (FTTH) service providers with a wide variety of symmetric speed options. For the new home, my parents opted to go with Airtel's symmetric 100 Mbps plan (costing approximately US $12 inclusive of taxes). Their network demands were not too heavy - a smart TV, couple of mobile phones, a desktop, and a notebook - with only a couple of clients being simultaneously active.

While I had multiple VLANs at home, with a specific subnet for guests isolated from the rest (automatically created when a guest Wi-Fi network is configured), the configuration for the UniFi Dream Machine had only one primary network and another guest network.

During the initial configuration of the UniFi Dream Machine, Airtel had provided a public-facing WAN IP for the UDM to pick up. There was a necessity to call up the ISP to put their gateway (FTTH terminal / Wi-Fi AP) in bridge mode, but that is outside the scope of this article. The UDM was configured with the appropriate credentials to authenticate over PPPoE and pick up the WAN IP via the bridge connection. With the public-facing WAN IPs of both sites at my disposal, configuring the site-to-site VPN was a breeze.

On the US side, activating the site-to-site VPN network creation prompted for the required details - network name, VPN protocol, the pre-shared key, and the server address. The USG Pro 4 supports manual IPSec and OpenVPN, with the former capable of getting hardware-accelerated. A random pre-shared key can be generated and copied over. The server address was set to the WAN IP of the USG Pro 4. Under the 'Remote Device Configurations' section, it was required to specify the remote subnets desired to be made visible locally, along with the WAN IP of the UDM.

Similar information was entered in the UDM, with the pre-shared key generated on the USG Pro 4 placed in the PSK field. Ubiquiti has slightly reworked the UI in the UDM's Network application (Network 7.2.94 vs. 7.1.68 in the USG Pro 4), with the 'server address' tag being replaced by 'UniFi Gateway IP', making things slightly more user friendly. The remote device configuration section is filled with the required subnets from the US side, along with the USG Pro 4's WAN IP.

Upon adding the new VPN network on both ends, there was a handshake between the two devices and I was able to access the devices in the Indian network from the US and vice-versa. The web UI configuration transparently handles all the port openings required on either end.

The site-to-site VPN setup was further augmented with an old NUC connected to the UDM. The PC was set up to run a squid proxy server. In the US, an Android tablet was dedicated to accessing the Indian OTT services and set up to access the Internet using the NUC as a proxy. This configuration worked fine for more than a month. There were a few interruptions due to power failures and DHCP WAN IP changes on the UDM side. The latter had to be reflected in the site-to-site VPN setup and resulted in some downtime, but was not a cause for major concern.

I was fairly happy with the setup and would have left it as-is, if not for waking up one fine morning and finding the VPN link down. Expecting the customary WAN IP change, I fired up the UniFi Network app and tried to figure out the new IP assigned to the UDM.

Using the 100.107.xx.xx IP in the site-to-site setup was not helpful in re-activating the VPN link. Given my lack of formal network administration skills, this ended up being my introduction to the nitty-gritty details of carrier-grade network-address translation (CGNAT) - a term I had only encountered in passing earlier.

Introduction The CGNAT Spanner in the Works
Comments Locked


View All Comments

  • bradh352 - Wednesday, December 21, 2022 - link

    First thing that comes to mind is why you didn't attempt to use ipv6 addresses to create the ipsec vpn? I know comcast/xfinity supports ipv6, and I'd have to imagine anyone deploying CGNAT for ipv4 is providing a public ipv6 address, thus negating any of the issues described.

    Typically these ipsec vpn sessions are in tunnel mode which means they can transport both ipv4 and ipv6 packets, even if the public ips being used are only ipv4.

    Maybe ubiquiti doesn't support this in their UI for some reason? The underlying system should be capable of this though (strongswan afaik).
  • edzieba - Wednesday, December 21, 2022 - link

    "I'd have to imagine anyone deploying CGNAT for ipv4 is providing a public ipv6 address"

    Sadly, such a sane and customer-oriented approach remains firmly in your imagination for the vast majority of ISP CGNAT deployments. Most commonly, IPv6 will just not be available at all.
  • ganeshts - Wednesday, December 21, 2022 - link

    Airtel does provide an IPv6 address with their CGNAT configuration. Sadly, there is no IPv6 support on the Comcast front over here in the US, and Ubiquiti doesn't support IPv6 in their VPN configuration either (at least from the web UI perspective).
  • ViRGE - Wednesday, December 21, 2022 - link

    "Sadly, there is no IPv6 support on the Comcast front over here in the US"

    Huh? Comcast was one of the very first major US ISPs to implement IPv6. They've been running a full dual stack implementation for nearly a decade now.
  • - Wednesday, December 21, 2022 - link

    I was amused to see that the IPv6 post I was contemplating was ninja-ed before birth by the very first post.

    I suspect the reason Ganesh isn't seeing IPv6 is his "ancient cable modem". Very likely it's not DOCSIS 3.0 or later and doesn't do IPv6. The DOCSIS 3.0 standard was released in 2006, 3.1 in 2013. Upgrade, already!
  • - Thursday, December 22, 2022 - link

    Also: Comcast was one of the major leaders and instigators of "World IPv6 Day". That was back in January 2012. As I recall, somewhere around 50% of their customers were IPv6-enabled then.

    Why? They were running out of IPv4 addresses to give to customers, which also explains why Ganesh is seeing CGNAT now.

    My ISP had (and probably still has) IPv4 addresses in reserve. They haven't enabled IPv6 for consumer internet service yet.
  • dersteffeneilers - Saturday, December 24, 2022 - link

    with my ISP over in Germany, you can use both IPv4 with CGNAT and IPv6, but you only get an IPv6 address if you already have an IPv4 one. Ridiculous, I can't imagine a technical reason for that.
  • Leeea - Thursday, December 22, 2022 - link

    Nobody in their right mind uses ipv6 unless they absolutely have to.

    They really should come up with another standard that is less ideologically pure and way more practical.
  • ballsystemlord - Thursday, December 22, 2022 - link

    Could you expand on that a bit more Leeea?
    It's unclear to me why an IP addressing scheme would be impractical as a result of ideological purity.
  • jack21159 - Thursday, December 22, 2022 - link

    IPv6 was made for ultra-nerd and it's difficult to understand.

    I mean, IPv4 still is a learning curve, but at least it's easier to understand. Most people don't know how to segment a network (/23 , /24) or do custom routes, but that's fine you can use it even if you don't understand all the concept.

    IPv6 by contrast, no. a course is needed to understand that.
    they could have just added a few Bytes to the standard 4 Bytes scheme (ie for exemple) But nooo let use hexadecimal, something than only computers and ultra nerds understand !

Log in

Don't have an account? Sign up now