Google Fiber - Troubleshooting Random Disconnects and Slow Speeds

Shortly after Christmas I noticed something odd, my downloads were slower than usual. The speed of my downloads was very consistent but slow at about 95Mbps or about 11.8MB/s.

So I started troubleshooting the issue.
Step 1: I went to http://speedtest.googlefiber.net/ and ran the test on a wired computer. The results were exactly 95Mbps down and up. There's something special about that 95Mbps speed, it's almost exactly the maximum speed that's possible over 100Mb/s Ethernet (extra ~5Mbps is overhead.)

Step 2: So we have a link somewhere between my wired test computer and Google Fiber that's auto-negotiated to 100Mb/s instead of 1000Mb/s. The question is where is the bad link? Let start with the Fiber Jack and work our way back to my test computer. I go to My Fiber and select the Fiber Jack to see what speeds have been auto-negotiated.

That's strange, the Ethernet connection between the Fiber Jack and my router is only 100Mb/s. That's probably the issue!
At this point, most people can resolve this problem by simply replacing the cable between the Fiber Jack and the Network Box with a new Cat5e or better network cable.




Unfortunately for me, my Fiber Jack and router are quite a distance away from each other and I only have one cable that runs between them. I'll have to figure out where the fault is in the cable and fix it.

Step 3: I disconnected and reconnected the cable hoping it would come back up at 1000Mb/s...and it does! For a little while at least. After a few days I notice websites are slow to load and programs are complaining about my Internet connection dropping. Since I have a 3rd party router (Ubiquiti EdgeRouter) I SSHed to it to see if I can get any more information about what's happening.

I see where the link to my Fiber Jack negotiated to 100Mb/s:
Dec 24 12:42:02 ERL-CT kernel: eth1: Link down
Dec 24 12:42:09 ERL-CT kernel: eth1: 100 Mbps Full duplex, port 1

I also see the link is now repeatedly bouncing up and down rapidly every few hours:
Dec 27 14:17:49 ERL-CT kernel: eth1: Link down
Dec 27 14:17:51 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 14:17:53 ERL-CT kernel: eth1: Link down
Dec 27 14:17:55 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 14:18:01 ERL-CT kernel: eth1: Link down
Dec 27 14:18:02 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 16:15:26 ERL-CT kernel: eth1: Link down
Dec 27 16:15:28 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 16:21:18 ERL-CT kernel: eth1: Link down
Dec 27 16:21:20 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 16:24:32 ERL-CT kernel: eth1: Link down
Dec 27 16:24:38 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 18:58:12 ERL-CT kernel: eth1: Link down
Dec 27 18:58:14 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 18:58:18 ERL-CT kernel: eth1: Link down
Dec 27 18:58:19 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 18:58:21 ERL-CT kernel: eth1: Link down
Dec 27 18:58:23 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 18:58:27 ERL-CT kernel: eth1: Link down
Dec 27 18:58:32 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1

Step 4: There are several possible culprits and I need to start eliminating some of them to find the root cause. The first thing I think of is that maybe my PoE injector that I'm running the Fiber Jack on has started to fail. To eliminate this possibility I connect the Fiber Jack to power through the wall adapter that came with it. After a day the connection to the Fiber Jack starts bouncing again. So that's one possibility eliminated. (I also learned the Fiber Jack can switch between PoE power and the wall adapter without interruption and vice versa.)

Step 5: The cable between my Fiber Jack and router is made of several segments. Maybe one of the connections has been damaged/corroded/etc. I start at the Fiber Jack, disconnect the Ethernet cable make sure the connectors are clean, reconnect the cable, gently pull on the cable after it's connected to make sure it's making good contact. I then go outside where the cable connects through a keystone jack to the long run across my house and do the same steps. Everything looks good until I get to the other side of that long run where I have an Ethernet surge protector (since the cable passes outside.) 

When I disconnect the cables from the surge protector I see some dust on the connectors. I reconnect the cables and gently pull on them, one side at a time and each time I do the link bounces between the Fiber Jack and my router. I blow out the connectors with compressed air, reconnect the cables and gently pull on the cables one at a time again. The link no longer bounces and the connections are solid. Problem solved!

Since I've had this experience, I went back and read some of the Google Fiber Help Forum posts about slow speeds and random drops and I'm convinced at least a few of them are bad connections/cables. Maybe this post can help someone else solve this problem in hours instead of days like it took me. :)

Google Fiber - Large increase in IPv4 address space

Google Fiber is gearing up to provide access for many more users.

Before October 6th, 2015 Google Fiber owned eight IPv4 address blocks totaling around ~147 thousand IPv4 addresses. On October 6th, 2015 Google Fiber acquired the /11 IPv4 address block 136.32.0.0/11. Google Fiber first began announcing this new address block to the global BGP table on November 24th, 2015.
This /11 address block adds an additional ~2.1 million IPv4 addresses.

Google Fiber - EdgeRouter Remote VPN Setup

Google Fiber's Network Box currently doesn't have support for remote access VPNs but the Ubiquiti EdgeRouter does. I'll walk through setting up remote access via L2TP over IPsec.

The firewall rule changes can be done through the GUI but everything else must be done through the CLI or the config tree.

Enable hardware offload for IPsec:

set system offload ipsec enable

Reorder a rule so there's space to add two new rules:

edit firewall
edit name WAN_LOCAL
rename rule 3 to rule 6
top

Add firewall rules for L2TP:

set firewall name WAN_LOCAL rule 4 action accept
set firewall name WAN_LOCAL rule 4 description 'Allow L2TP'
set firewall name WAN_LOCAL rule 4 destination port 500,1701,4500
set firewall name WAN_LOCAL rule 4 log disable
set firewall name WAN_LOCAL rule 4 protocol udp
set firewall name WAN_LOCAL rule 5 action accept
set firewall name WAN_LOCAL rule 5 description 'Allow ESP'
set firewall name WAN_LOCAL rule 5 log disable
set firewall name WAN_LOCAL rule 5 protocol esp

Setup IPsec:

set vpn ipsec auto-firewall-nat-exclude disable
set vpn ipsec ipsec-interfaces interface eth1.2
set vpn ipsec nat-networks allowed-network 10.0.0.0/8
set vpn ipsec nat-networks allowed-network 172.16.0.0/12
set vpn ipsec nat-networks allowed-network 192.168.0.0/16
set vpn ipsec nat-traversal enable

Setup L2TP:
#Change the username and password

set vpn l2tp remote-access authentication local-users username testuser password 'testpassword'
set vpn l2tp remote-access authentication mode local
set vpn l2tp remote-access client-ip-pool start 192.168.100.101
set vpn l2tp remote-access client-ip-pool stop 192.168.100.110
set vpn l2tp remote-access dhcp-interface eth1.2
set vpn l2tp remote-access dns-servers server-1 8.8.8.8
set vpn l2tp remote-access dns-servers server-2 8.8.4.4
set vpn l2tp remote-access ipsec-settings authentication mode pre-shared-secret

#Change the L2TP Shared Secret

set vpn l2tp remote-access ipsec-settings authentication pre-shared-secret testsecret
set vpn l2tp remote-access ipsec-settings ike-lifetime 3600
set vpn l2tp remote-access mtu 1492

Commit and Save:

commit
save

Additional Resources:
EdgeMax - L2TP Server Setup
Android Client Setup

Google Fiber - Dynamic DNS Setup on an EdgeRouter

Google Fiber's Network Box supports several dynamic DNS providers but I wanted to use one they don't currently support called afraid.org. Now that I have a router that can supports almost any dynamic DNS provider I can configure it!

Here's how to set it up on a Ubiquiti EdgeRouter.

From WebUI:

Go to the Service Tab
Then go to the DNS Tab
Press the "Add DDNS Interface" button
Select interface eth1.2 (assuming eth1.2 is your WAN interface)
Select afraid for the service
Set your hostname, login and password
Set the server to freedns.afraid.org
Press the Apply button



From CLI:

configure
set service dns dynamic interface eth1.2 service afraid host-name mydomain.mooo.com
set service dns dynamic interface eth1.2 service afraid login dyndns
set service dns dynamic interface eth1.2 service afraid password 123456
set service dns dynamic interface eth1.2 service afraid server freedns.afraid.org
commit
save

Testing the new configuration

From CLI:

Run this command to force an update:

update dns dynamic interface eth1.2

Run this command to see the status information for dynamic DNS.

show dns dynamic status

interface    : eth1.2
ip address   : 1.2.3.4
host-name    : mydomain.mooo.com
last update  : Sat Oct 17 10:53:36 2015
update-status: good

Notes:
If you have any uppercase letters in your afriad.org username, convert them to lowercase when setting your username on the EdgeRouter. (The afraid.org site is converting your username to lowercase when you login and the API calls the dynamic DNS client tries to make will fail if your username doesn't match exactly.)

Additional Resources:
Guide from Ubiquiti's Help Center on Setting up Dynamic DNS

Google Fiber - EdgeRouter IGMP Proxy for TV Service

In my first blog post about switching to the Uqituiti EdgeRouter I noted that users with TV service would have issues running Atlantisman's TV script as is after running the modified script I posted.

I've modified the TV script he created so it will work with the IPv6 script I posted before.
Disclaimer: I don't have TV service, so I have no way to verify that this script actually works.

Modified TV Script

If you have the first gen Google Fiber equipment, attach your Storage Box (GFMS100) to your LAN and your TV boxes should work again.

If you have the second gen Google Fiber equipment things are more complicated because you have a combined Network and Storage Box in the Network+ box (GFRG200, GFRG210).

There are two options to get TV working with the second gen Network+ boxes:

First Option - 

• Plug a switch into the Fiber Jack and connect both the EdgeRouter and Network+ box to the switch. 

There are a couple problems with this option though:
First problem, by connecting two routers to the Fiber Jack you'll be assigned two public IPv4 addresses. While Google Fiber currently allows this, in the future it will probably be restricted to only one IPv4 address and this option will no longer work.

Second problem, Google Fiber already restricts IPv6 address assignment and prefix delegation to one MAC address or DUID (DHCP Unique Identifier) per Fiber Jack. What this means is the Network+ Box will almost always win the race to get an IPv6 address and prefix before the EdgeRouter and the EdgeRouter will be stuck with only IPv4 connectivity and the CPU at 100% as the DHCPv6-PD process tries in vain to get an IPv6 address and prefix. By powering on the EdgeRouter first waiting a few minutes then powering on the Network+ box you can get around this issue.

Second Option - 

• Login to fiber.google.com and in the Advanced drop down for your network box change the "Router LAN IPv4 address", "DHCP start address", and "DHCP end address" to be on a different network than the LAN side of your EdgeRouter. (e.g. Change 192.168.1.1 to 192.168.3.1, etc)
• Plug the Network+ box into the LAN side of the EdgeRouter

Advantages of this option:

With this option you won't have to rely on a second IPv4 address that may go away or need another switch or vlan connected to the Fiber Jack. 

You can continue to use the wireless from the Network+ until your real access point arrives. :)

Current issues with this option:

Devices plugged into the Network+ won't receive an IPv6 address.

Devices plugged into the Network+ box will be going through a double IPv4 NAT.

Other Important Notes:

If you have a switch connected to the LAN side of the EdgeRouter and plan to connect your own wireless access point later, make sure your switch supports IGMP snooping and that you have it enabled/configured. The TV channels are multicast to the Network+ box/Storage Box and a switch without IGMP snooping will flood the traffic out all ports on the same vlan. (All ports if it's an unmanaged switch.) You can also put your WAP on the third EdgeRouter port and leave IGMP proxy disabled on it to get around this issue.

Google Fiber - Hardware Change for Basic Tier

Google Fiber's Basic Tier is their 5 Mb/s down, 1 Mb/s up tier that's free for seven years after installation.

Currently customers signing up for the Basic Tier have been getting the first generation Network Boxes (GFRG100 or GFRG110) from the stockpile that was originally manufactured. Sometime in the near future Basic Tier customers will be getting a different Network Box, the Mini Network Box (GFMN100).

Unofficial Specs:
One WAN port (With PoE to power a newer Fiber Jack GFLT110)
One LAN port
2.4 Ghz 2x2 WiFi (802.11n)

Google Fiber - Next Steps with the EdgeRouter

Ubiquiti's EdgeRouter is amazingly versatile for the price point. There's also a lot of documentation on their support and help center that covers almost anything you would want to configure.

Here's the link to the main documentation page:
EdgeMAX - Ubiquiti Networks Support and Help Center

I've put together a list of stuff I want to configure and I'll be creating a blog post for each one as I configure them.

IGMP Proxy (For TV service)

Port Forwarding

Dynamic DNS

Remote Access VPN

If you want to play with the latest and greatest features on your EdgeRouter, you can also sign up to join the version 1.8 alpha/beta testing group.