Google Fiber - Troubleshooting Random Disconnects and Slow Speeds

Shortly after Christmas I noticed something odd, my downloads were slower than usual. The speed of my downloads was very consistent but slow at about 95Mbps or about 11.8MB/s.

So I started troubleshooting the issue.
Step 1: I went to http://speedtest.googlefiber.net/ and ran the test on a wired computer. The results were exactly 95Mbps down and up. There's something special about that 95Mbps speed, it's almost exactly the maximum speed that's possible over 100Mb/s Ethernet (extra ~5Mbps is overhead.)

Step 2: So we have a link somewhere between my wired test computer and Google Fiber that's auto-negotiated to 100Mb/s instead of 1000Mb/s. The question is where is the bad link? Let start with the Fiber Jack and work our way back to my test computer. I go to My Fiber and select the Fiber Jack to see what speeds have been auto-negotiated.

That's strange, the Ethernet connection between the Fiber Jack and my router is only 100Mb/s. That's probably the issue!
At this point, most people can resolve this problem by simply replacing the cable between the Fiber Jack and the Network Box with a new Cat5e or better network cable.




Unfortunately for me, my Fiber Jack and router are quite a distance away from each other and I only have one cable that runs between them. I'll have to figure out where the fault is in the cable and fix it.

Step 3: I disconnected and reconnected the cable hoping it would come back up at 1000Mb/s...and it does! For a little while at least. After a few days I notice websites are slow to load and programs are complaining about my Internet connection dropping. Since I have a 3rd party router (Ubiquiti EdgeRouter) I SSHed to it to see if I can get any more information about what's happening.

I see where the link to my Fiber Jack negotiated to 100Mb/s:
Dec 24 12:42:02 ERL-CT kernel: eth1: Link down
Dec 24 12:42:09 ERL-CT kernel: eth1: 100 Mbps Full duplex, port 1

I also see the link is now repeatedly bouncing up and down rapidly every few hours:
Dec 27 14:17:49 ERL-CT kernel: eth1: Link down
Dec 27 14:17:51 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 14:17:53 ERL-CT kernel: eth1: Link down
Dec 27 14:17:55 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 14:18:01 ERL-CT kernel: eth1: Link down
Dec 27 14:18:02 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 16:15:26 ERL-CT kernel: eth1: Link down
Dec 27 16:15:28 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 16:21:18 ERL-CT kernel: eth1: Link down
Dec 27 16:21:20 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 16:24:32 ERL-CT kernel: eth1: Link down
Dec 27 16:24:38 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 18:58:12 ERL-CT kernel: eth1: Link down
Dec 27 18:58:14 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 18:58:18 ERL-CT kernel: eth1: Link down
Dec 27 18:58:19 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 18:58:21 ERL-CT kernel: eth1: Link down
Dec 27 18:58:23 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1
Dec 27 18:58:27 ERL-CT kernel: eth1: Link down
Dec 27 18:58:32 ERL-CT kernel: eth1: 1000 Mbps Full duplex, port 1

Step 4: There are several possible culprits and I need to start eliminating some of them to find the root cause. The first thing I think of is that maybe my PoE injector that I'm running the Fiber Jack on has started to fail. To eliminate this possibility I connect the Fiber Jack to power through the wall adapter that came with it. After a day the connection to the Fiber Jack starts bouncing again. So that's one possibility eliminated. (I also learned the Fiber Jack can switch between PoE power and the wall adapter without interruption and vice versa.)

Step 5: The cable between my Fiber Jack and router is made of several segments. Maybe one of the connections has been damaged/corroded/etc. I start at the Fiber Jack, disconnect the Ethernet cable make sure the connectors are clean, reconnect the cable, gently pull on the cable after it's connected to make sure it's making good contact. I then go outside where the cable connects through a keystone jack to the long run across my house and do the same steps. Everything looks good until I get to the other side of that long run where I have an Ethernet surge protector (since the cable passes outside.) 

When I disconnect the cables from the surge protector I see some dust on the connectors. I reconnect the cables and gently pull on them, one side at a time and each time I do the link bounces between the Fiber Jack and my router. I blow out the connectors with compressed air, reconnect the cables and gently pull on the cables one at a time again. The link no longer bounces and the connections are solid. Problem solved!

Since I've had this experience, I went back and read some of the Google Fiber Help Forum posts about slow speeds and random drops and I'm convinced at least a few of them are bad connections/cables. Maybe this post can help someone else solve this problem in hours instead of days like it took me. :)

Google Fiber - Large increase in IPv4 address space

Google Fiber is gearing up to provide access for many more users.

Before October 6th, 2015 Google Fiber owned eight IPv4 address blocks totaling around ~147 thousand IPv4 addresses. On October 6th, 2015 Google Fiber acquired the /11 IPv4 address block 136.32.0.0/11. Google Fiber first began announcing this new address block to the global BGP table on November 24th, 2015.
This /11 address block adds an additional ~2.1 million IPv4 addresses.

Google Fiber - EdgeRouter Remote VPN Setup

Google Fiber's Network Box currently doesn't have support for remote access VPNs but the Ubiquiti EdgeRouter does. I'll walk through setting up remote access via L2TP over IPsec.

The firewall rule changes can be done through the GUI but everything else must be done through the CLI or the config tree.

Enable hardware offload for IPsec:

set system offload ipsec enable

Reorder a rule so there's space to add two new rules:

edit firewall
edit name WAN_LOCAL
rename rule 3 to rule 6
top

Add firewall rules for L2TP:

set firewall name WAN_LOCAL rule 4 action accept
set firewall name WAN_LOCAL rule 4 description 'Allow L2TP'
set firewall name WAN_LOCAL rule 4 destination port 500,1701,4500
set firewall name WAN_LOCAL rule 4 log disable
set firewall name WAN_LOCAL rule 4 protocol udp
set firewall name WAN_LOCAL rule 5 action accept
set firewall name WAN_LOCAL rule 5 description 'Allow ESP'
set firewall name WAN_LOCAL rule 5 log disable
set firewall name WAN_LOCAL rule 5 protocol esp

Setup IPsec:

set vpn ipsec auto-firewall-nat-exclude disable
set vpn ipsec ipsec-interfaces interface eth1.2
set vpn ipsec nat-networks allowed-network 10.0.0.0/8
set vpn ipsec nat-networks allowed-network 172.16.0.0/12
set vpn ipsec nat-networks allowed-network 192.168.0.0/16
set vpn ipsec nat-traversal enable

Setup L2TP:
#Change the username and password

set vpn l2tp remote-access authentication local-users username testuser password 'testpassword'
set vpn l2tp remote-access authentication mode local
set vpn l2tp remote-access client-ip-pool start 192.168.100.101
set vpn l2tp remote-access client-ip-pool stop 192.168.100.110
set vpn l2tp remote-access dhcp-interface eth1.2
set vpn l2tp remote-access dns-servers server-1 8.8.8.8
set vpn l2tp remote-access dns-servers server-2 8.8.4.4
set vpn l2tp remote-access ipsec-settings authentication mode pre-shared-secret

#Change the L2TP Shared Secret

set vpn l2tp remote-access ipsec-settings authentication pre-shared-secret testsecret
set vpn l2tp remote-access ipsec-settings ike-lifetime 3600
set vpn l2tp remote-access mtu 1492

Commit and Save:

commit
save

Additional Resources:
EdgeMax - L2TP Server Setup
Android Client Setup

Google Fiber - Dynamic DNS Setup on an EdgeRouter

Google Fiber's Network Box supports several dynamic DNS providers but I wanted to use one they don't currently support called afraid.org. Now that I have a router that can supports almost any dynamic DNS provider I can configure it!

Here's how to set it up on a Ubiquiti EdgeRouter.

From WebUI:

Go to the Service Tab
Then go to the DNS Tab
Press the "Add DDNS Interface" button
Select interface eth1.2 (assuming eth1.2 is your WAN interface)
Select afraid for the service
Set your hostname, login and password
Set the server to freedns.afraid.org
Press the Apply button



From CLI:

configure
set service dns dynamic interface eth1.2 service afraid host-name mydomain.mooo.com
set service dns dynamic interface eth1.2 service afraid login dyndns
set service dns dynamic interface eth1.2 service afraid password 123456
set service dns dynamic interface eth1.2 service afraid server freedns.afraid.org
commit
save

Testing the new configuration

From CLI:

Run this command to force an update:

update dns dynamic interface eth1.2

Run this command to see the status information for dynamic DNS.

show dns dynamic status

interface    : eth1.2
ip address   : 1.2.3.4
host-name    : mydomain.mooo.com
last update  : Sat Oct 17 10:53:36 2015
update-status: good

Notes:
If you have any uppercase letters in your afriad.org username, convert them to lowercase when setting your username on the EdgeRouter. (The afraid.org site is converting your username to lowercase when you login and the API calls the dynamic DNS client tries to make will fail if your username doesn't match exactly.)

Additional Resources:
Guide from Ubiquiti's Help Center on Setting up Dynamic DNS

Google Fiber - EdgeRouter IGMP Proxy for TV Service

In my first blog post about switching to the Uqituiti EdgeRouter I noted that users with TV service would have issues running Atlantisman's TV script as is after running the modified script I posted.

I've modified the TV script he created so it will work with the IPv6 script I posted before.
Disclaimer: I don't have TV service, so I have no way to verify that this script actually works.

Modified TV Script

If you have the first gen Google Fiber equipment, attach your Storage Box (GFMS100) to your LAN and your TV boxes should work again.

If you have the second gen Google Fiber equipment things are more complicated because you have a combined Network and Storage Box in the Network+ box (GFRG200, GFRG210).

There are two options to get TV working with the second gen Network+ boxes:

First Option - 

• Plug a switch into the Fiber Jack and connect both the EdgeRouter and Network+ box to the switch. 

There are a couple problems with this option though:
First problem, by connecting two routers to the Fiber Jack you'll be assigned two public IPv4 addresses. While Google Fiber currently allows this, in the future it will probably be restricted to only one IPv4 address and this option will no longer work.

Second problem, Google Fiber already restricts IPv6 address assignment and prefix delegation to one MAC address or DUID (DHCP Unique Identifier) per Fiber Jack. What this means is the Network+ Box will almost always win the race to get an IPv6 address and prefix before the EdgeRouter and the EdgeRouter will be stuck with only IPv4 connectivity and the CPU at 100% as the DHCPv6-PD process tries in vain to get an IPv6 address and prefix. By powering on the EdgeRouter first waiting a few minutes then powering on the Network+ box you can get around this issue.

Second Option - 

• Login to fiber.google.com and in the Advanced drop down for your network box change the "Router LAN IPv4 address", "DHCP start address", and "DHCP end address" to be on a different network than the LAN side of your EdgeRouter. (e.g. Change 192.168.1.1 to 192.168.3.1, etc)
• Plug the Network+ box into the LAN side of the EdgeRouter

Advantages of this option:

With this option you won't have to rely on a second IPv4 address that may go away or need another switch or vlan connected to the Fiber Jack. 

You can continue to use the wireless from the Network+ until your real access point arrives. :)

Current issues with this option:

Devices plugged into the Network+ won't receive an IPv6 address.

Devices plugged into the Network+ box will be going through a double IPv4 NAT.

Other Important Notes:

If you have a switch connected to the LAN side of the EdgeRouter and plan to connect your own wireless access point later, make sure your switch supports IGMP snooping and that you have it enabled/configured. The TV channels are multicast to the Network+ box/Storage Box and a switch without IGMP snooping will flood the traffic out all ports on the same vlan. (All ports if it's an unmanaged switch.) You can also put your WAP on the third EdgeRouter port and leave IGMP proxy disabled on it to get around this issue.

Google Fiber - Hardware Change for Basic Tier

Google Fiber's Basic Tier is their 5 Mb/s down, 1 Mb/s up tier that's free for seven years after installation.

Currently customers signing up for the Basic Tier have been getting the first generation Network Boxes (GFRG100 or GFRG110) from the stockpile that was originally manufactured. Sometime in the near future Basic Tier customers will be getting a different Network Box, the Mini Network Box (GFMN100).

Unofficial Specs:
One WAN port (With PoE to power a newer Fiber Jack GFLT110)
One LAN port
2.4 Ghz 2x2 WiFi (802.11n)

Google Fiber - Next Steps with the EdgeRouter

Ubiquiti's EdgeRouter is amazingly versatile for the price point. There's also a lot of documentation on their support and help center that covers almost anything you would want to configure.

Here's the link to the main documentation page:
EdgeMAX - Ubiquiti Networks Support and Help Center

I've put together a list of stuff I want to configure and I'll be creating a blog post for each one as I configure them.

IGMP Proxy (For TV service)

Port Forwarding

Dynamic DNS

Remote Access VPN

If you want to play with the latest and greatest features on your EdgeRouter, you can also sign up to join the version 1.8 alpha/beta testing group.

Google Fiber with a 3rd Party Router/Firewall

So you've had Google Fiber for a while and played with the advanced settings but you still can't do what you want?

Maybe you want to port forward to multiple devices that all listen on the same port, monitor bandwidth usage with SNMP, allow traffic to an IPv6 address in your home network, change DHCP server settings, use static routes, or even something far more advanced.

What options are available you might ask.

For non-power users, the easiest way to get the feature you want added is to create an idea or vote on an existing idea here on the Google Fiber Customer Ideas page. New features can take some time to get added to the product though and some features may never be added.

What if you are a power user and don't want to wait for the feature(s) you want?
In general there are two options that are popular so far. Replacing the Network Box with a computer running pfSense or replacing the Network box with a Ubiquiti EdgeRouter Lite.

pfSense has more features but will require more expensive hardware for gigabit throughput than the EdgeRouter Lite. I decided on the EdgeRouter Lite since I've never used Ubiquiti products before and wanted try one out.

If you decide to get an EdgeRouter, here's how to set it up.

After receiving your EdgeRouter you'll want to follow Ubiquiti's Guide to upgrading the firmware. The download link for EdgeRouter firmware is here.

A user named Atlantisman already has a great guide on setting up an EdgeRouter here.
Since he created that guide though, Ubiquiti has released EdgeOS 1.7, which offers many new features including full support for DHCPv6-PD (Prefix-Delegation, used by Google Fiber to assign IPv6 addresses). I've created an updated version of the script based off of the IPv6 config from a user named TK. I recommend following Atlantisman's guide but substituting the script linked below for his if you want IPv6.

GF-ERL-Commands - Update version of Edge-Setup-Interfacesv2
Note: I created this script off of the completed config, so it may need a little tweaking.

You'll probably want to change at least line 132 for the hostname and line 141 for the time zone.
Don't forget to change the password for the ubnt user (either through cli or the gui).

GF-ERL - Full Config - Easier to read if you only want to see how things are setup

Important Note:
When switching from the Network Box to the EdgeRouter you must wait for the DHCPv6 prefix the Network Box had to expire or the CPU on the EdgeRouter will be at 100% until it does. This is because of the way the EdgeRouter handles what are called unsolicited DHCPv6-PD advertisements. The EdgeRouter spawns a new DHCPv6-PD responder for every unsolicited advertisement but is never able to receive a prefix or address, this causes high CPU on the EdgeRouter which will cause slow throughput and no IPv6 connectivity.
What I did was unplug the Network Box before I went to bed and plugged in the EdgeRouter went I woke up. The lease time is one hour, so you don't really need to wait as long as I did.

Important Note about TV Service:
Since I don't have TV service I didn't create an updated version of the TV script yet. Do NOT run the TV script after running the updated one I linked, it won't work without changing a few things in the TV script. If you have gen 2 gear (e.g. GFRG2XX written on the bottom of your network box) then you have a combined Network and Storage box that can't just be removed or your TV service will stop working.

Powering the Fiber Jack with PoE (Power over Ethernet):
If you have a newer fiber jack (model GFLT110) you can power it with any 802.3af compliant power injector. I'm running mine with the TP-LINK TL-PoE150S.

Screenshots:
This is from a program named Cacti. It's monitoring the EdgeRouter via SNMP. Cacti is easy to setup on a Linux distribution like Ubuntu.

Monitoring CPU, interface discards/errors, and interface bandwidth. (There's an extra interface that's not in the script/config for guest WiFi - eth2.)







EdgeRouter WebUI:

Dashboard

The Config Tree view that can be used to configure anything that the CLI can.

Ubiquiti is still working on adding all the features in the main web UI but most of the basic stuff is there.






Update 10-13-2015: Added DHCPv6 lease time comment.

Phase III Pre-Registration Continues

West Johnson County, KS signups will open on October 1st, 2015 at 9am and close on November 19th.

Stop by an event to learn more:

Shawnee Great Grillers
September 25th
5 p.m. - 10 p.m. CST, Shawnee Town Hall
and
September 26th
10 a.m. - 2 p.m. CST, Shawnee Town Hall

Shawnee Oktoberfest
October 2nd
5:30 p.m. - 10 p.m. CST, Shawnee Civic Centre

Lenexa Enchanted Forest
October 22nd
6:30 p.m. - 8:30 p.m. CST, Sar-Ko-Par Trails Park

Phase III Pre-Registration Continues

Olathe East sign-ups opened today August 11th and will close on September 24th.

All fiberhoods in Olathe East have met their goals!

Google Fiber Expansion

On August 5th Google Fiber announced one more metro area: San Antonio.

Here's the link to follow progress with each city:
https://fiber.google.com/newcities/

Google Fiber's blog post about Salt Lake City:
http://googlefiberblog.blogspot.com/2015/08/san-antonio-fiber.html

Google Fiber Routing and Latency

When I first got Google Fiber I noticed the latency to some locations was higher than expected. After poking around with traceroute it was easy to see why. The path traffic was taking out of Google Fiber's network was much longer than it needed to be. Going to a location in Kansas City from my KC Google Fiber, it would route through San Jose. Other cities around the US had similar odd routing, except for locations near San Jose.

Early in the morning on March 19th, 2015 Google Fiber made routing changes that have now significantly improved latency for all the test locations I'd been monitoring that were higher than they should've been.

Note: All of these monitors are to Hurricane Electric core routers, so the loss statistics aren't necessarily accurate. (ICMP is low priority to respond to.)

The jump in latency in KC is the most staggering. From 33.3 ms average to 8.6ms.

To the person at Google Fiber made this change: I want to buy you a beer for making my latency in games that much better!

Google Fiber Expansion

On March 24th Google Fiber announced one more metro area: Salt Lake City.

Here's the link to follow progress with each city:
https://fiber.google.com/newcities/

Google Fiber's blog post about Salt Lake City:
http://googlefiberblog.blogspot.com/2015/03/google-fiber-is-coming-to-salt-lake-city.html

Areas Opened for Sign-ups Indefinitely in Kansas City

As of today Kansas City, KS, KC Central, MO, and the South Kansas City, MO areas have been opened for sign-ups indefinitely. Only fiberhoods that met their goals during one of the previous rallies in those areas are eligible to signup though.

This change may have been because of this letter that Rep. Emanuel Cleaver (D-Mo.) sent to Larry Page (CEO/cofounder of Google) on February 5th, 2015.

http://cleaver.house.gov/sites/cleaver.house.gov/files/Letter%20-%20Close%20Digital%20Divide.pdf

This news site made the link that these two events might be related:
http://thehill.com/policy/technology/237472-after-dem-reps-request-google-reopens-local-fiber-signup

Phase III Pre-Registration Continues

Lee's Summit sign-ups opened yesterday March 10th and will close on April 30th.

If you have a contract with your current provider, think about signing up for the free plan until your current contract is over. If your fiberhood doesn't meet it's signup goal then no one will be getting Google Fiber in your neighborhood until signups reopen again.

Updated 5-5-2015:

All but four fiberhoods met their signup goals.

Google Fiber Expansion

On January 27th Google Fiber announced four more metro areas: Atlanta, Charlotte, Nashville, and Raleigh-Durham.

Still waiting on word for the other five metro areas that were interested in.

Here's the link to follow progress with each city:
https://fiber.google.com/newcities/

Google Fiber's blog post about the announcement:
http://googlefiberblog.blogspot.com/2015/01/google-fiber-is-coming-to-atlanta.html

Which Cloud Storage Provider is the Fastest?

Most applications and services people use day to day on the Internet were designed with users only having access to normal broadband speeds in mind. When a user suddenly has Gigabit Internet, it can make the rest of the Internet feel slow in comparison. I thought it might be helpful to compare providers for different Internet applications and services and show which ones are the fastest (and therefor the "best" to use with Google Fiber.) I'll be comparing them only by speed and nothing else.

First up is Cloud Storage Providers.

I came up with the four most popular providers I could think of and tested both their Windows application and website for upload and download speed with the same 572MB test file. All tests are wired in, downloading/uploading the test file from an SSD, each direction was only tested once.
Note: iCloud Drive doesn't have a Windows app, so I tested their website only.
Second Note: Graphs are measuring speed in Megabytes per second (MB/s)

Test Machine:
Windows 7 Professional 64-bit SP1
Google Chrome was used for Website tests
Google Fiber's speed was verified using their KC speedtest server before running these tests (at least 940Mb/s up and down)

 Dropbox is the clear winner among the Windows Apps with the highest average speeds.

The result for fastest website was less clear. Apple iCloud had the fastest upload, while Google Drive had the fastest download. Overall Apple iCloud had the fastest combined average upload/download speed.

If you have ideas for other stuff to test or think I screwed this test up, feel free to leave a comment.

Google Fiber Austin - First Fiberhoods Are Open for Sign-Ups

The first five fiberhoods in Austin, TX are open for sign-ups starting today!

Bluebonnet           Deadline 1/29/2015 All met their goal except one!
Lady Bird Lake     Deadline 3/12/2015 All met their goals!
Emerald Forest     Deadline 4/23/2015 All met their goals!
Ben White            Deadline 6/4/2015 All met their goals except three.
Onion Creek         Deadline 7/16/2015 All met their goals except eight.

https://fiber.google.com/cities/austin/fiberhoods/

Google Fiber Business Service

Google Fiber's Business service is switching from the pilot to early access phase.

Gigabit service will cost $100 a month during the early access phase at least. Guaranteed price for one year. Plus $20/mo for one static IP, $30/mo for five static IPs. Looks like they get the standard Network Box that everyone else gets or they can use their own equipment.

As of 11/14 there are now many more fiberhoods covered by Google Fiber Business. This was expanded to include certain fiberhoods in Austin, TX on 12/1/14.

They took the map down that showed what areas small business service covers in KC. You have to enter an address to check now.

https://fiber.google.com/smallbusiness/

Google Fiber - KC Northeast and KC Northwest Delays

People in the KC Northeast area have received an email stating the installation goal date is now Summer 2015 and in the KC Northwest area they received an email stating the installation goal date is now Spring 2015.

These are goal dates for the whole area. Specific fiber hoods inside the area, particularly ones that had the highest signup percentage over their goal will likely be installed sooner than the goal dates.

The only information I've been able to get so far on the cause of the delays is that they are having major delays in creating the infrastructure and they are expecting a harsh winter to slow down crews.