VoIP question

[Dinsdale]

Probably a long shot, but ET seems to know about this stuff.

First off up front -- I'm the "backup IT Guy" at this location... which is good, because you sure the fuck don't want me as your primary IT guy.

This will explain my lack of exact product models and whatnot at this moment.


But... went with a VoIP setup. Not sure of the manufacturer of the phones, but the PBX is a Netgear.

Installed the phones and the PBX, and after some basic configuration (which I didn't do, since I was working on other stuff on the network at the time, and frankly, it was a fairly informal, unexpected work session, and I was about half in the bag), things seemed fine.

Except...

First, the basic network config:

3 buildings (actually 4 now, 5 in another week or two... yeah, the company is doing quite well). All 3 on same subnet (no more than 15 PC clients, subnetting not a big deal at this juncture). 3 buildings connected by wireless antannae (some crazy booyah freaking units, since there was just too much RFI to maintain consistant signal across parking lots). I believe the WiFi is WPA, but would have to look to determine which version/type. The wireless antennea aren't in the DMZ, all internal (no connection to the Sonicwall, strictly LAN traffic, although we'll eventually upgrade the wireless security, there's bigger fish to fry at the moment).

VoIP works fine... except in one building. There's no other network issues in that building, and the phone (only one in that building) can even be pinged.

Yet, for some reason, the VoIP traffic just won't flow to that particular building. There's no router/firewall stuff between the sites (all gigabit switches). The wireless antennae have all been reset (and possibly had firmware upgrades, although I didn't do them) to factory defaults, just to ensure there was no firmware corruption.


And of course, Netgear are being absolute CUNTS when it comes to support -- won't even tell us what ports the phones talk on (seems like an important thing for a "support tech" to know).


Just can't get the VoIP to that one damned building. We can ping any client (including phones) from anywhere in any building (except the New New building, but that's a corrupt IP stack/software issue, since it's a lax environment, and a user was allowed to install sync software for some phone or some bullshit, but that's an entirely seperate issue... boss likes the informal setting, and trusts his employees, which just means more work for me).


Since I remember ET talking about VoIP some time back, I figured I'd see if you had any thoughts (or anyone else)?

[Dinsdale]

Thanks for all the help, guys.

Still not fixed, but starting to think the RTP threshold doesn't jibe with the voice data portion of the IP packet... maybe. I guess VoIP is pretty sensitive to how RTP chops it up over wireless.

[Screw_Michigan]

Get fucked. There's your help.

[Dinsdale]

Don't be mad just because my technical skills go beyond operating the wringer on a mop bucket.

[ElTaco]

Sorry for the delay but I've been busy with similar issues at work:

So there is a lot to consider and we may need to touch base in person if you really want to continue discussing this but:

VoIP is fairly standard stuff, certainly netgear hasn't really invented anything that has stormed the market so they are most likely using standard VOIP protocols.

In general all phones do certain things when on VoIP. They boot up and initially do the standard DHCP or Static addressing. Next, most systems will connect using TFTP to a TFTP server and download the settings. Again, this can be set up manually but in general it is easier to just set a TFTP server and then just do everything centralized. Now I should state that I haven't worked with NetGear, nor do I have time to do much research right now to try to figure anything out about it, but depending on your phone, you should be able to go into the phone settings and actually look at the setup.

I'm assuming when you are talking about your wifi-connection you used 3 APs that do Bridging and the base AP that is connected in the main building just plugs directly into the internal network switch, along with the NetGear VOIP server.

Does the same phone work in another building and does another phone (that is proven to be working) not work in the broken building?
Do you have network connectivity to the Netgeat PBX server from that building? (are you sure the PBX is netgear? I didn't really see them have one, are you sure the switch isn't netgear and a server is doing PBX functionality?) (ping and tracert)

On the phone (under settings - most even have a web interface) look if there is a tftp server setting and see if you can make a tftp connection to it. If it is using a TFTP server, it will be open so you should be able to use a tftp client from a computer to connect and download the config file, which could also be an issue, no config file for that particular phone.

Sorry for the delay I started this last week and then forgot to press submit so its been hybernating.

[Dinsdale]

First ET, let me say in sincereity, you seem like a heck of a nice guy (not to ruin your board cred or anything).

Second, I'm not sure where I came up with Netgear. It's a DLink system.

I was on the phone talking to my kindasorta boss when I opened this thread, discussing the STILL nonworking VoIP.


Wherever the base unit/PBX thingy is located, it functions. But there's always at least one of the wireless segments that gets left out (which the one employee in that building still has a regular phone, so it's not the end of the world in the short term. But having an intercom and being able to use an extension would be freaking nice, for sure).


Everything is hunky-dory, except that one wireless segment.

The wireless architecture is about how you describe it, sort of. Building 1 (company is expanding so quickly, it's taking up residence in multiple buildings in the same industrial park, basically making the network infrastructure a mess, kind of a "design on the fly" sort of deal. Network demands are literally changing every week)... anyhoo, Building 1 (I'll call it) has the DC, the DSL, the firewall (Sonicwall, but the VoIP network is all inside the DMZ, definitely not a firewall issue). The wireless on Building 1 is the AP for the (now) 3 other wireless antannae, which are in bridge mode (the wireless are all bigboy Tranzeo units... this company ain't messing around with cheesy infrastructure... but the DLink setup was several times more cost effective).

But here's the crazy part -- even on the "dead" segment, the phones get a DHCP addy, and show up as clients on the network. We can hit the extension and make the "dead" phone ring. We can ring other phones in different buildings from the "dead" phones. But the actual voice-data gets lost somewhere along the way. No sound comes out of either end, but every other aspect of the system seems to function, regardless which building the base unit/PBX is located.


But Homeboy just got off the phone with one of his old connections, and dude made some suggestions. Maybe you can tell me what you think:


Subnetting the different buildings independently (seems basic now, but as mentioned -- there's been some explosive physical growth, but bandwidth has never been even the slightest issue. 30 clients or so including phones aren't struggling over gigabit, nor are the bridges taxed in the slightest. Clients were kind of added one at a time, so the previously single subnet isn't as nuts as it sounds at present).

Each subnet will use the wireless as its gateway.

All of the wireless antannae (not sure of the proper terminology, since the Tranzeos can be used in many different modes) will exist on their own subnet, and instead of bridging them, they'll be set to router mode (and will handle DHCP for their subnet), which shouldn't be too evil of a routing table.

By using routing mode, it's possible to enable QoS on the wireless devices. The "consultant" seemed to think this might solve the voice-data issue.

If it doesn't, that sounds like a lot of work running between buildings and a whole buncha typing for nothing. But dude is apparetly something of a veteran when it comes to the whole "VoIP should be pretty basic," which seems like peoples' standard response, along with "VoIP will ALWAYS be transparent to wireless"... which I can say for fact is complete bullshit.

I still have my doubts about how the RTP settings and frag threshold æffect the voice data payload. In my crash-course on VoIP-over-wireless troubleshooting, it's hard to get specifics, but it seems VoIP is sensitive to RTP and frag thresholds. BUT... it's been suggested that enabling the QoS will fix those things... I have my doubts.

You'd think it would be easy to come up with a sniffer setup to figure out where the data is being lost... but can't even find a contractor to do it in this tech-rich area... ponderous.

Speaking of contractors, they were looked into -- and all wanted several times the money to install the system than has been spent so far... like several times the money, so a little headache is worth it.


But I'll tell you what -- skilled VoIP techs are going to be worth their weight in gold in the coming years. Better to learn about it now, and be ahead of the game. Of course, I now know just enough to be dangerous, but not enough to have skeelz.

Regardless, if we all exchange what we learn about it, we'll all be better off in the long run. I'd like to hear any suggestions, and I'll report back with what I find out. Surely this can't be the first time someone has run into this, and it surely won't be the last.

[Dinsdale]

Update:

A VoIP specialist came and checked it out.

His conclusion:

"Damn, that's messed up. Might be in the wireless bridge."

My reaction:

"Thanks, Network Sherlock."

[Shlomart Ben Yisrael]

Update:

A VoIP specialist came and checked it out.

His conclusion:

"Damn, that's messed up. Might be in the wireless bridge."

My reaction:

"Thanks, Network Sherlock."

If I saw a TCP/IP stack full of bad checksums, I'd say that's fucked up and leave it at that.

[ElTaco]

I've got some ideas but you've cleared up a few things. Also I have a pregnant gf in the next door screaming so I best be on my way but I"d look into doing QOS. Sounds like it will solve your issues. I was going with the idea that your phone wasn't getting an IP or its setup instructions (most likely from tftp) but obviously it is working. What is however happening is that more then likely either you have some port that is blocked (not likely) or your throughput from your one segment isn't good enough to get consistent VOIP traffic. You wouldn't necessarily see this in the data because it has a lot of built in ways to cope with slower/crappier connections but voip and similar protocols would have a fit. This often comes out in only hearing one way traffic but if the connection was bad enough, I'd say it would be easy to hear nothing.

Your best solution is to look into doing some QOS action for your VOIP stuff. Basically, on cheaper SOHO stuff you can define QOS based on MAC address but more then likely you would be setting QOS for a type of traffic. Using QOS, you would prioritize your VOIP traffic so that it will take priority over the other stuff (regular data) and can even do a little traffic shaping to guarantee a certain amount of your total bandwidth to VOIP.

Of course your other option is to do some testing of your connection. I'll have to look around but there is software out there that can look for problems between two machines (bandwidth and such) and perhaps if you got a real network guru in, they might have equipment to measure your wifi power and connection and make sure it is working correctly. You might walk around with a laptop and make sure that you are the only one using your channel or look into some other ideas then using WIFI between buildings.

[Dinsdale]

look into some other ideas then using WIFI between buildings.

We'd LOVE to.

Ain't happening. Although, this company has just exploded at a jaw dropping rate (very interesting business they're involved in), and it's likely that they'll get their own building in the next year... ONE building. Got 5 now, although at least one has a CAT5 running under the pavement, since it's a very short run (which I think was there previously).


The VoIP guru dude suggested that with the Tranzeo radios, there might be an issue with multiple users of a single port across the bridges. Of course, my bud/boss/coworker and I figured that since TCP Port 80 seemed to work just fine with as many connections as we care to make, that theory falls on its face.

But, dude is coming out this evening with some test radios on tripods, to see if it isn't some funky dealio with the radios.

Also considering setting up the QoS, as you suggested. Don't know enough about it, but Guru tells us he'll give a primer on QoS and how to configure it.


But I can ping the damned phones in the "dead" segments. They get DHCP, they register on the network, and they pretty much do everything but start a session with the base unit. The traffic to start the session and whatnot even hits the phone -- but for some reason, it won't return the traffic (trying to remember what that procedure is called on the SIP session).

We haven't completely ruled out the radios overpowering themselves and bouncing double-packets off the surrounding buildings/terrain... but I would think that particular problem would be much more random, rather than specific to one protocol/port. All other traffic rips through there like nobody's business... infrastructure/physical stuff is solid for the amount of traffic -- to the point of being "plentiful."

And two of those bridged segments worked over the wireless from the moment they were put in.... freaking aggrivating. I'm about ready to cut a groove in the asphalt and run a freaking cable through it... there's my newest freaking solution... or go late at night and twist it around the phone lines and hope no one notices (of course, the short runs where I might actually get away with such silliness are the ones that work).

[ElTaco]

Well So here is what I've found:

Testing ping and routing and such is a good start but will not actually really trouble shoot VOIP. Also QOS isn't necessarily the best solution because any time you have plenty of bandwidth and a good network, all traffic, including VOIP should work fine. The reality is that VOIP, video, etc... requires steady traffic with the packets arriving on time. If you have too many dropped packets or anything similar, that is when you start having issues. I usually see this with home users. Technically for example, most VOIP traffic only needs on average 60 or so kbps. Almost any dsl/cable internet user should have plenty of that. The problem is that by users who have slower connections (300kbps or less) going either way and they have other data going through their soho router (kids downloading, vpn connection, etc...) you get enough of a drop that you end up not being able to use your phone. Most of the time it will do something crazy like work one way but I assume if the problem is bad enough, you might have bigger issues like establishing calls or not being able to hear from either side.

QOS is fairly simple. Ideally, you would want to implement it network wide, but it sounds like your network isn't huge right now. Also since you are moving (and growing) hopefully you'll have plenty of $$ to build your network appropriately. In the mean time I would look to test reliability of your wifi connection. I would look for packet drops and see how fast the packets are arriving.

To test yourself, you can use a couple techniques. One is to get your hands on a tool that will send and receive packets over the connection. You'll need two computers (one on either side of the network) and then start a test. There are a lot of QOS and network monitoring tools. Some have free/30 day test periods. One that is nifty and open source is called k-perf. You load it on PCs on either side and off you go. If you have quality equipment, it may also have built in testing ability that you can access using SNMP.

Another option is to use online tools such as the ones that http://www.dslreports.com/tools offers to test and see what kind of difference you see in the results. Remember however that this would also have to account for your testing going over the internet, which is why I say that you would be more interested in the difference between the two sides of your connection rather then the actual quality report you get from either side. We use a company called icore and they have a speed test application free as well to use online. Again this would be going over the internet but might be an indication if you have a good internet connection otherwise. http://www.icore.com/speedtest

There are similar tools out there. In terms of your wireless, certainly if your bandwidth and connection is strong, you could try turning down your wireless power or you might look into directional antennas, which would help with interference from other wifi connections and from the signal bouncing back.

I would think that if you have interference, you should be seeing a lot of bad traffic and your speed across would be dialed back.

Now when it comes to QOS, you can keep it fairly simple or make it complex and very efficient. Generally, QOS works from point to point if all devices in between speak QOS, however, you can occasionally take a shortcut and only apply QOS to a single or a few select devices that communicate over your slow WAN/MAN connections. In other words, even though you can't implement QOS accross the internet, you can enable it on your home router and it will prioritize all traffic leaving your home network so your VOIP or your Game traffic will go first and your other data will go second.

With that said, you could enable QOS just on your wifi routers, it should prioritize the data from your phone over the wifi connection and not have to compete with web traffic and such.

If you want to make it more complicated and you are using decent, managed equipment all the way through, you can prioritize / use QOS across all your equipment.

My test would be to enable QOS on the wifi router (in the far building with the phone that doesn't work) and enable it for either the phone MAC address or the VOIP traffic port. Then if it works, you can enable it across your enterprise. If you do it based on the port/traffic type vs the MAC address, it will work for all VOIP traffic/devices so you don't have to keep adding MAC address/IP address information as you add/change phones.

Hope this helps. Good luck.

[Dinsdale]

Thanks, ET.

I really don't think it's a straight-up bandwidth issue, since we seem to be getting every bit of the 54mbs.

There have in fact been issues with the radios "overgaining," or whatever the heck you call it. The contractor who installed them came out and adjusted them, and claimed they were fine... I still have doubts.

But I think your idea of testing PtP, or at least comparing DL speeds on either side of the link is a darn good idea.

I'm also thinking it's not a bad idea to rig up a wireless adapter to sniff the WEP ltraffic... might lend some insight into what's failing where. We were at least able to sniff enough to figure out which part of the session-initiation was failing, just not enough info to figure out exactly where it dropped.

The brand-new building has the same issue... and both the first building and the last have exactly one phone each... had the rest of the VoIP network working like a freaking champ in a couple of short hours (most of that spent walking between buildings and naming phones)... the last two (which started out as only one) has been one mean mofo.

[Dinsdale]

OK. It would appear that our several hundred dollar radios aren't quite enough hundreds of dollars.

Seems to boil down to the access point not being able to multiport -- since the phones only talk on port 5060 whichever link gets the phone base-unit establishes a session on 5060, and that's that. 5060 doesn't swap to a different port, like other ports, notably port 80, do.


Suppose to be getting a better access point tonight, we'll see how that goes.

[ElTaco]

interesting... I suppose the bridging may cause issues like that but kind of weird considering you aren't NATing so you shouldn't have to jump ports. If you are doing NATing, then my question is why are you doing that since you should be able to bridge the lan without having to do any NAT tricks.

Either way, let me know how that works out.

[Dinsdale]

The Guru Consultant is still on the case.

We set up a test network with different radios, ones that Guru researched and said would connect with our Tranzeo equipment, and support multiporting.

It didn't work. Same deal -- the phone DHCPs, joins the network, pings, does everything a network object is supposed to... except register with the base unit. Just can't pass SIP port 5060 over multiple wireless links.

Guru has a VoIP test kit... in Florida. It's on its way.

Unfortunately, I think the solution might end up being installing two more Tranzeo radios as access points. Each building-to-building connection will be its own SSID. Sounds cumbersome, but at least it would do away with the need for multiporting... since industrial access points that support that type of network are several thousand bucks, as opposed to the few hunsky that the client is currently into it for. Less than a grand would probably seperate the links, and I suppose would increase speed between the buildings (which isn't really an issue at present).


The only NAT is on the outbound firewall.

[Dinsdale]

UPDATE -- egads.

ADVICE -- If anyone ever offers you the opportunity to work on a cheapass VoIP system over multiple wireless bridges, I'd sincerely recommend declining.


Tranzeo is a big name in Wimax shit, but it sucks, btw. One typo in a static route (oops), and it's on a slow boat to Vancouver, BC... where apparently, they yank chips out and reflash them... SINCE THERE'S NO FUCKING RESET BUTTON ON THE FUCKING THINGS... stupidest shit I ever heard.

Oddly enough, when we replaced some of the Tranzeo radios and did some subnetting, that DLink POS magically started working across those bridges.

Still sitting here looking at a looped Tranzeo.

Talking with a wireless/VoIP consultant, those newfangled Microsoft Response Point cheapass VoIP systems are starting to sell like hotcakes, due to the significantly lower-than-previous-systems pricing. And people with wireless/wifi bridges are apparently having all sorts of problems with them.

But at least I was on the cutting edge of discovering their myriad weaknesses.

NONE of the infrastructure setup decisions were mine-btw. I was just trying to make the shit work. But a $60K Cisco VoIP setup wasn't in the cards, nor was a set of multiple $4000 radios.


But damn, what I crash course in wifi/wimax I took recently.The 5 gig shit ain't fucking around.

[ElTaco]

Well good to hear that you got it working. I was recently at a Hospital IS trade show with my company and ran into their booth. Snickered a bit. Routing, Routed protocols, slow response, etc.. can definitely screw with you. Its amazing that a bad rule can't be reset on the stupid box. You would probably have been better off with a few Cisco WAPs with some directional antennas on them personally. In terms of functionality, QOS, routing and manageability, can't really be beat.

As far as cheap voip, really this stuff has been out there already. Many companies have rolled their own voip solutions that they are more then willing to sell you for fairly cheap price. Other outside vendors are also ready to walk into your place and set you up, which doesn't make that much sense for small organizations, but if you have 100 or more people, with multiple sites, its not a bad deal. A DS1 (1.5mb/s - ~$300/mo) can run a good number of calls over to a 3rd party VOIP provider for a small company.

Of course if you want free, there is even a really good solution for that called Asterisk, which is awesome. I played with it for a while and there are a number of ready to go, live cd style installations that you can install on an older P4 server and run it as a voip server, with voice mail and many other features. There are some voip providers that will provide you with over the internet (voip) lines and phone numbers for your voip server. At the end of the day it comes down to if you want to/have the expertise in house to build the system and support it, although there are even companies who will do that for you.

I'd say for $5000 you can roll out a decent system for a good amount of people and then pay a bit once a year for support and your DS1.

[Dinsdale]

Solid advice, ET.

The Cisco AP's that I saw were pretty steep in price.

What worked really well for the application were Ubiquiti Nanostation 5's. They don't have the super long range the Tranzeos do, but we only needed a couple hundred yards max.

After the Tranzeo configuration nightmare, the Nanostations were up and running perfectly in very short order.

Ubiquiti = Recommended

Tranzeo = Not Recommended


Did I mention that I now know more about wimax radios and RTS and all that jazz than I ever wanted to?

I suppose I should reassemble this Tranzeo Paperweight and send it somewhere one of these days.... there's even a spot for a factory default switch on the circuit board... says so right on the PCB... Tranzeo just didn't spend the $0.25 hooking it up (and no, shorting the terminals on the board doesn't do it on a TR5a).


Fuggin canadians.