Having the exact same issue with Opnsense, it must be a bug in the firmware. It just doesn’t want to grab the IP. it just sits on 0.0.0.0/8
Yet I can plug the ONT directly into my PC, set the NIC to VLAN 911 and it works instantly.
Really Bizarre issue. I may post on the Opnsense forums as there must be some way/logs that are generated as to why its happening.
Very strange stuff. It seems crazy to me that a bug could have existed all these years in OPNsense or pfSense where you cannot pick up a DHCP IP on a VLAN, but stranger things have happened.
Hi Lee,
Makes me feel a bit better as I thought I was going mad as it shouldn’t really be that hard to do! If you do get anywhere, please post on here and I will do likewise. I have a bit of spare time coming up so I am going to throw yet more of my time at this as it’s just nuts. There was an Opnsense update only a few days ago and I haven’t tried it since then so you never know.
Cheers.
Keeop
Is there anything specific that needs to be done to get this working with Opnsense/Pfsense, i’ve exhausted pretty much all avenues.
A packet capture from opnsense just shows endless discovers when its trying to obtain the IP.
ethertype 802.1Q (0x8100), length 346: vlan 911, p 0, ethertype IPv4, (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 0e:6d:a4:4a:73:86, length 300, xid 0x21521273, Flags [none] (0x0000)
Client-Ethernet-Address 0e:6d:a4:4a:73:86
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message Option 53, length 1: Discover
Client-ID Option 61, length 7: ether 0e:6d:a4:4a:73:86
Hostname Option 12, length 8: “OPNsense”
Parameter-Request Option 55, length 10:
Subnet-Mask, BR, Time-Zone, Classless-Static-Route
Default-Gateway, Domain-Name, Domain-Name-Server, Hostname
Option 119, MTU
I doubt @Yayzi_Team can advise specifically since it’s definitely an issue on the customer side with a custom setup.
I did mention in an earlier post that it’s very strange to see no incoming traffic packets at all, to me, I could be wrong, but that immediately signifies that the outgoing traffic from OPNsense/pfSense is probably going out without a VLAN tag at all.
I think the best place to get help is likely to be the OPNsense or pfSense forums as they will have a better idea about what might be happening and how to debug it further.
Maybe they can’t advise Peter, but asking never did any harm.
Theres alot of people running Opnsense/PFSense so keeping it in this thread may help somebody in the future who is stuck. My OPNsense connects to Virgin Media using DHCP just fine, no additional configuration needed at all. So something isn’t playing nice, so it theoretically could be on Yayzi’s/CityFibres side or it could be in Opnsense. And to your point about the traffic going out without a VLAN tag, its pretty clear in the capture i posted that the tag is in the packet header.
Theres also the customer service side of it, you could have alot of people thinking about joining Yayzi that run OPNsense, if they see if doesn’t work, its lost custom for Yayzi. I know what i’d be doing if i were management of Yayzi, getting involved and helping the community out 
Sorry I missed the VLAN in your packet dump but yeah it’s still strange. Can you maybe try dumping the VLAN interface itself instead of the physical interface?
Here is a dump of my DHCP Discover firstly from the VLAN:
18:40:27.182135 xx:xx:xx:xx:xx:xx > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x0, ttl 16, id 0, offset 0, flags [none], proto UDP (17), length 328)
0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx, length 300, xid 0x87ddbb23, secs 12, Flags [Broadcast] (0x8000)
Client-Ethernet-Address xx:xx:xx:xx:xx:xx
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Hostname (12), length 8: "MikroTik"
Parameter-Request (55), length 8:
Subnet-Mask (1), Classless-Static-Route (121), Default-Gateway (3), Static-Route (33)
Domain-Name-Server (6), NTP (42), Unknown (138), Vendor-Option (43)
Client-ID (61), length 7: ether xx:xx:xx:xx:xx:xx
And then from the physical interface:
18:45:07.680176 xx:xx:xx:xx:xx:xx > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 911, p 0, ethertype IPv4 (0x0800), (tos 0x0, ttl 16, id 0, offset 0, flags [none], proto UDP (17), length 328)
0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from xx:xx:xx:xx:xx:xx, length 300, xid 0xce2e8a3b, secs 36, Flags [Broadcast] (0x8000)
Client-Ethernet-Address xx:xx:xx:xx:xx:xx
Vendor-rfc1048 Extensions
Magic Cookie 0x63825363
DHCP-Message (53), length 1: Discover
Hostname (12), length 8: "MikroTik"
Parameter-Request (55), length 8:
Subnet-Mask (1), Classless-Static-Route (121), Default-Gateway (3), Static-Route (33)
Domain-Name-Server (6), NTP (42), Unknown (138), Vendor-Option (43)
Client-ID (61), length 7: ether xx:xx:xx:xx:xx:xx
Perhaps it’s an MTU issue? Seems unlikely though.
Did you folks originally buy the Yayzi connection with their router, and then switch to your own, or did you tell them from the beginning you’d be using your own equipment? Wondering if there’s some kind of MAC locking going on if it’s the former - you could try spoofing the provided router’s MAC if so. I guess that’s ruled out by you testing using your PC directly though…
Okay got more interesting info.
Seems like the DHCP response is tagged as a high priority packet:
d4:6d:50:2d:b9:b4 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 345: vlan 911, p 6, ethertype IPv4 (0x0800), (tos 0xc0, ttl 30, id 28293, offset 0, flags [none], proto UDP (17), length 327)
The tos 0xc0 being the indicator of high priority. It seems like there’s an issue in FreeBSD that drops high priority packets by default, see references from Netgate employees:
Failing to pull a dhcp lease because the ISP is sending back all the dhcp responses priority tagged and pfSense 2.6 just drops those is a (relatively) common failure.
If they are sending priority tagged replies to dhcp requests you can just use a switch to strip those tags.
If you need to send and receive priority tagged traffic then you still need the netgraph script to do that and something in the iflib based Intel drivers has broken that. So you ca try using the non-iflib driver or use a different NIC.
FreeBSD should probably be accepting priority tagged packets and not dropping them by default.
OPNsense issue which covers a VLAN 0 issue and touches on the priority issue as well - dhclient does not decode 802.1q-encapsulated replies · Issue #114 · opnsense/src · GitHub
So while I am not 100% sure this is a smoking gun, it feels like it. Really this is a fault in FreeBSD (as stated by Netgate employees), but maybe Yayzi can stop sending DHCP responses with increased packet priority and it might fix your issue.
Some good packet sniffing there Peter - nice one! Certainly makes sense then.
Do we know if this is common with Cityfibre FTTP or Yayzi specific? Mind you, I would have thought if it was something common among Cityfibre ISPs there would be more chatter on the web and I never came across anything while trying to get this resolved.
So, yep. Over to you Yayzi. I’m currently on a monthly rolling to try this out and if there’s no way of getting it to work with Opnsense then it won’t be an option to switch to once my VM contract is up. It works fine with my Unifi gateway but I’m not sure I’ll be keeping that as it seems to have its own foibles.
Cheers.
Keeop
It’s Yayzi’s DHCP server configuration - most other Cityfibre ISPs are using PPPoE not DHCP so you wouldn’t see this (unless PPPoE packets were sent with higher priority). Most UK ISPs use PPPoE but arguably it’s better to not be using it.
It seems like this setup is relatively common to see when the DHCP server is handled by Cisco equipment (which it is on Yayzi). The fact that FreeBSD is dropping these packets by default seems just crazy though.
Well after 3 days of troubleshooting I have finally found the problem, with my setup anyway. We are in business.
Don’t leave us in suspense!
Not leaving any one in suspense and as I stated in an earlier post, keep it in this thread rather than posting on *sense forums. @Keeop are you running opnsense on Proxmox by any chance? If so i’ve found the fix, for that Virtulisation at least. I suspect Hyper-V/VMware etc will be a similar implementation. Let me know and i’ll post it in here.
That’s what I am suggesting to do. As you stated yourself earlier in the thread, it could help others who want to use Yayzi so why not post the solution you found regardless of what hypervisor @Keeop is using?
I mentioned keeping people in suspense as despite all the earlier details you’d posted when you had the problem, when you fixed it you didn’t give any details.
What does it matter to you Peter? You don’t have the issue so drop the bolshy comments. Please don’t dictate to me what I should and shouldn’t be posting when the issue has no bearing on you what so ever.
I asked which hypervisor it is out of curiosity. End of.
1 Like
I’ve been nothing but friendly and helpful on the forums here, and I’m sure many people would tell you I’m the exact opposite of bolshie, but ironically that seems to be a perfect definition for your tone in your posts right now.
My only request (and it was a request - not a dictation - so I’d appreciate if you didn’t twist the wording in my posts) here was for you to post your solution because as you stated earlier, it would be helpful to others trying to use OPNsense or pfSense on Yayzi - nothing more.
In any case, I’ll stop posting in the thread, just maybe be a bit more friendly in the future to those spending their own free time trying to help you?
2 Likes
Hi Lee,
I am running on Hyper-V. Please do share your solution and I’ll give it a whirl.
Cheers.
Keeop
1 Like
Ditto, I have spent three evenings meticulously troubleshooting this.
All I asked was what the virtulisation platform was. It may not matter to you, but i’d like to know.
Hi @Keeop
In the end I didn’t create any vlans in OPNsense. Proxmox had a issue where it was not VLAN aware on the Linux bridge until the proxmox host was rebooted. Even after the settings were applied apparently!
From there I tagged the interface bridge connected to the ONT on 911
As you can see no VLANS in Opnsense
So for Hyper-V i’d remove the WAN VLAN you have created in OPNsense and set the WAN interface back to just the NIC, as I have in my screenshot.
Then in Hyper-V go to virtual switch manager select your NIC that is connected to ONT, enable VLAN and tag it on 911 like below. Reboot OPNsense and it should pull the DHCP address straight away.
In the end this isn’t a FreeBSD issue or a Yayzi issue, its a virtualisation config issue and the intricacies that go with it. Lessons learnt.
Let me know if it works, if not, i’m more than happy to set this up on my Hyper-V host and replicate it and see if I can find where the issue is. I know just how frustrating the issue can be!
Hi Lee,
Damn, didn’t even think of that which is a bit nuts as I have had to do something similar with my Sophos UTM VM to get VLANs piped through it from my L3 switches. Too many moving parts!!!
Thank you very much though - just keeping the standard WAN connection as-is on Opnsense (which is normally connecting to VM), and adding the VLAN tag to the virtual NIC as part of the host config and can now connect to Yayzi. Nice one.
Thanks to @Peter as well for your time on this.
Cheers.
Keeop