Blog

Final thoughts on Halo

Hey there, guys. Zach Wigal here. It’s rare I would ever write an article on the Gamers for Giving website from a first person perspective. However, in this case, I believe it’s appropriate to help further address the cancellation of this past weekend’s Halo tournament.

First and foremost, whether you made a request or not, we will be issuing direct refunds to all players who purchased passes to compete in the Gamers for Giving 2016 Halo tournament. We know you all enter for a variety of reasons. Some of you participate to support our fundraising efforts, others participate to enjoy a competitive experience. Whatever the case, you had an expectation to compete, and that went unfulfilled. Each team pass will be refunded to the original purchaser.

The refund process will take about a week to complete. LANFest handles the ticketing process for our event, and we’ll need to coordinate on transactions that may be outside PayPal’s normal refund window. If you prefer to support the cause, you are welcome to make a donation upon receiving your refund.

Our team is just as disappointed by this year’s Halo tournament cancellation as you are. We spend months preparing, promoting, and organizing each activity at Gamers for Giving. Our event is managed almost entirely by volunteers. Many of our volunteers pay their own travel, take vacation days at work, or spend time away from friends and family so they can be a part of creating a memorable experience for each of our attendees. For the entire week leading up to Gamers for Giving, our core staff averaged less than 5 hours of sleep per night. We pour our hearts and souls into this event. While we’re proud to have surpassed this year’s fundraising goal, and we’re certainly glad other aspects of the event were enjoyed by many, it’s disheartening we couldn’t share a complete experience with our Halo attendees. For more than eight years, Halo tournaments have been a large component of Gamers for Giving. The franchise is what led to the start of our organization, (which oddly enough, was sparked because of another canceled Halo tournament) and we were really looking forward to hosting a great event for all of our Halo competitors.

Before diving too deep on a topic I feel is important to discuss, I wanted to provide some clarification as to our handling of the weekend’s tournament. On Sunday morning, we posted an announcement on the Gamers for Giving website which briefly touched on the efforts our team made to save the weekend’s Halo tournament from an issue we later realized was out of our hands. The original post was crafted for the sake of brevity, as we were in the midst of the event itself.

We have a capable and experienced staff, and you deserve to understand how we prioritized our decisions, since they ultimately affected your time and resources.

Our primary goal on Saturday was to do everything within our power to ensure the Halo tournament would take place. The following is a rough timeline for what transpired during the time of our event, which will at least provide you with additional context so you can know every effort was made by our team:

  • Consoles were acquired by our equipment sponsor and updated at my house on March 9th and March 10th. During our updating session, multiple volunteers connected the consoles to Xbox Live using generic Gamers Outreach accounts, each loaded with XBL Gold Memberships. We installed the most recent console / Halo updates, and experienced no issues connecting batches of consoles to custom matchmaking games.
  • On Friday, our team arrived to EMU to begin setting up Gamers for Giving. When all our network runs were complete, tournament admins were again able to successfully connect batches of consoles into matchmaking lobbies. It was at this point our team verified the tournament consoles should have been fully ready for play, given their ability to see each other on the network, join custom game lobbies, and connect to matchmaking.
  • As attendees began to arrive on Saturday ~11 AM, the main ethernet line from the EMU Convocation Center from which we were drawing our internet became unstable. The issue first became apparent among our streamers, who began to experience connection flapping as their streams intermittently dropped to a redundancy line we had constructed for network emergencies. Simultaneously, the first few consoles players who entered our venue experienced matchmaking drops after attempting to initiate warm up matches. Our network team ran a variety of tests in an attempt to identify the cause of the problem, but could not find any reasonable error within our own hardware. When our team decided to inspect the physical cabling within the venue (~12 PM – 1 PM), we discovered the Convocation center’s primary delivery ethernet cable was physically unsuitable to carry the load of traffic being generated by our attendees (the cable was being run directly next to electrical wiring inside the venue, which can interfere with network transfer rates – see #4).
  • Our team then spent the next 2 to 3 hours acquiring access to the University’s core router. Staff made trips to the nearest store where ethernet cabling could be purchased (as most of ours had been used the previous evening for set up) and subsequently installed a new run of cable from the University’s core to our NOC. Meanwhile, on the console side of the room, when connections were stable, Halo players began receiving this error message. At that point, we believed the error was being caused by the other connection problems affecting the rest of the event, and didn’t have much reason to believe or prioritize otherwise. Tournament admins were instructed to keep attendees informed by speaking to players and team captains as often as possible. I have read a few posts online from attendees who mentioned they felt early communication from our admins was sparse. In hindsight, it would have been more appropriate to make multiple overhead announcements throughout the room – and this is a learning we will take forward. In previous years we’ve hosted our event, it was easy for news to spread via word of mouth. In a bigger space like EMU’s Convocation Center, this simply isn’t as feasible. That said, we also had a designated “Tournament Info Desk” at the event, which was always available to provide players with an opportunity to ask questions as they arose. This desk was located at the NOC, and attendees could have made use of staff there at any point during the weekend.
  • While our new run of cabling to the University’s core restored internet usage for the remainder of our event (~5 PM), Halo players continued to receive the aforementioned error message. While all players could create / join custom games and see each other in custom game lobbies, the ability to actually start a game was intermittent. In certain instances, 8 players could start a custom match, but upon actually loading the game, 7 players would get booted, and only 1 player would load properly into the game. All other aspects of the game appeared functional. Players were able to play online matchmaking and campaign. We also hadn’t received any reports from other console LAN attendees regarding Xbox connection problems. It was at this point we began making further attempts to decipher the error message, and reached out to 343 Industries directly. I would also like to note it was at this point the issue left the hands of our moderators. Finding a solution was largely placed in the hands of our networking team. Some comments online have called into question whether our volunteers took appropriate initiative to address the tournament’s problems. From an attendee’s perspective, I can understand that it’s hard (as an outsider) to recognize the flow of responsibility at the event. Our moderators at certain points were in a relative holding pattern, as our networking team worked on potential solutions. During these times, moderators should have been available for questions, but otherwise were playing the same waiting game as attendees.
  • Our direct contacts at 343 did not recognize the error, and we spent the next few hours simultaneously talking with staff at 343, and testing various network configurations to try and solve the issue. I have read comments from people online who’ve questioned the technical competency of our team. The crew who runs the Gamers for Giving network is the same crew that manages the LAN network at major conventions such as PAX, so I don’t have any doubt about their qualifications. Those of you who are IT savvy have also mentioned the need to put consoles on a separate router, create a VLAN for the consoles, enable UPnP, ensure an open NAT, open all ports, etc, etc. Our networking team ran through nearly all of the various network configurations that could possibly have an affect on Xbox gameplay. At one point we even assigned each Xbox its own public IP, and placed it straight on outside, unfiltered bandwidth that didn’t pass through the University’s connection. The error message still remained constant across all consoles. Towards the end of these tests, we also sent our packet traces to 343 in an attempt to see if they might be able to find any information that could point us towards a solution.
  • Between the hours of 5 PM to 10 PM, our intention was still to work towards a resolution. While we knew the format of the tournament was likely compromised due to time constraints, we had every intention of hosting the tournament as planned within our venue. One of our lead tournament admins collected the phone numbers of each captain to begin providing updates via text message. Towards the end of the night, as finding a solution seemed less and less likely, we began to explore the idea of finding an alternate venue. Again, keep in mind: attendees could access online matchmaking, and we had no problems with any other console games within the venue. Players couldn’t consistently enter custom matches even after each Xbox was assigned an open IP. It seemed unlikely the issue resided within our venue or our network. Unfortunately, the evening was drawing late. We informed attendees that if a resolution could be found by 10:30 PM, we would at least begin competition for the top half of the tournament’s bracket, which would extend late into the night.
  • When 10:30 PM did finally roll around, we were no closer to finding an on site solution. We began formulating a hail mary play to find a venue in close proximity. One of our event sponsors, Michigan esports, frequently hosts Call of Duty tournaments at a hotel approximately 30 minutes from the Convocation Center, and proposed we attempt to move the tournament to their location. Conveniently, MES had hosted a Halo tournament at the same venue just two weeks prior, so there was reason to believe this new location could be feasible. The venue was willing to let us use their space for free, and we began making preparations to move the tournament. For team captains included in our text message thread – it was around this time that we issued a message stating the tournament would be moved to a new location. We asked players to arrive to the venue the following morning by 9 AM. Retrospectively, it may have been slightly preemptive to have issued this text message. Our team was in the midst of packing equipment, and we had not yet sent forward any volunteers to deploy a full station and test for connectivity problems. However, our intention behind the text was to issue an early communication to players that would give them enough time to prep for an early morning of tournament play. Had we waited until even later in the evening, it would have been possible more players would have missed our communication, and not have been able to arrive in time for the start of the tournament. It was more likely minimal damage would be incurred if players received a cancellation message early in the morning as opposed to a new schedule and location. Some players have noted the mixed communication here – and for that we certainly apologize. It was not our intention to cause anyone to become confused or misled by the texts. We jumped the gun on scheduling the new venue, but later felt it was an appropriate notice given the potential of it being a viability, and risk associated with informing players too late.
  • By this time, the clock was hovering between 11 PM to 12 AM. MES took some of the consoles to the new venue for testing, and we decided to keep a portion on-hand for our networking team to run some final network configurations. A few hours rolled by as we waited for MES to work with the new location to set up routers and test Xbox consoles. Ultimately, by the time testing was underway at the new venue, it was about 3 AM. Logistically, given our equipment was split in two places, it became seemingly impossible to transport our gear, move volunteers, and expect our crew members to be in proper condition to effectively moderate a tournament by 9 am. Additionally, and finally, MES ended up experiencing similar problems with the custom games system, and had instances where players were booted from the game after briefly connecting. It was at this point we decided it best to cancel the tournament, and we issued our announcement via text, email, blog post, and twitter. While the rest of our event remained unaffected, we addressed the cancellation of the tournament on our main stream the next day. We still had an event to finish, and made internal plans to address the Halo community further once the event closed (i.e. publish this article).

So what really happened in all this? To be honest, we’re still not quite sure. Our current leading theory is that we may have been the victim of a potential in-game coding bug that could have been created after the game’s latest patch, which specifically makes reference to stability and networking improvements that “Improved stability when joining custom game lobbies” and “Resolved issues surrounding multiple Xbox Ones playing on the same network.” We are the only tournament (we’re aware of) to have taken place since the patch went live, so it’s possible our system update caused the problem, but was otherwise unnoticed by the vast majority of the community playing games from home. This theory is further reinforced given MES also had issues with custom games within their home venue, despite having successfully run a tournament a couple weeks prior. With that said, we were also informed that on Sunday ~2 PM, players who had signed up to compete in our Halo tournament were able to successfully connect to custom games at a local house LAN party. Unless some type of hot fix was introduced, if other players were indeed able to consistently connect to custom games, it would seem unlikely we’d be affected by a patch. If the patch was an issue, it’s also possible that our packet traces sent to 343 in the late evening provided them with information to issue a hot fix overnight or early next day.

Whether or not we were impacted by the patch is unconfirmed and purely speculative, but we’ve not yet heard from 343 as to whether that could be true. Many of their staff members are preparing for the upcoming HCS event, so we may not be able to have full conversations for another week. We also heard various reports from players that in the evening of Saturday, similar occurrences were arising from players who later went home to test their own connectivity. Again, this is somewhat unconfirmed – and could simply be word of mouth rumors. But if true, it would further support our leading theory internally.

With all that said, I would like to take a moment to express my own thoughts on a few talking points gamers have raised via social media and Reddit. The following portion of this article is my own, sole opinion, and does not necessarily reflect the views of Gamers Outreach / Gamers for Giving staff as a whole:

We have received some inquiries as to whether we plan to host another Halo tournament online. To be entirely honest, given the context of our situation, I don’t believe that would be right for us to do. The reason is this: I believe multiplayer stability for competitive console titles is a topic the community deserves to have a dialogue about. A central theme of Gamers for Giving is the tangible, communal act of gamers gathering together for a good cause. Our event is an opportunity for gamers to share our passion for a greater good – and to do so in person. As a community, we were prevented from that happening this weekend, despite our staff’s due diligence. An online tournament would simply be lip service, and would require more resources from both our attendees and the charity. Attempting to exercise those resources (which after refunding players, would certainly be a detriment to the cause) would entirely miss the point of why we organize Gamers for Giving in the first place. It’s not just about a tournament. Online tournaments are hosted almost every day by Beyond Entertainment, MLG, ESL, etc. Any player can enter those at any time. Gamers for Giving is about an underlying purpose which is meant to inspire involvement and action. Everyone’s participation is contributing to a palpable vision. Our event celebrates gamers coming together in real life to create tangible change. That’s a big part of what makes the event so special. From our perspective, that’s what the Halo community was barred from participating in this past weekend.

On a more general level, there is something else I think is important: console LANs don’t happen anymore. The number of conditional standards TOs have to satisfy just to have a simple tournament in a local college is obstructive. I don’t believe that’s healthy for community development. Even 343 hosts the HCS tournaments on a portable dedicated server. What hope is there for a grassroots scene to flourish when a game’s developer can’t have faith in accessible, on-hand infrastructure?

Don’t misunderstand me. The HCS program and work being done by the team at 343 and Microsoft is pushing competitive Halo to heights its never seen before. It is incredible what players can hope to achieve if they demonstrate the skill and commitment necessary to compete at a high level. But without an infrastructure or stable platform functionalities that support even the humblest tournament, our community will suffer the effects of ivory tower syndrome. A culture will be created that lacks independence, and discourages the community from self-organizing offline. That last part is important. It is my belief our ability to organize for socialization, competition, and interaction offline should be amplified by what we can do online, not deterred by it.

I’ll say this in closing – Halo community: this past weekend’s Gamers for Giving was a tremendous success. Despite the Halo tournament being canceled, you share in that success with us. You are the community from which we were born. Our Halo tournament’s cancellation was due to a situation that was beyond our control during a weekend where a lot of good was accomplished. I write this post to simply let you know that, as we move forward and celebrate that success and exercise the resources the community has helped generate, we have not and will not forget about you. We hope and plan to host Halo at next year’s Gamers for Giving once again, even though the current state of the world certainly makes it difficult to do so. I hope you’ll be joining us.

  • Anon ymous

    Zach, thank you for sharing insight into your team’s effort. I can appreciate the effort that goes into hosting an event like this so it must be frustrating to hear anything negative when holistically the event was a huge success. For that I apologize…

    I won’t respond to the specific technical tasks and timeline chosen as it serves no benefit now. I’ll also not respond to the communication topic as you’ve already addressed that.

    One point I will make is to highlight the disconnect between your organization, its customers that benefit, and its customers that participate/contribute. No doubt the organization and especially the many volunteers are being altruistic and the world clearly benefits by the efforts of you and your team (which makes me feel like an ass for even commenting here). The disconnect is that many of these gamers consider GFG as a tournament first and foremost, they’re not motivated to travel great distances because of the great cause. Yes they should be but they’re not (ok some might be but it is the minority). That’s ok though because GFG can still leverage that. The players probably donate to many other great causes in their local communities. We can condemn them all we want but these are your paying customers and many great companies have failed because they thought they could change their paying customers way of thinking. The challenge is to first ensure GFG meets their needs so that GFG can then convert love of the games to the greater good for the real GFG customers that benefit by your efforts. As the gamers get older and their value systems grow many will likely join the cause by being volunteers. GFG provides significant value by educating the gaming community to think of others before themselves by helping participants make a difference (albeit as a secondary benefit).

    To back up my point, I saw blog comments like “Why would you fly out or drive long distances for small LAN like this”. The answer is obvious… because many of your paying customers were excited to play in the tournament, have some fun, network with peers, and possibly win bragging rights. The only part they achieved was networking but that had a stench to it given they were upset. The prize money helps but it can’t be the motivation for many because they invested more money than they were capable of winning. Perhaps the locals were motivated by prize money but you need more than locals to be successful.

    I agree that hosting an online tournament now is difficult and fails to meet the goals of GFG. Personally I feel the damage is already done so just move on but I suspect those registered for the tournament would say otherwise (and according to you they are). One final note about the online tournament idea, your FAQ section includes a note about how failed tournaments are handled (see link below). If your paying customers are asking for an online tournament then you could justify it by the fact your existing FAQ (Terms) indicates that GFG would. Since you’re refunding their registration fees (which I agree is the right thing to do), perhaps you can turn this in your favor by having a closed tournament for the original teams only, and have them pay a reduced entrance fee to enter to cover the prize money with some extra funds for GFG and the cause. A win-win.

    http://gamersforgiving.org/faq-contact/

    Q: What happens if there’s a technical issue that causes a tournament to be delayed / canceled?
    A: In the event of a technical issue, such as a power or internet outage, affected tournaments will take place online at a later date.

    This is probably my last comment here, and I do not expect a response. I wish GFG success going forward.

  • Jeff Leitson

    As I posted in another discussion, the gamers, many who are much more technically savvy than normal, should understand what happened, even if disappointed. Since HALO multiplayer pop up location gaming is obviously more technically challenging than the other consoles and games, I suggest a LOW TECH answer (sometimes the best tech is low tech?). Nicely solicit (GOF and Players) HALO’s parent company to provide an ON-SITE game server at your venue for the weekend, and support. This seems to me to be in their best short & long term interest. Since I am not a volunteer and just an old retired tech guy, I will say it…sometimes the problem is not ON-SITE and no matter how hard you try the “fix” is elsewhere..

    I will say what is seemingly obvious…that the weekends HALO problems reflect on the games parent company. This is not in any way an insult, or assigning fault (which we may never know for sure) but, a community challenge to them to learn from this user experience and address this for the tournament and the charitable community. We should all work together to discuss & suggest that they allocate resources (like local servers) for events such as this to support their very loyal users and avoid any chance of these issues most the time. (things still break, even onsite).

    Lastly…people, all this is very technical hardware, firmware, O/S, software, routers, switches, cables, & Internet traffic. It is a near MIRACLE when it does work perfectly. We just forget this today. I give GOF and their tech team an A++ for effort and Zach is right, it is not about (at GFG) the game itself, but about bringing the community together socially, meeting face to face, having some food & fun, some competitions, some prizes and mostly…taking the games you love to the sick kids laying in a hospital bed. This is what sets GFG apart from many other “tournaments”. Most of you attending that I have met said the same thing and made me very PROUD of all of you and that my son is one of those many volunteers..