SDCC 2017 Hotel Lottery – Analysis and Reflection

by Mark

The 2017 Comic-Con International hotel lottery is now behind us, the dust has settled, and hopefully everyone knows where they will be getting whatever sleep they manage to find during their stay in San Diego. There were big changes to the hotel lottery this year and we once again asked members of the Friends of Comic Con forum to share their experiences. That data has been crunched and collated and in this article we will discuss what effects the changes had, what has improved, what still needs work, and what questions still remain.

Method

There will be a lot of numbers listed below, but first a few comments and caveats. None of these numbers are official ones from Comic-Con International (CCI) or onPeak and we were not provided with data, special access, or any information from either organization. Our data consists of information from roughly 250 hotel request forms, including the options selected, the end result, and the times that the participant accessed and submitted the request form. We did not ask for the names of requested hotels or what specific dates were requested. Wherever possible we used form access and submission times obtained from browser caches; if only approximate times were available we accounted for this in the analysis. This data was provided voluntarily and, since it does not represent a random sample taken from all participants, it is possible that our summaries may differ from the general experience.

Comparisons with previous years should be made with caution as, in addition to the changes that were announced, there are several important things that we don’t know, such as whether there were more requests made this year or how the number of available hotel rooms may have differed.

Before the Lottery

Instructions for the lottery were posted on Comic-Con’s website, including a list of hotels, a link to a page that tested browser functionality, and a link to an example request form. The example request form allowed participants to familiarize themselves with the form and practice. There were a number of new elements in the request form this year: an optional field to enter a CCI member id, separate lists for requesting downtown and non-downtown hotels, and a new set of options to be used if you did not get a requested hotel. The written documentation this year was much improved: in particular, the description of how randomization was to be used was clear and consistent. Likewise, the new options in the form were detailed well in advance, allowing people time to ask questions and adjust their strategies. The quality of statements made by onPeak employees was mixed; for example, some participants were told that form submission times mattered while others were told that it did not. However, onPeak was able to clarify some important questions, such as if people would be considered for their listed downtown hotels before any non-downtown hotels.

Day of the Lottery

This year, the waiting room link was not posted online, but only sent to people who had obtained a badge for 2017. In addition to those who obtained badges through the regular sales, this included groups like press, professionals, and contest winners. However, the official instructions stated that neither a badge nor a CCI member id was required to take participate in the lottery.

Participants were able to enter the waiting room an hour before the start of the lottery. Once it started, participants were taken to a page that refreshed periodically, with a graphic of a person running along a progress bar, which gave a rough indication of how close the session was to being directed to the request form. The replacement of last year’s queue number with the running person was an important improvement, as the queue numbers were not explained before or during the 2016 lottery and many participants assumed that they showed a request’s overall place in line, while it later it appeared that they were actually one’s place in a large number of smaller queues. This year, the lottery ran smoothly, with no reported unresponsiveness or server issues. There were, however, a few glitches with the request form. Some people reported being taken back to the start of the request process after submitting their form. A few others were unable to choose a specific hotel from the downtown list, although people who refreshed the form said that the problem went away. Coronado Island hotels were not listed in either the example or actual request form, despite being part of the posted list of hotels. Their absence on the example form allowed some people to adjust their lists accordingly, but others were taken by surprise. A form submission usually resulted in a generic page indicating success, but several people did not get this page and were uncertain whether or not their request had gone through.

This year’s options provided more flexibility and participants made positive comments about the changes overall. Participants were now able to make separate lists for downtown and non-downtown hotels, up to six of each, and had the ability to only choose from one of those two pools, if they desired. Participants also choose from a richer set of options to be used if they were not assigned a requested hotel: assign the hotel closest to the convention center, the hotel with the lowest price, any hotel on the shuttle route or to discard the request completely. In the data we collected , 82% of requests listed only downtown hotels, with 91% of these listing the maximum of six. 84% of requests selected the “closest to the convention” option, 7% requested a hotel on the shuttle route, 2% requested the hotel with the lowest price, and 7% chose to have their request discarded. 98% requested to be waitlisted. Overall, many requests had a similar set of selected options, with 64% of participants listing 6 downtown hotels, 0 non-downtown hotels, choosing the closest to the convention option and asking to be waitlisted. Many participants seemed to take advantage of the removal of the dependency on form submission time; on average, participants took 3 times as long to fill out their forms as compared to last year. Also, despite the new CCI member id field not being a required field, roughly 70% of requests filled it out. For the most part, these new options worked as expected, with a few wrinkles that we will mention when we discuss the results below.

Another major change this year was how and when results were communicated. Hotel assignments were sent out in two groups, with participants in the first group notified on May 1st and participants in the second notified on May 8th. Both groups had until the following Friday to place a deposit or their assignment would be discarded, though the deposit was fully refundable through May 15th. Waitlist notifications were sent after the two groups had been notified.

Results

It does appear that browser sessions were directed to the request form at a faster rate than last year. This table shows a rough breakdown of form access times observed this year and last.

Access time 2017 2016
Under 1 minute 28% 21%
1 to 2 minutes 28% 21%
2 to 3 minutes 12% 14%
3 to 4 minutes 12% 9%
4 to 6 minutes 11% 16%
over 6 minutes 9% 19%
Total 100% 100%

We don’t know why things happened faster this year. It’s possible that a higher rate was used due to confidence in the servers being able to handle the load, or in reaction to a larger number of total sessions. In a notable departure from previous years, no requests were flagged as duplicates and discarded. This lack of duplicate detection likely played a role since it would have disproportionately affected early requests in 2016.

As in 2016, the form access time was the primary factor in determining what result a request received, and also into which notification group a request was placed. In general, looking at results by access time shows some clear trends, but also a good deal of ambiguity. Part of this is uncertainty is due to the lack of accuracy in some of the access times in our data. Also, there is the likelihood that a session’s actual place in line is something that resides only in onPeak’s servers and correlates with, but does not exactly match, a participant’s observed access time. (A browser session has to refresh in order to be redirected, there are network lags and page load times, etc.) Given all of this, in addition to the speed with which sessions were taken to the form, a general trend may be the best we can hope to detect with our self-reported participant data.

These two tables show how access times related to the notification groups. The first shows the percentage of requests in each time range that were in each group.

Access Time Group 1 Group 2 Total
Under 1 minute 90% 10% 100%
1 to 2 minutes 57% 43% 100%
2 to 3 minutes 6% 94% 100%
3 to 4 minutes 0% 100% 100%
4 to 6 minutes 2% 98% 100%
over 6 minutes 0% 100% 100%

The second table shows the percentage of requests in each group that fell into the time ranges.

Access Time Group 1 Group 2
Under 1 minute 59% 8%
1 to 2 minutes 39% 33%
2 to 3 minutes 1% 25%
3 to 4 minutes 0% 23%
4 to 6 minutes 1% 9%
over 6 minutes 0% 2%
Total 100% 100%

Overall results broke down as follows. This year we tracked how many requests received their #1 choice of hotel, along with results corresponding to the new non-downtown list. This table only tracks the result of each request, not what was actually booked, so rooms that were declined in Group 1 and re-matched in Group 2 will be represented twice.

Result Overall percent
Top choice 29%
Other listed downtown hotel 15%
Other downtown hotel 19%
Listed non-downtown hotel 1%
Other non-downtown hotel 18%
Waitlisted 18%

This next table shows the breakdown of results within each notification group, (waitlisted requests were not part of either group and therefore were not included here.)

Result Group 1 Group 2
Top choice 56% 12%
Other listed downtown hotel 22% 11%
Other downtown hotel 21% 28%
Listed non-downtown hotel 1% 1%
Other non-downtown hotel 0% 48%
Total 100% 100%

And finally, here are how the overall results look by access time.

Access Time Top choice Other listed downtown
hotels
Other downtown
hotels
Listed non-dt
hotels
Other non-dt
hotels
Waitlisted Total
Under 1 minute 69% 18% 12% 1% 0% 0% 100%
1 to 2 minutes 30% 26% 38% 2% 1% 3% 100%
2 to 3 minutes 0% 6% 39% 3% 41% 11% 100%
3 to 4 minutes 0% 0% 2% 0% 77% 21% 100%
4 to 6 minutes 0% 5% 0% 0% 37% 58% 100%
over 6 minutes 0% 0% 0% 0% 18% 82% 100%

In all of the tables, there is a strong, albeit imperfect, trend toward better results and a greater chance of an earlier notification with lower access times. Keeping in mind the previously mentioned warnings about comparisons with 2016, it does seem that the changes in the process resulted in improvements in matching more requests with downtown hotels, (both listed ones and in general,) and in reducing the percentage of waitlisted requests. (Very few people listed any non-downtown hotels, so any low numbers for that result are due to that.)

Issues

While we understand that no process is perfect, a few participants experienced some hiccups. For example, if a participant listed non-downtown hotels and ended up at the tail end of Group 1, then they may have been assigned a listed non-downtown hotel, whereas if they had only listed downtown hotels, they likely would have been matched to a downtown location not on their list. Similarly, a request at the tail end of Group 1 may have received a downtown hotel they did not list, while a request at the head of Group 2 might have received a top pick (a returned room that had been allocated, but not booked, during Group 1;to be fair, in last year’s process, such rooms would have just gone into the waitlist pool.) People who reported being taken back to the start of the process when submitting their form did not appear to do any worse than those for whom the form worked properly.

The popular “closest to the convention center” option also had some curious outcomes. The US Grant, Westin San Diego, and Westin Gaslamp hotels were listed on the onPeak site with incorrectly short distances to the convention center, making them seem to be the closest hotels. It appears that this resulted in a large number of requests being matched to those hotels around the time when results shifted from listed to general downtown hotels. Additionally, the one Old Town hotel in the lottery is also one of the closest non-downtown hotels to the convention center, but it does not have shuttle access. Many participants who had used the “closest to the convention center” option but had later access times were matched to that hotel instead of a hotel on the shuttle route. This would have been exacerbated if the Coronado Island hotels had been included in the lottery, as they are also some of the closest non-downtown hotels. This is something of which participants should be aware in future years.

A few people reported reporting getting listed downtown hotels in Group 1, despite having very late access times. It is difficult to know what to make of these outlying results. The presence of a lag between the time that a request was eligible to go to the form and the time it actually got there allows the possibility of extremes in that lag. It should also be noted that other reported oddities, such as people getting different roommates from those they had entered on their submission form, happened at roughly the same level.

A more common issue was people getting a different room type than requested, suggesting that the process prioritized hotel selection and location over room type. onPeak did allow people to transfer reservations, reducing the impact for some.s. Overall, people had positive comments about working with onPeak during the exchange process.

All in all, the use of multiple notification groups this year appears to have resulted in requests being matched to downtown rooms more efficiently and in reducing the number of waitlisted requests. So despite the multi-week process, and the additional waiting and worrying, it appears that more people were able to find an acceptable room without having to go through the waitlist free-for-all.

A place where more details and clearer communication is needed is with the waitlist process. People that asked onPeak for details about the waitlist were given wildly differing answers, with the main confusion centering on who would be eligible to participate. Some people were told that only requests that had not received a Group 1 or 2 notification would take part in the waitlist and this was partly accurate, as those were the only people that received a waitlist email. However, people that had received a Group 1 or 2 notification had also received a link to the same pool of hotel room inventory that was used for the waitlist, but they were able to access it before the waitlist emails had been sent out. Some people also reported issues with having to reset or bypass a password requirement on the onPeak website, and having a hotel room disappear from inventory while they were in the process of booking it.

Clearer instructions on how the waitlist process works and a consistent set of answers for onPeak employees to give to frequently asked questions would also help the many people who found themselves in the position of having to make a decision without knowing what the ramifications would be, for example whether to accept a far away hotel room in Group 2 or to take one’s chances in the waitlist. This is especially important this year since cancellation penalties are stricter, with people that find a better hotel listed on onPeak’s site having to pay a penalty to cancel their previous reservation and make a new one.

Another suggestion for improvement next year is to show a summary of the submitted request form, either on the submission page or delivered later via email. This would provide many people with a bit more peace of mind and also let them know if a confusing result was actually due to a mistake they had made on the form.

Conclusion

The new form options provided greater flexibility and allowed people to better plan for and prioritize their choices, but did result in unlucky situations for some participants and left the process more vulnerable to bugs. The speed with which the lottery happens and the uncertainty in how the time you get to the form relates to your actual position in line make it difficult to know when to change your hotel lists and options without disadvantaging yourself. It also generates some confusion over results, with many people not getting a room they had requested, while someone with a seemingly later time did. However, our aggregated results do match CCI’s statements and show form access time aligning with results. While we may never know exactly what is going on behind the scenes, the simplest likely explanation is that each request does have a place in some absolute ordering, but we the users might never be able to see exactly what it is.

There were some clear improvements this year, and we hope to see them build on this progress and continue to improve in the future. The lottery would greatly benefit from better accuracy and more consistency in the information given out by onPeak, and from a more thorough explanation of the waitlist process. Also, the ability to change to a more preferable hotel in the remaining inventory without a penalty would allow some people the opportunity to get a better result than they got in the lottery. On the plus side, the best way to improve the waitlist is to keep people off of it in the first place, and we saw a much smaller percentage of waitlisted requests as compared to last year, along with a higher percentage of participants getting both listed hotels and downtown hotels overall. The migration from a system based on submission time to one using randomization has been difficult, but there is a definite sense that the CCI and onPeak are listening to participant feedback and that the process is evolving into something more stable.

Questions? Comments? Please join the conversation on the forum, here.