Hackathon Post-Survey Template

Alexander Nolte, Eindhoven University of Technology, The Netherlands; Carnegie Mellon University, Pittsburgh, PA, USA, a.u.nolte@tue.nl Anna Filippova, Snowflake, San Francisco, CA, USA James D. Herbsleb, Carnegie Mellon University, Pittsburgh, PA, USA

This template contains a collection of measures to assess the perception of hackathon participants after an event has ended. The measures included in this template cover individual aspects such as a participant’s motivation to participate, prior hackathon experience, programming experience, and demographics. They also cover team aspects such as a participant’s perception of the team composition, the team process, and the team’s project. In addition, the template contains measures related to a participant’s perception of the community that organized the event and the hackathon itself.

The measures included in the template are based on previously validated scales. The respective sources are referenced next to each scale. The scales included here have been utilized as part of the data collection for various studies of hackathons that have been published in various venues in the domains of software engineering [1, 3, 8, 9], human-computer interaction [2, 5, 6], and beyond [4, 7, 10].

We do not suggest using all measures contained in this template for each event. In fact, measures are often only suitable for hackathons that focus on specific goals (e.g., programming experience might only be relevant for a programming-related event). Moreover, there are also measures that assess very similar aspects of an event but in a slightly different way (e.g., the participant’s overall satisfaction with an event and their intentions to participate in a similar event again in the future). We thus encourage whoever uses this template to create a survey that fits their specific needs. This includes adding scales that might be important for a specific event and removing scales that are not perceived to be relevant. However, when adapting the survey, please make sure to keep the scales intact. Altering them might compromise their integrity. In addition, we suggest including items that are marked with a hash (#) in every hackathon survey. We found these helpful for most, if not all, hackathon events.

If you utilize this instrument to study an event, we would appreciate it if you would be willing to share anonymized responses with us. Please contact the main author for further details. A downloadable and citable version of this planning kit is available on Zenodo:

Nolte, A., Filippova, A., & Herbsleb, J. (2025). A survey instrument for hackathon organizers and researchers. Zenodo. Nolte et al. (2025)

Survey Cover Page¶

As an attendee of the [HACKATHON_NAME] hackathon, you are invited to complete the following survey. This hackathon is organized by [HACKATHON_ORGANIZERS]. The main purpose of this survey is [SURVEY_PURPOSE].

The questions are mostly multiple choice / agree-disagree and will take approximately [COMPLETION_TIME_IN_MINUTES] minutes to answer. The risks that are associated with this survey are no greater than those ordinarily encountered in daily life. Your decision regarding whether or not to participate in this study will not result in any loss of benefits to which you are otherwise entitled. Your participation in this survey is voluntary, and you may discontinue participation at any time. Your responses will be de-identified and will remain confidential to the study team.

Anonymized responses to this survey might also be shared with a group of researchers at Eindhoven University of Technology.

If you have any further questions about the survey, please contact [ROLE] [HACKATHON_ORGANIZER_NAME] (HACKATHON_ORGANIZER_EMAIL).

I am age 18 or older, I have read and understood the above information, and I wish to participate in the research and continue with the survey.

• Yes (continue the survey)

• No (end the survey)

Section 1: Previous Experience and Motivation¶

The following questions are related to your prior hackathon experience and your motivations to participate in this event.

1. (#) How many hackathons have you participated in the past? If you cannot recall the exact number, please provide a rough estimate.

   • Short Answer (Number)

2. (#) To what extent was your decision to participate in this hackathon motivated by... (scale, [1, 2])

3. (#) If your participation is motivated by other than the above reasons, please describe additional motivations below.

   • Short Answer (text)

Section 2: Teamwork During this Hackathon¶

The following questions are related to your perception of the team you were a part of during this hackathon.

1. (#) What was the name of your team / project? (mandatory)

   • Short answer (text or number)

2. (#) How many people were in your team (including yourself)?

   • Short answer (number)

3. (#) Did your team use a code repository? If yes, please provide the URL of that repository below.

   • Short answer (URL)

4. (#) Did your team use any other online resources to e.g. create documentation? If yes, please provide the URL of that resource below.

   • Short answer (URL)

5. (#) Was there a team leader? A team leader is someone who provides guidance, instruction, direction and leadership to the team.

• Yes, I was the team leader.

• Yes, someone else was the team leader.

• No, there was no clear leader in the team.

6. (#) How do you estimate your programming experience compared to your team members? ([11])

• Very inexperienced

• Inexperienced

• Comparable

• Experienced

• Very experienced

7. (#) How well did you know your team members? (scale, [12])

Section 3: Teamwork During this Hackathon Continued...¶

The following questions are related to your perception of the team you were a part of during this hackathon.

1. (#) Would you describe your team process as more... (scale, [1])

2. (#) Please indicate your level of agreement with the following statements related to your GOALS as a team. (scale, [1])

3. (#) Please indicate your level of agreement with the following statements about your TEAM. (scale, [1])

Section 4: Your hackathon project¶

The following questions are related to your perception of the project you worked on during this hackathon.

1. (#) Please indicate your level of agreement with the following statements related to your SATISFACTION with your project. (scale, [1])

2. Please indicate your level of agreement with the following statements related to the USEFULNESS of your project. (scale, [13, 14, 15, 16])

3. Please indicate your FUTURE INTENTIONS related to your hackathon project. (scale, [13, 14, 15, 16])

4. Please indicate your level of agreement with the following statements related to your ABILITY to continue working on your hackathon project. (scale, [13, 16])

Section 5: Community relationship¶

The following questions are related to your perception of the community that organized this event.

1. Please indicate your level of agreement with the following statements about your RELATIONSHIP with the community that organized this event. (scale, [17])

2. Please indicate your level of agreement with the following statements about your PERCEPTION of the community that organized this event. (scale, [18])

Section 6: Hackathon experience¶

The following questions are related to your perception of this hackathon.

1. (#) How do you feel about your OVERALL EXPERIENCE participating in this hackathon? (scale, [14, 19])

2. Do you plan to participate in a similar event in the future?

• Definitely not

• Probably not

• Might or might not

• Probably yes

• Definitely yes

3. (#) What did you like about this hackathon?

   • Long answer (text)

4. (#) What would you wish to be different about this hackathon?

   • Long answer (text)

Section 7: Demographics and individual background¶

The following questions are about your personal background.

1. (#) How old are you currently?

• 18 to 24

• 25 to 34

• 35 to 44

• 45 to 54

• 55 to 64

• 65 to 74

• 75 or older

• Prefer not to say

2. (#) Are you...? (Can be replaced by a text entry field: “If you feel comfortable, we invite you to share your gender with us.”)

• Female

• Male

• Non-binary

• Prefer not to say

3. What is the highest level of formal education that you have completed until now?

• High school diploma or GED

• Some college

• Associate and/or bachelor’s degree

• Bachelor’s degree

• Professional degree

• Master’s degree

• Doctorate

• Prefer not to say

4. (#) Do you consider yourself a minority? (For example in terms of background, gender, expertise or in another way)

• Yes

• No

• Prefer not to say

References¶

1. Filippova, A., Trainer, E., & Herbsleb, J. D. (2017). From diversity by numbers to diversity as process: supporting inclusiveness in software development teams with brainstorming. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) (pp. 152-163).

2. Nolte, A., Pe-Than, E. P. P., Filippova, A., Bird, C., Scallen, S., & Herbsleb, J. D. (2018). You Hacked and Now What? - Exploring Outcomes of a Corporate Hackathon. Proc. ACM Hum.-Comput. Interact., 2(CSCW), 129:1-129:23.

3. Pe-Than, E. P. P., Nolte, A., Filippova, A., Bird, C., Scallen, S. & Herbsleb, J. D. (2019). Designing Corporate Hackathons With a Purpose. IEEE Software, 36(1), 15-22.

4. Pe-Than, E. P. P., & Herbsleb, J.D. (2019). Understanding Hackathons for Science: Collaboration, Affordances, and Outcomes. In: International Conference on Information (pp. 27-37).

5. Affia, A. O., Nolte, A., & Matulevičius, R. (2020). Developing and Evaluating a Hackathon Approach to Foster Security Learning. In: Collaboration Technologies and Social Computing.

6. Nolte, A., Hayden, L. B., & Herbsleb, J. D. (2020). How to Support Newcomers in Scientific Hackathons - An Action Research Study on Expert Mentoring. Proc. ACM Hum.-Comput. Interact., 4(CSCW1), Article 25 (May 2020), 23 pages.

7. Powell, J., Hayden, L. B., Cannon, A., Wilson, B., & Nolte, A. (2021) Organizing online hackathons for newcomers to a scientific community – Lessons learned from two events. In: Proceedings of the 6th International Conference on Game Jams, Hackathons, and Game Creation Events (pp. 78–82).

8. Affia, A. O., Nolte, A., & Matulevičius, R. (2022). Integrating Hackathons into an Online Cybersecurity Course. In: 44th International Conference on Software Engineering: Software Engineering Education and Training.

9. Imam Mahmoud, A. S., Dey, T., Nolte, A., Mockus, A., & Herbsleb, J. D. (2022). One-off events? An empirical study of hackathon code creation and reuse. Empirical Software Engineering, 27.

10. Medina Angarita, M. A., Kolnes, M., & Nolte, A. (2023). Startup creation beyond hackathons – A survey on startup development and support. In: International Conference on Software Business (pp. 205-221).

11. Siegmund, J., Kästner, C., Liebig, J., Apel, S., & Hanenberg, S. (2014). Measuring and modeling programming experience. Empirical Software Engineering, 19, 1299-1334.

12. Janssen, J., Erkens, G., Kirschner, P. A., & Kanselaar, G. (2009). Influence of group member familiarity on online collaborative learning. Computers in Human Behavior, 25 (1), 161-170.

13. Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50, 179-211.

14. Bhattacherjee, A. (2001). Understanding information systems continuance: An expectation-confirmation model. MIS quarterly, 351-370.

15. Eriksson, K., & Nilsson, D. (2007). Determinants of the continued use of self-service technology: The case of Internet banking. Technovation, 27 (4), 159-167.

16. Liao, C., Chen, J. L., & Yen, D. C. (2007). Theory of planning behavior (TPB) and customer satisfaction in the continued use of e-service: An integrated model. Computers in human behavior, 23 (6), 2804-2822.

17. Ellison, N. B., Steinfield, C., & Lampe, C. (2007). The benefits of Facebook “friends:” Social capital and college students’ use of online social network sites. Journal of computer‐mediated communication, 12 (4), 1143-1168.

18. Ellemers, N., Kortekaas, P., & Ouwerkerk, J. W. (1999). Self‐categorisation, commitment to the group and group self‐esteem as related but distinct aspects of social identity. European journal of social psychology, 29 (2‐3), 371-389.

19. Spreng, R. A., MacKenzie, S. B., & Olshavsky, R. W. (1996). A reexamination of the determinants of consumer satisfaction. Journal of marketing, 60 (3), 15-32.