We love walls of text here! Compared to the previous ones, this one is written after my vacation, and just after torturing myself by reading research papers for two hours. I’ll decompress by writing a bit about Castagne and one of the subsystems of the editor to allow continuous improvement of the engine!

Metrics and Feedback

That’s right, I’m making spyware now. I’m going to gather every keystroke and network packet on your computer to sell it back to LLMs. Just need to find a way to add blockchain in there and raise a gorillion dollars in venture capital. Of course it’s a joke (although I do remember someone asking in the Castagne discord if it was a virus lol, as if I was gonna say “oh yeah dude you got me”), but after talking about communication from me to you and how to handle beginners, this update’s subject is about the communication from you to me!

A big part in improving software is feedback. I love getting multi-page google docs telling me how the software can improve (thanks to those that send them to me!), bringing in new ideas and refining existing ones. What you do with said feedback is another question, but to do that you need to actually have some, which is fairly rare in practice. So, how do we currently get feedback from users? (not counting bugs and the like, which while these are feedback it’s more of a technical kind)

• One way is the spontaneous feedback: someone having an idea, or trying out the engine, and giving their opinion and experience. Happens fairly rarely, it’s usually either after the first couple hours or a month or two of use. It’s a snapshot in time, fairly rare but gives some qualitative insights.
• Another way is the surveys, which I did twice now. That’s where I get most of my data, as the questions help people formulate thoughts and I can get answers from people already using the engine daily or following closely. They represent a fair bit of effort to push through and happen infrequently.
• Quick informal feedback when I announce something or ask a simple question in an update, usually helps correct shots before they go live. The number of interactions on my posts on social media and Discord also give a quick view into how liked something is.
• Indirect feedback by observing what people have trouble with and if patterns emerge. This is a very honest form of feedback, but also very imprecise and disorganized.
• Website stats. I’ve got no cookies or tracking on the website, but I can gather how many times a page is accessed. Mostly relevant is the number of downloads, and the number of checks for updates, which give me a quick idea on the pulse of the community.

As you can see, getting feedback is not an easy task. There’s a lot of filters, and it requires a constant pulse on the community. There’s also a few MAJOR flaws in these methods:

• Feedback is infrequent, and has no structure over time. It’s hard to see the evolution of a question.
• Feedback is filtered through the lens of the questions I ask, which will color what people talk about and their answers. They also answer from memory and thus can be somewhat inaccurate.
• And as a major point: no information on the users that fall through. Giving feedback requires going through downloading the engine, trying it out, joining the discord, and giving their opinion to me. Understandable, very few people will go through that whole process, and it might not even cross their mind.

A lot of this can be traced back to not having an actual plan on this lol, which is honestly very okay. The main aim of the software is to be functional first and foremost. Still, having more data, both objective and subjective, in a more structured way, and at higher frequency would be great, and allow finding out a lot of hidden pain points and paper cuts of the software, or seeing how tutorials help people, etc. An integrated system will also help understand where the silent users or the ones that drop out early go. From what I remember of the stats, there’s like 10-15 new users on the Discord each month (vibes only) while the software is downloaded like 500 times a month (varies but it’s around that part).

Therefore, with that in mind, I want to add software usage metrics inside of Castagne. Stuff like seeing which help rubrics are most used, heatmaps of the interface, what tutorials and genres are used… All of that should give a stronger view into the actual use of the software and its potential improvements, and thus give a third layer after me dogfooding the engine and invested users giving their opinion. However, I don’t want this to be predatory, as it goes against my whole worldview. Data is not going to be shared automatically: this is purely opt-in and respects the user.

This does limit the amount of users that get caught in it, but I hope the more respectful approach will lead to people being more willing. The actual mechanism of sharing leans into this through the upcoming Pulse Survey: as opposed to doing one survey every whenever that takes a long time to answer, a small survey is kept and updated in the engine and prompted for every X weeks. The idea is that the survey takes very little time to answer but also allows space for open ended questions. Answering the form also optionally shares your metrics, putting you in control and allowing both quantitative and qualitative data. Questions are cycled in or out over time depending on what I want to know.

One of the aims here is transparency and privacy: I don’t want to get too much information, the idea is not to be able to track actual games through it. Beyond the fact that some are under NDA, this would be disrespectful and I would not like that to be done to me. I also want to avoid a lot of that personal data management by anonymizing it, I just want to know some details on how the engine is used and can be made better, not see what anyone specifically is doing. Exact details on what metrics to gather in a respectful way and how much information can be had on a user will have to be determined and discussed. One major aspect is going to decide if a user should be able to be tracked from one submission to the next to see their evolution: very interesting data, but might go a bit too far even if anonymous.

Under this system, that means that metrics are going to be gathered in a few key ways:

• First off, basic website statistics to get the number of downloads or daily users. This is not personal data, it’s a classic phone call home you can already disable and is gathered through basic server administration. Some little changes on the software itself such as only checking for updates once a day are going to allow this first layer to get an idea of base traffic and usage.
• After the first few moments of use, a special early prompt allows early onboarding feedback and asks for automatically sending metrics. This is a first check to catch a lot of casual users who might not use the forms and/or drop early. Finding a balance of not making this too obnoxious and not happen too early will allow for a good source of information and quick catches of people who might drop out.
• Then you have the Pulse Survey coming into action, for invested users to keep giving feedback quickly and regularly. This is the community step: people who answer are likely to be regular users of the engine, even if not always talking in the community. This is the core of the system: how to make the software better for its main users.
• Finally, some extra tracking of the indirect help requests. Having an actual strategy there can get data outside of the normal channels by catching what people are actually struggling with. One of these is the “Help Me!” button in Castagne itself, which is going to be a recurring theme.
• This doesn’t preclude the previous forms: big survey from time to time, quick informal questions, people sending feedback spontaneously… The whole system just helps put that top of mind and engages more users into that loop.

While just like the tutorials this is very much not a core system for the engine, it’s something I want to implement early to get the best insights and integration with the rest of the engine. Bugs also get caught earlier, and considering how it affects early users, this is important.

What do you think? Does this seem like a proper strategy? What data would you not want to share specifically? Next time we’re talking about the Community itself!