Real-Time Learning: The Art of Getting Better as We Go

7 min readOct 29, 2024

A 4-minute read about knowing if your program is working (while it’s happening)

At the bottom of the page, find sample questions you can use during your sessions.

“Do the best you can until you know better. And when you know better, do better.”
- Maya Angelou

Maya Angelou’s words have become our mantra in evaluation work. Too often in youth development, we witness a familiar pattern — waiting until programs end to evaluate their impact. Post-program surveys, end-of-year reports, final focus groups — all valuable tools, but often too late to benefit the young people currently in our programs.

The Power of Real-Time Feedback

Those evil mega-corporations we’re all addicted to have mastered the art of real-time feedback. Amazon won’t even let you finish opening your package before bombarding you with “Rate your purchase!” Uber asks you to rate your driver before you’ve closed the car door, and Netflix constantly asks if you’re “still watching.” Sure, this is eerily like the Black Mirror Nosedive episode, and they’re doing it to squeeze every penny out of their algorithms. But…these profit-driven giants are obsessively fine-tuning their services. Meanwhile, we in youth development — who are focused on creating safe, engaging spaces where young people can grow and learn new skills — often wait until the end of the program to figure out if our programs have worked.

Creating effective feedback loops takes real work. It means pushing yourself to reflect and adjust when you’re exhausted after a session, when your to-do list is screaming for attention, and when it feels easier to just stick with the original plan. But that persistent commitment to getting better is where the real transformation happens.

Quality Frameworks: Like GPS for Your Program

You don’t have to reinvent the wheel when it comes to defining quality. Most likely, your gut instincts about what makes a program great are spot on — you just might not have the language to articulate it yet. That’s where established frameworks come in. Take the Weikart Center’s pyramid of program quality. It’s like someone took all those things you inherently know make a program successful — the importance of safety, supportive relationships, engagement, and skill-building — and organized them into a clear roadmap for improvement. The framework doesn’t replace your expertise; it amplifies it, giving you precise language and structure to describe what you already know in your bones about quality programming.

The Full Picture of Feedback

Here’s a truth we’ve learned the hard way: when it comes to feedback, the squeaky wheels don’t tell the whole story. It’s natural to be swayed by the participants who regularly speak up — their feedback is immediate, detailed, and right there in front of us. But relying solely on these vocal few can give us a skewed picture of how our program is really landing. That’s why it’s important to create multiple channels for feedback. Mixed methods isn’t just about collecting different types of data — it’s about weaving together multiple ways of understanding what’s happening in real-time:

Quick post-session surveys (5 minutes or less) let you hear from everyone — not just the most vocal participants. Using a mix of closed-ended questions (like rating scales) helps you understand overall group sentiment, while open-ended responses reveal the ‘why’ behind those ratings, giving you a more complete picture of how your session landed.

Individual reflection exercises reveal insights from youth who have brilliant ideas but prefer to share them one-on-one.

Observation using modified Youth Program Quality Assessment tools. Never underestimate what you can learn by simply walking around the room. Observing group dynamics and watching how participants engage with activities tells you volumes about what’s working (and what’s not).

Interactive projects and short assignments allow youth to demonstrate their learning and serve as windows into how well participants grasp the content.

A Low-stakes quiz can reveal whether you’re hitting your learning objectives.

By mixing up our assessment methods, we get to hear from everyone—not just those who raise their hands first. It’s about building a complete picture, not just the most visible pieces.

The Feedback Loop

Collecting data is only half the battle. The real magic happens in what we call the “feedback loop.” After most sessions:

Review feedback immediately (not next week, not next month)
Share what you heard WITH participants
Make immediate changes where possible
Explain why, when changes aren’t possible

Analyzing Feedback in Bite-Size Pieces

There are countless ways to do this. But try to come up with a simple way to summarize the feedback. One example that has worked for us:

Continue: What’s working? Let’s keep that.
Improve: What could be better? Let’s fix that.
Next Steps: What actions are we taking? Let’s be specific.

Participatory Approach

This isn’t just about improving programs (though it does that, too). It’s about showing young people that their voices matter and that when they speak up, change happens.

Looking Ahead

This blog only skims the surface. There are countless ways to implement real-time assessment, and we’re constantly learning new approaches. That’s why we’re ending with an invitation: If you have methods that work in your context, share them with us: Chicagoyouth@praxisinst.com

Real-Time Assessment Tools

Looking for specific ways to track progress in your program? Here’s a sample menu of indicators we’ve found useful for gathering real-time feedback across different areas of youth development.

Express Care

If I have a problem, I know my Facilitator(s) will help me.
My Facilitator(s) really listens to me.
When I work hard, my Facilitator(s) encourage me to keep going.
My Facilitator(s) do things that make me feel like I matter.
My Facilitator(s) shows me they enjoy being with me.
If I am treated unfairly, my Facilitator(s) say or do something to help.
My Facilitator(s) give me chances to be a leader.
Facilitators and staff in the program encourage my development
I have a best friend or close friend in the program

Expand Possibilities

My Facilitator(s) help me think of different possibilities for my future.
My Facilitator(s) help me discover new things that interest me.
My Facilitator(s) introduce me to other adults who offer resources or support that I value.

Share Power

My Facilitator(s) treat me with respect.
My Facilitator(s) consider my ideas when making decisions.
If I have challenges, my Facilitator(s) work with me to find a solution.

Provide Support

My Facilitator(s) make it clear what behaviors are acceptable and not acceptable.
My Facilitator(s) teaches me how to ask for help when I need it.
My Facilitator(s) help me figure out how to do things that are new or challenging to me.

Expectations

I know what is expected of me
During most sessions, I feel prepared and in control.
I have the materials and equipment I need to do quality work

Youth Voice

My opinions seem to count
My Program provides me opportunities to choose work on things based on my interests.
My Program seeks my ideas and feedback
I have opportunities to lead or design activities
I help make decisions or rules for the program
My Program asks the group what rules and norms we should follow as we work together
My Program gives us choices in how we work together.
My Program works with me to design or lead activities.

Youth Participant Engagement

My Program uses interesting & different types of activities
My Program uses interesting & different types of media and technology
I am given opportunities to work with my peers
My Program asks questions that make me think in different ways

Clarity

My Program gives me clear & specific feedback to help me improve
My Program gives clear instructions that I understand
My Program makes me feel comfortable to ask questions

Making Connections

My Program encourages me to make connections with what I am learning and my own life experiences
What we do at My Program is important to me

Challenge Growth

My Facilitator(s) expects me to do my best.
My Facilitator(s) challenges me to try things that are difficult for me.
My Facilitator(s) hold me responsible for what I do and say.
When I make mistakes, my Facilitator(s) show me how I can learn from them.
My Facilitator(s) motivates and encourages me to try hard
My Facilitators(s) make me think in ways I have never thought before
My Program helps me learn how to be patient with others
My Program helps me to handle stress
When my solution to a problem is not working, I try to find a new solution
My Program does not let me give up even when work is hard.
My Program helps me set personal goals to achieve

Group Management

My Program leads activities that create a positive environment
My program holds us accountable for the group norms in a fair and consistent manner.
My Program helps me when I have a problem
My Program has high standards for everyone to respect and support one another

Open Ended Qs

What 2 -3 of the most important things you have learned this past session?
What are 1–2 things you have enjoyed the most about your experience?
What is one thing you are still thinking about?
What have been some strategies and practices you plan to use immediately?
What are you most proud of?
What are some key challenges you are facing?
What are you excited about for this week?
What are you worried about?
During the program you feel most energized when…
During the program you feel most frustrated when…
What have you done lately to relieve stress and focus on your own mental health
Is your meeting schedule allowing you to accomplish what you want?
Do you have other strengths or skills that you feel are not currently being utilized? What are they?

Extra special thanks to Leslie Beller @ MHA Labs.