Criterion Recommendation System: Looking for Testers

[cdnchris]

While rebuilding the site last year I decided to implement a few other features since the new tech stack would allow me to do them, and one of those was a recommendation system for Criterion releases. It's been a lot of work, but it's finally getting very close to what I had originally envisioned, and I'm now at a point where I need others to help test it.

I'm looking for a handful of users who would be interested in trying it out.

If you're interested, please continue reading.

The current plan is to open it up within the next 2 to 3 weeks. It won't be complete and a number of features will still be missing, but I want to start testing the core recommendation/scoring system as soon as possible.

I'll get more into what it does and how it works once it's ready to be tested, but the basic idea is that it's designed around exploration or "expanding one's horizons," whether you're a seasoned cinephile or someone just getting into film. A user enters their current interests (directors they like, genres, movements, etc.) and from there the system makes recommendations based on how far outside your comfort zone you want to go.

It's not a "you like Wes Anderson so here's every Wes Anderson movie in the collection, conveniently available in one big box set" type system, and the results can differ quite a bit depending on the parameters provided. Results are not based on surface-level similarities, but draws from things like cinematic movements and eras, influences and similarities between filmmakers, style, themes, and other areas within film history. The hardest part about building this was building the datasets (and the math, there's a lot of freaking math).

What I'm mainly looking to test is:

• General functionality
• Whether the interface makes sense (though the current interface is mostly placeholder at the moment)
• Impact on server resources with multiple users running searches at the same time
• Whether the results mostly make sense

Resources are a big concern. Initial runs during the early stages last year could take upwards of 2 minutes, but I've managed to get that down to anywhere from about 1.5 to 20 seconds depending on the complexity of the request.

If you do volunteer I'd ask that you test it periodically, and there will be a dedicated thread where you can leave comments, suggestions, or notes. Every result will also be stored in a database for me to review afterward. There will additionally be a scoring system around whether a recommendation "makes sense," though I'm still deciding whether that should be a scale-based system or just a simple thumbs up/down setup. In either case, I'd ask that you score each item in the results when possible.

If you're interested, or have any questions, please respond in this thread.

I'm not sure yet how many people I'm looking for, so for now I'll just base it on the amount of interest.

[LastMinit]

Hey, so these days I'm heavily interested in a movie's backdrop, I'm always on the lookout for a film that strongly depicts a city or a setting or an era, and the more exotic and off-the-beaten path the better. I'm sure there are plenty of good examples in the film world I've yet to discover, but it's a hard thing to search for, and asking for recommendations is tricky because at the same time I'm very particular about a director's touch. So it's a bit of a niche within a niche. I recently caught Bi Gan's Kaili Blues on the channel and LOVED it, it's exactly the kind of thing I'm after. Not sure if this set of parameters would fall within your equations, but if it might, then I'd be glad to take part.

[cdnchris]

Actually, that is exactly what I'm trying to test for, at least in this initial phase. One of the parameters (and the only required parameter as of now, with everything else being optional) are directors you are interested in/fond of. So yes, you would enter Bi Gan and then adjust various knobs and get recommendations for Criterion releases (haven't expanded to their Premiers or Eclipse releases yet).

So Bi Gan with all the basic settings (meaning "safer") leads to:

Badlands
Chungking Express
A Man Escaped
In the Mood for Love
Mysterious Object at Noon
Japon
Yi Yi
Flowers of Shanghai
Mulholland Drive
Werckmeister Harmonies

Cranking things a bit more then gives

Color of Pomegranates
Eraserhead
Andrei Rublev
Japon
Flowers of Shanghai
Mysterious Object at Noon
Werckmeister Harmonies
Stalker
This Is Not a Burial, It's a Resurrection
La Cienega

So the idea is not necessarily "you like Kaili Blues or Resurrection so here are films exactly like that" but more "you like the work of these directors so here's some you may find of interest in your comfort zone."

For another example, Spielberg (with full "Safe" settings) would be:

Pee-wee's Big Adventure
WALL-E
I Wanna Hold Your Hand
The Big Chill
Thelma and Louise
The Others
The Three/Four Musketeers
Following
The Princess Bride
The Manchurian Candidate

More adventurous:

Seconds
Ashes and Diamonds
Robinson Crusoe on Mars
His Girl Friday
Foreign Correspondent
The Most Dangerous Game
It Happened One Night
Thief of Bagdad
My Darling Clementine
The Others

I've been much happier with the results it's throwing at me the last few months, but there's still some off items and I think it comes down to how it's weighting things in certain circumstances, so I need to tune it and having people throw whatever at it and ranking results will help.

[onedimension]

I don't have time to contribute to testing right now, but this sounds like a great project!

[godshammgodard]

Hi - I'm definitely interested in assisting with this. I'm a long time user of Criticker, which I use primarily for its recommendation algorithm. I also have my own personal recommendation spreadsheet. Anyway, this is right up my alley. Please let me know when and how I can assist.

[cdnchris]

I don't have time to contribute to testing right now, but this sounds like a great project!

Just to clarify, I appreciate if people are able to spend a bit of time on it, but even if it's something someone hits up once in a while, that's fine, too.

Hi - I'm definitely interested in assisting with this. I'm a long time user of Criticker, which I use primarily for its recommendation algorithm. I also have my own personal recommendation spreadsheet. Anyway, this is right up my alley. Please let me know when and how I can assist.

Great! It'll be a few weeks out but I'll send out more info once it's ready. I'm ironing out some issues, still, including one where I broke one of the most important parts of it and didn't realize it until this morning. So there's a chance it's going to be rough

[cdnchris]

I'll be opening this up in a couple of weeks and I'll send messages with a link to those that showed interest with a link.

If anyone else is interested, let me know. Again, I'm not looking for people to hammer the thing, I'm just looking for results generated by real users that I can parse through, so it's something that can be used casually, even just once. Plus(!) with the sale coming up, maybe it will list something that peaks your interest).

[nicolas]

Hey Chris, I’d be willing to test your feature.

[hearthesilence]

Actually, that is exactly what I'm trying to test for, at least in this initial phase. One of the parameters (and the only required parameter as of now, with everything else being optional) are directors you are interested in/fond of. So yes, you would enter Bi Gan and then adjust various knobs and get recommendations for Criterion releases (haven't expanded to their Premiers or Eclipse releases yet).

So Bi Gan with all the basic settings (meaning "safer") leads to:

Badlands
Chungking Express
A Man Escaped
In the Mood for Love
Mysterious Object at Noon
Japon
Yi Yi
Flowers of Shanghai
Mulholland Drive
Werckmeister Harmonies

Cranking things a bit more then gives

Color of Pomegranates
Eraserhead
Andrei Rublev
Japon
Flowers of Shanghai
Mysterious Object at Noon
Werckmeister Harmonies
Stalker
This Is Not a Burial, It's a Resurrection
La Cienega

So the idea is not necessarily "you like Kaili Blues or Resurrection so here are films exactly like that" but more "you like the work of these directors so here's some you may find of interest in your comfort zone."

For another example, Spielberg (with full "Safe" settings) would be:

Pee-wee's Big Adventure
WALL-E
I Wanna Hold Your Hand
The Big Chill
Thelma and Louise
The Others
The Three/Four Musketeers
Following
The Princess Bride
The Manchurian Candidate

More adventurous:

Seconds
Ashes and Diamonds
Robinson Crusoe on Mars
His Girl Friday
Foreign Correspondent
The Most Dangerous Game
It Happened One Night
Thief of Bagdad
My Darling Clementine
The Others

I've been much happier with the results it's throwing at me the last few months, but there's still some off items and I think it comes down to how it's weighting things in certain circumstances, so I need to tune it and having people throw whatever at it and ranking results will help.

I feel like this is a feature that should be incorporated into a dating app between cinephiles.

[cdnchris]

Hey Chris, I’d be willing to test your feature.

Thanks! Appreciate it. Again, it's at your leisure, I'm just looking for results to go through.

I'm thinking I'm going to get it ready for testing sooner. I was looking to fix some known issues and also improve the user interface, which is rough. But I might just fix those issues when going through results, and also generate a cleaner UI so that it's at least usable if only a placeholder. The backend stuff that does all the calculations is more-or-less done (now it needs to be fine tuned) but I'm always pretty slow when it comes to the user interface.

Also want to stress this isn't close to it's final form; I just want to test the film selection capabilities right now. After I get that to a better place there will be other features added like format, types and quantity of supplements, maybe audio preferences, and so on.

I feel like this is a feature that should be incorporated into a dating app between cinephiles.

A criterionforum dating app? Hmmmm

[hearthesilence]

If the idea takes off, I want a cut!

[colinr0380]

"Well they like Tarkovsky but also said that Adam Sandler was tolerable in Punch-Drunk Love..."

[hearthesilence]

Here's an idea - if the app labels a pairing as a bad match and you go ahead and try to contact the other person anyway, the app should play an audio clip of Bugs Bunny yelling "YOU'LL BE SORRY!"

[cdnchris]

I sent out messages with instructions to those who were interested. If you were interested and didn't receive a message from me let me know.

[LastMinit]

Reporting in. I've had success with this.
I find the interface very clear and straight-forward to use. The range of possible inputs is impressive, makes the collection almost seem like too narrow a band of answers, but even still it will take a lot more experimenting before the possibilities become exhausted for me, and that's a good thing. It's fun to mess around with.
I went in with a clear mission, looking for movies with a focus on backdrop. I input two sets of movies for two different answers, one for city settings, and one for pastoral. In both cases the main input I used was 'Familiarity with Setting' which I had pushed all the way to 'Alien Landscape', I also pushed the 'Age of Film' all the way to 'Freshly Minted' to narrow the scope a little. I turned most of the other inputs off, kept on 'Formal Experimentation' and 'Narrative Clarity' tweaked to the safer side of the mid-point, and I turned the genre and director bars down to 'somewhat important' or lower.
Going through the results, I cross-referenced the film descriptions, and stuck to what was available to watch the channel, and came away with one city setting to watch - Chop Shop by Ramin Bahrani, and one pastoral - Before the Rain by Milcho Manchevski. Had not heard of either before and they both delivered. Whatever other qualities they had, they both had exactly what it took to keep me watching - a real exposure of real, hidden away places. Even when the latter switched settings to London, it was shot with the backdrop very much in mind.
So, as far as I can see, this is working well so far.
I assume the info is all drawn from the existing blurb on each release? And even a film's more obscure qualities could affect answers if it is mentioned within that material?

[cdnchris]

Reporting in. I've had success with this.
I find the interface very clear and straight-forward to use. The range of possible inputs is impressive, makes the collection almost seem like too narrow a band of answers, but even still it will take a lot more experimenting before the possibilities become exhausted for me, and that's a good thing. It's fun to mess around with.
I went in with a clear mission, looking for movies with a focus on backdrop. I input two sets of movies for two different answers, one for city settings, and one for pastoral. In both cases the main input I used was 'Familiarity with Setting' which I had pushed all the way to 'Alien Landscape', I also pushed the 'Age of Film' all the way to 'Freshly Minted' to narrow the scope a little. I turned most of the other inputs off, kept on 'Formal Experimentation' and 'Narrative Clarity' tweaked to the safer side of the mid-point, and I turned the genre and director bars down to 'somewhat important' or lower.
Going through the results, I cross-referenced the film descriptions, and stuck to what was available to watch the channel, and came away with one city setting to watch - Chop Shop by Ramin Bahrani, and one pastoral - Before the Rain by Milcho Manchevski. Had not heard of either before and they both delivered. Whatever other qualities they had, they both had exactly what it took to keep me watching - a real exposure of real, hidden away places. Even when the latter switched settings to London, it was shot with the backdrop very much in mind.
So, as far as I can see, this is working well so far.

That makes me happy to hear, especially how you came at it and that it worked for you. I'm also happy to hear the interface makes sense; I wasn't sure on a lot of it. It still needs some modifications, but I'm happy to hear I'm on the right track. I'll be adding other filters on the more technical side of things specific to the discs (types of features, soundtracks, etc), and ways to group results. I also need to clean up how results are shown. I also need to fix the format specifications (4K, Blu-ray, DVD) as it doesn't work as expected. It can also hang for a bit at the 50% point.

I assume the info is all drawn from the existing blurb on each release? And even a film's more obscure qualities could affect answers if it is mentioned within that material?

Not really; the data is built off of a few things. I spent the last year building the datasets for it, and the amount of time it takes is why it's really only limited to Criterion titles right now (and they're perfect because there is a wide range of titles covering several genres, periods, and so on). But basically, each film has a fingerprint representing what it is based on a few parameters, and then each director receives fingerprints to represent their work (and this includes directors without a release in the collection). So when you input all your selections it builds a representation of the type of films you're looking for (your fingerprint, I guess), and then finds ones that match strongly to that. It then ranks them based on how well they match in all or some areas. I'm still playing with weighting, but I feel I'm much closer to where it needs to be than where I was a few months ago.

I've received encouraging comments around it so far, so I'm going to keep at it. I've been happier with its results lately, but I really needed other input around it to see if it was still worth pursuing, so I appreciate everyone who has tried it out.

[cdnchris]

I'm opening this up for the next few days for anyone who is interested in playing with it. This will help me a lot because it will give me more real world use cases and find where issues are. You can use this link first:

https://www.criterionforum.org/Recommen ... 7741257ec0

After that you should be able to just go to

https://www.criterionforum.org/Recommendation

It will shut down around Sunday but if the link stops working try the first one again and it should reset access.

Just a few notes about its current incarnation because I want to stress it's still not done.

• It's much better at picking films than it was months ago but there are still some quirks. That's what I'm trying to iron out, and it may come to data or weight adjustments.
• There are bugs in the UI. I'm aware of most of them, but it should be useable.
• For those that used it before you'll notice I made more progress on the UI and cleaned it up significantly, but I've disabled areas that are still being worked on; they're showing but you won't be able to click on anything.
• The results have been cleaned up but they're still not done and do show some random data that may not make sense.
• There is a debug tab in case you want to see how it came to those results but keep in mind this is not done yet; it overly simplifies things and may only list one film as a reference when multiples were used.
• I have it set up to queue requests. I don't expect a lot of use, but it can only do a few at a time. This is so I can also see how the server handles things. While waiting (if multiple users are trying at the same time), it will show a screen saying your request has been queued. As long as I didn't mess it up, you should be able to leave and return to see the status, but you can also just leave the page open and running: it's a background service that sends updates to your browser. When it starts running it will show a progress bar.
• There has been an issue where it will hang at the 50% point and then eventually pick up again after 30 or more seconds. I've rewritten this portion and it runs MUCH MUCH faster now, though can still hang for a few seconds in certain cases. I'm still working on this. But if does hang for a bit it should be fine; an error will be shown if something does break.
• If you're searching for a director and they are not listed you will be presented with the option to suggest one. Feel free to do so using the form, but right now it takes a bit of time to map a director into the system, so they probably won't be available before the weekend is over.
• "New Recommendation" does not clear the form, so it will display it again with all your entries. This allows you to modify what you entered in case you want to run it again. If you want to clear the form you need to refresh the page.
• When you click the "Generate Recommendations" it is supposed to take you to the top of the screen but it's not working at the moment, so depending on your screen resolution you may have to scroll up after submitting.

[cdnchris]

I posted a link to reddit as well and more people are using it than I anticipated, so if it crashes I apologize. It also runs in its own container so shouldn't impact the rest of the site, but if it bleeds over I'll shut it down.