Netflix recently opened up their Watch Instantly service to external platforms such as the XBOX 360, TiVo, Roku, and others increasing their potential audience of the service to users who don’t want to sit at their computer to watch a movie or TV show.

I’ve since watched over 40 videos through the service on my 360 and after the most recent one I received an email from Netflix asking me to rate picture quality was.  The options are The quality was very good, The quality was acceptable, and The quality was unacceptable.  The survey seems simple and innocent enough but has some interesting implications behind the answers.

The quality of your video stream is determined when you first start the video and adjusts if the player notices your connection slowing.  A change in quality is fairly seamless on a computer and happens while the video streams.  If it happens, you’ll see the sharpness of the video increase and decrease as the video continues to play.  The player on the 360 has to stop the video and re-buffer the stream when changing the quality.

What makes the survey interesting is that it allows Netflix to conduct an a/b test to find the right blend between quality and cost to stream a video.  Netflix can test different levels of stream quality, say on a scale of 1-5, and then survey their customers to find the ideal quality level right between where their customers are happy and when they start to complain.  If customers complain at level 2 but say level 3 is acceptable then Netflix can stream at level 3 and no worry about the added cost of levels 4 or 5.

Netflix could take this a step further and adjust stream quality per user based on their individual feedback.  If I consistently rate videos streamed at level 4 as acceptable and level 3 as unacceptable yet another user rates level 3 as acceptable and level 2 as unacceptable, they could customize the stream quality for each of us to minimize their cost.  Again, there’s no point in streaming at a higher quality if neither of us notice.

This is the ideal case when it comes to conducting a/b testing and gathering customer feedback.  This allows Netflix to better serve their customers and also reduce their costs.  In the end, I answered the survey that the The quality was acceptable.  I’m not sure what video quality I was receiving but it was perfectly watchable and I never once thought that it didn’t look as good as it should.  At the same time, it was an older movie and not one that would be commended on its special effects or impressive scenery so I didn’t think it deserved a The quality was very good.

You can read more about how Netflix’s streaming technology works in a surprisingly detailed post on their blog titled Encoding for streaming.