Give Our Abilities Time

As I mentioned in my prior post, I have been "broadcatching" HD content in x264/MKV format, and converting to DVR-MS to play on my extender, and so far I've been pretty happy with it. I do have to keep my eye on things more than I'd hoped but it's working fine.

However, I had to find a new way to convert downloaded MKV content to the iPod. (A 30GB "iPod Video" or "5G" device. It's a couple of years old but it has worked fine, and my birthday's coming up in five months and the iPod Touch will probably be reasonably priced or replaced with something even more technologically awesome, if my dear readers get the hint :D ) It so happened that I came into possession of a bunch of MKV files for a 10-year-old show I've been watching a lot of, with no XviD copies available online.

First attempt: SUPER

Thus I turned to SUPER as it is free and seems to convert anything to anything else one you figure out its many options. My first try used these settings, which I, for convenience's sake, exported as iPod_Video:

• Output container: mp4
  
• Output video codec: MPEG-4
  
• Output audio codec: AAC LC
  
• video size: 320:240
  
• aspect: 4:3 (as I edit this after the fact, should've been 16:9)
  
• frame/sec: 30
  
• bitrate: 1248 (in retrospect, could've been lower)
  
• options: Hi quality, top quality, stretch it
  
• (audio) sampling freq: 44100
  
• channels: 2
  
• bitrate: 96
  

(I know it would've been easier to show the picture, but I don't want to confuse my family by using my flickr account which is typically used for showing my entire extended family and friends pictures of my son.)

These settings resulted in a 54MB file that had no sound (in WMP, I didn't check iTunes/Quicktime). Several other attempts (that I cannot now remember, as it's been five days since I started working on this) were similarly slow and unsuccessful.

Second try: iPodifier/ffmpeg

I then downloaded iPodifier and set it up to watch the Recorded TV directory, figuring that MKV files were automatically converted to DVR-MS via the method documented in my previous blog post, and then iPodifier could pick them up for conversion to iPod Video-compatible MP4.

Seemed simple but, when I launched it, this showed up in the log window over and over:

3/8/2009 6:17:23 PM : Transcoding started: ffmpeg.exe -y -i "C:\Users\Public\Recorded TV\myfile.dvr-ms" -f mp4 -s 320x240 -acodec libfaac -async 4800 -dts_delta_threshold 1 -threads auto -vcodec libxvid -qscale 7 "S:\Temp\myfile-Mar 08 09 0330pm{TV}.mp4"

3/8/2009 6:17:23 PM : Transcoding finished successfully in 0min

According to the wiki this is related to some sort of problem that mencoder is having with the DVR-MS file causing an infinite transcoding loop, the first of its known issues. I am having the second known issue but had already renamed the files, so I just tried with a renamed file. Same thing:

3/8/2009 6:50:47 PM : Transcoding started: ffmpeg.exe -y -i "C:\Users\Public\Recorded TV\myfile.dvr-ms" -f mp4 -s 320x240 -acodec libfaac -async 4800 -dts_delta_threshold 1 -threads auto -vcodec libxvid -qscale 7 "S:\Temp\myfile-Mar 08 09 0645pm{TV}.mp4"

3/8/2009 6:50:48 PM : Transcoding aborted

So I checked what ffmpeg had to say for itself:

C:\Program Files\iPodifier>ffmpeg.exe -y -i "C:\Users\Public\Recorded TV\myfile.dvr-ms" -f mp4 -s 320x240 -acodec libfaac -async 4800 -dts_delta_threshold 1 -threads auto -vcodec libxvid -qscale 7 "S:\Temp\myfile-Mar 08 09 0645pm{TV}.mp4"

FFmpeg version Sherpya-r11050, Copyright (c) 2000-2007 Fabrice Bellard, et al.

libavutil version: 49.5.0

libavcodec version: 51.48.0

libavformat version: 51.19.0

built on Nov 18 2007 09:00:58, gcc: 4.2.1 [Sherpya]

Input #0, asf, from 'C:\Users\Public\Recorded TV\myfile.dvr-ms':

Duration: 00:43:53.6, start: 0.240000, bitrate: 2265 kb/s

Stream #0.0: Audio: 0x0000, 48000 Hz, stereo, 192 kb/s

Stream #0.1: Video: mpeg2video, yuv420p, 696x480 [PAR 320:261 DAR 16:9], 980

0 kb/s, 23.98 tb(r)

Output #0, mp4, to 'S:\Temp\myfile-Mar 08 09 0645pm{TV}.mp4':

Stream #0.0: Video: libxvid, yuv420p, 320x240 [PAR 4:3 DAR 16:9], q=2-31, 20

0 kb/s, 23.98 tb(c)

Stream #0.1: Audio: libfaac, 48000 Hz, stereo, 64 kb/s

Stream mapping:

Stream #0.1 -> #0.0

Stream #0.0 -> #0.1

Unsupported codec (id=0) for input stream #0.0

Apparently this file had some sort of conversion problem between MKV and DVR-MS. A different file from the same batch had AC3 audio and converted with the command line:

ffmpeg.exe -y -i "C:\Users\Public\Recorded TV\myfile.dvr-ms" -f mp4 -s 320x240 -acodec libfaac -async 4800 -dts_delta_threshold 1 -vcodec libxvid -qscale 7 "S:\Temp\myfile-Mar 08 09 0645pm{TV}.mp4"

I had to remove iPodifier's suggested "threads auto" option as that doesn't seem to work on my relatively recent build of ffmpeg (SVN-r16596-Sherpya). iPodifier was also making a mess of the filename so this was about the point at which I gave up on iPodifier and just used command-line ffmpeg.

The size was still tiny (70MB) and you could tell in iTunes that the quality was much worse than my ~320MB episodes converted from XviD using my old method.

I tried again with qscale 4 (iPodifier's "Best" quality) and the result was a 107MB file in ~44 minutes. I didn't even watch it (because I overwrote it accidentally in the next run).

I tried again with qscale 3, having found this article and noting that it seemed to be a better tradeoff between quality and size. Its result was a 140MB file. On this one, I noticed that in both QuickTime and when synced to the iPod the audio goes out of sync pretty early on, and also that the rendered video size was actually 426x240, not 320x240 as I had asked it to convert to. I guess I don't know what it ought to do in this case, but the program I've been using gives me a letterboxed 320x240. iTunes accepted the import of the video and the iPod showed it properly, but maybe the other program (which typically yields a ~330MB file) is inserting the black bars into the material itself, hence its larger size? Seems silly given that iTunes (or maybe the iPod itself) can handle the conversion.

Pressing on, I tried again with qscale 2, resulting in a 214MB file, and tried again with qscale 1, which created a 721MB file. Both looked great, not noticeably different from each other, but it was with this run that I noticed that both files had out-of-sync audio (though qscale 2 was further off).

So I decided to stick with qscale 2, but needed to dig into the settings to fix the audio. Here was the command line I started with, courtesy of iPodifier (minus the "threads auto" option that didn't work in my relatively recent build of ffmpeg):

ffmpeg.exe -y -i "C:\Users\Public\Recorded TV\myfile.dvr-ms" -f mp4 -s 320x240 -acodec libfaac -async 4800 -dts_delta_threshold 1 -vcodec libxvid -qscale 2 "S:\Temp\myfile-qscale-2.mp4"

My guess was that the "async" parameter was the problem. I also imagined that dts_delta_threshold had something to do with it but I don't actually understand at all what the description ("Timestamp discontinuity delta threshold") means. Somebody fixed something at the bottom of this page by setting it to 0 so that may be worth a try as well. Yeah, I'm not that scientific with this shit. I tried async 1 and dts_delta_threshold 0, but the sound was immediately, horribly off.

Thus I tried async 1 with the dtb_delta_threshold option left off (per this admittedly several-builds-old post on andy vt's forum). I threw in -threads 2 and it went blazingly fast, but the sound was still worse than with the original async 4800.

This post suggested using async 2 so I tried that (with dtb_delta_threshold turned off). No luck.

I tried once more with async 2 and dtb_delta_threshold set to 1, but that didn't work for me, either. I decided that maybe MKV --> DVR-MS --> MP4 was not that efficient. But what converted MKV to MP4?

Third try: XviD4PSP

Good old ViDEOHelp.com. They offered like 8 ways to convert MKV to iPod MP4 and the simplest used XviD4PSP. Though I had no need for any of the formats described in the name of this free product, and was worried that most people are converting for use with newer iPods than mine, I figured it was worth an hour of my time to try given the many frustrating hours I'd spent on the other programs. On my first try, I used these settings:

• Format: MP4 iPod 5.0G
  
• Denoise/sharpen: disabled
  
• Brightness/contrast: disabled
  
• Video encoding: x264 HQ Ultra (yeah, it bothered me that only x264 and XviD options were present, but I gave it a try)
  
• Audio encoding: AAC-LC ABR 128k
  

  
  

This worked! The result was 320x184 (which I guess makes sense given the 843 x 480 source resolution) and it was only a ~115MB file, but the play was very smooth and the sound was pinpoint-accurate.

I looked at the output in the XviD4PSP window to see if I could reverse engineer the command lines and utilities used, but I'm not sure. Clearly it is using the x264 utility for video encoding, and it does appear to output its settings, but I can't tell what it uses for audio or for muxing video and audio back together.

The major downside of this utility is that it doesn't appear to have a queue -- it's just one video at a time. This is exacerbated by the fact that it is much slower than DVR-MS to MP4 via ffmpeg -- that took 7 or so minutes for the file I was using to test, whereas XviD4PSP takes about 30 minutes for that same file. So, I can't queue up a bunch before I go to sleep and have them waiting for me in the morning. Nor can I schedule them until I figure out the underlying utilities that provide the actual conversion.

Update: I received a comment pointing out that there IS a queue -- you load each file and then click Save, and then you can load another, and so on until you click Encode to convert the whole batch. Frankly I'm using EncodeHD now which supports drag and dropped files, and it does all its work through ffmpeg so it's pretty fast.

But, it works and the quality is good. After a week of trying other methods and failing, that's saying a lot.

--sbreck

…and have been happy with the switch, there are a few reasons I still use Media Center:

• occasionally there is a conflict between my shows and my wife's shows and she gets dibs because she records so little TV compared to me (and doesn't know how to download her own shows)
  
• the DCT 6412 that RCN gave me in my HD DVR package only has a 120GB hard drive, so I have seen a few shows get dropped off for lack of space and feel like I'm constantly having to watch shows more frequently than I'd like to keep them there
  
• I still don't have pay channels and Big Love is back on HBO
  

Thus, for the above reasons, I still have SABNZBd+ and NZB TV running to grab a few shows that I would otherwise have missed, or might like to watch on my iPod. Typically I grab XviD versions with MP3 sound, which as I have mentioned before work fine on my Linksys DMA 2100 Media Center Extender.

But, maybe I have watched enough HD content in the last few months, but I find myself wishing I could watch higher-resolution stuff on the extender. I've been here many times and tried many different conversions, usually resulting in multiple steps and many hours of conversion followed by disappointment (skips, halts, no sound, way-out-of-sync sound). But today I noticed Ian's link to an Engadget HD tutorial on automatically downloading and converting 720p quality content via DVRMSToolBox, so that you can just watch it right in the Recorded TV and enjoy skip, FF, resume, etc. at the native resolution of the file. Seemed relatively simple so I gave it a try.

First things first, I knew I didn't need uTorrent or tvRSS since I have the aforementioned SABNZBd+ and NZB TV running to automatically download the TV shows I want. Thus my first step was to download and install DVRMSToolBox.

After messing around with the extensions to be watched (need to add .mkv to both Watched Ext and Video Types, not just the former as Engadget's instructions state), I ran into another problem which is that I think the wrong action ("nativecommdetect" which I'm assuming is the commercial skip processing) is running and my "convert MKV to DVR-MS - mencoder" is never running.

To troubleshoot, I:

• deleted all actions besides my own new action in the Processing Condition Editor -- it still tried to detect commercials
  
• changed Delay Processing from Find Commercials to Ignore (this worked, and as I scanned the comments I found the same advice)
  

I started the sample file (Big Love 3x06, 1,528MB) at 6:48 PM by dragging it into the Recorded TV directory, and it was done 50 minutes later, with a DVR-MS file waiting in the same directory. In my quick check, it looked fantastic and the audio synced perfectly.

So, later, my wife and I cracked open a bottle of prosecco (someone brought it to a brunch we hosted recently) and sat down to watch Big Love. Less than 10 minutes in, it froze, and then a few minutes later I was offered the chance to reconnect to the Media Center PC. I was unable to do so. After some troubleshooting and cursing, I found that the PC was totally locked up; not even the mouse would move. Coincidence? I hard-powered it off and tried again. I hate to use my wife as a guinea pig but I was pretty curious if this was a usable conversion of the show. Fortunately, I made it all the way through the episode without incident so I downloaded a couple more MKV files with which to test. Having made it through two more, I can say that this is working pretty well for me. Finally, reliable HD content on the extender without having to do a ton of crazy conversion steps!

Thus, my thoughts turned to automating it further, such that the conversion takes place once the download is complete. Since all that has to happen to convert the file is to move it into the Recorded TV directory, I didn't think it would be too hard since SABNZBd+ allows you to launch scripts. My read of the documentation didn't seem to make it quite clear what actions launch the script (besides telling it manually in the web console). Regardless, I set a scripts directory within SABNZBd+ (http://yourserver:yourport/sabnzbd/config/directories/) and created a batch file called moveMKV.cmd in there. The batch file contains the following:

@echo off

echo moveMKV.cmd starting against %3...

if not exist %1\*.mkv goto :nomkv

echo ...moving %1\*.mkv to Recorded TV...

cd %1

move %1\*.mkv "C:\Users\Public\Recorded TV"

echo ...moved

goto done

:nomkv

echo nothing to move

:done

echo ...done.

exit

Marvel at my batch file skills. Um, anyway, if it's not evident, it looks in the path of the newly downloaded file (%1) and if there's an MKV file in there, it moves it to the Recorded TV directory. %3 is the name of the job (which should be "Big Love 3x06" in the example) -- I'm just using it so I know what's running. To test, I just created a fake directory and fake MKV file somewhere and ran moveMKV.cmd twice. It worked both times. I think I might have confused the hell out of DVRMSToolBox with the fake MKV file, though, as it just sat there logging "Waiting to process file:" over and over until I deleted the file.

Then, I turned the batch file on by going to the Switches page (/sabnzbd/config/switches/) and choosing moveMKV.cmd as the Default user script (2nd last option on the right). To test, I searched Newzbin for a small x264 file and downloaded it. The script worked on the first try and moved the MKV file into Recorded TV as I'd hoped, at which point DVRMSToolBox's FileWatcher picked it up. Sweet!

Since I figured this out and wrote up the steps, it's been a couple of days and I've watched a couple more files without any hiccups. The picture is great, the sound is great, and the files start up much faster than XviD files. And the extender seems a lot more reliable and less likely to crash playing DVR-MS Recorded TV than lower-quality XviD files. The only downside is that the XviD content takes like 7 minutes to download and then you can immediately watch it, so it's closer to instant gratification than downloading a big MKV that takes 25 minutes, and then converting it for an hour. But when it's fully automated to happen in the middle of the night, so that stuff is waiting for you, it's really great, and I find myself watching stuff on the extender that I already have on my DVR just to avoid fast-forwarding through commercials...

Enjoy,

--sbreck