Saturday, March 14, 2009
converting DVR-MS/MKV to iPod
As I mentioned in my prior post, I have been "broadcatching" HD content in x264/MKV format, and converting to DVR-MS to play on my extender, and so far I've been pretty happy with it. I do have to keep my eye on things more than I'd hoped but it's working fine.
However, I had to find a new way to convert downloaded MKV content to the iPod. (A 30GB "iPod Video" or "5G" device. It's a couple of years old but it has worked fine, and my birthday's coming up in five months and the iPod Touch will probably be reasonably priced or replaced with something even more technologically awesome, if my dear readers get the hint :D ) It so happened that I came into possession of a bunch of MKV files for a 10-year-old show I've been watching a lot of, with no XviD copies available online.
First attempt: SUPER
Thus I turned to SUPER as it is free and seems to convert anything to anything else one you figure out its many options. My first try used these settings, which I, for convenience's sake, exported as iPod_Video:
- Output container: mp4
- Output video codec: MPEG-4
- Output audio codec: AAC LC
- video size: 320:240
- aspect: 4:3 (as I edit this after the fact, should've been 16:9)
- frame/sec: 30
- bitrate: 1248 (in retrospect, could've been lower)
- options: Hi quality, top quality, stretch it
- (audio) sampling freq: 44100
- channels: 2
- bitrate: 96
(I know it would've been easier to show the picture, but I don't want to confuse my family by using my flickr account which is typically used for showing my entire extended family and friends pictures of my son.)
These settings resulted in a 54MB file that had no sound (in WMP, I didn't check iTunes/Quicktime). Several other attempts (that I cannot now remember, as it's been five days since I started working on this) were similarly slow and unsuccessful.
Second try: iPodifier/ffmpeg
I then downloaded iPodifier and set it up to watch the Recorded TV directory, figuring that MKV files were automatically converted to DVR-MS via the method documented in my previous blog post, and then iPodifier could pick them up for conversion to iPod Video-compatible MP4.
Seemed simple but, when I launched it, this showed up in the log window over and over:
3/8/2009 6:17:23 PM : Transcoding started: ffmpeg.exe -y -i "C:\Users\Public\Recorded TV\myfile.dvr-ms" -f mp4 -s 320x240 -acodec libfaac -async 4800 -dts_delta_threshold 1 -threads auto -vcodec libxvid -qscale 7 "S:\Temp\myfile-Mar 08 09 0330pm{TV}.mp4"
3/8/2009 6:17:23 PM : Transcoding finished successfully in 0min
According to the wiki this is related to some sort of problem that mencoder is having with the DVR-MS file causing an infinite transcoding loop, the first of its known issues. I am having the second known issue but had already renamed the files, so I just tried with a renamed file. Same thing:
3/8/2009 6:50:47 PM : Transcoding started: ffmpeg.exe -y -i "C:\Users\Public\Recorded TV\myfile.dvr-ms" -f mp4 -s 320x240 -acodec libfaac -async 4800 -dts_delta_threshold 1 -threads auto -vcodec libxvid -qscale 7 "S:\Temp\myfile-Mar 08 09 0645pm{TV}.mp4"
3/8/2009 6:50:48 PM : Transcoding aborted
So I checked what ffmpeg had to say for itself:
C:\Program Files\iPodifier>ffmpeg.exe -y -i "C:\Users\Public\Recorded TV\myfile.dvr-ms" -f mp4 -s 320x240 -acodec libfaac -async 4800 -dts_delta_threshold 1 -threads auto -vcodec libxvid -qscale 7 "S:\Temp\myfile-Mar 08 09 0645pm{TV}.mp4"
FFmpeg version Sherpya-r11050, Copyright (c) 2000-2007 Fabrice Bellard, et al.
libavutil version: 49.5.0
libavcodec version: 51.48.0
libavformat version: 51.19.0
built on Nov 18 2007 09:00:58, gcc: 4.2.1 [Sherpya]
Input #0, asf, from 'C:\Users\Public\Recorded TV\myfile.dvr-ms':
Duration: 00:43:53.6, start: 0.240000, bitrate: 2265 kb/s
Stream #0.0: Audio: 0x0000, 48000 Hz, stereo, 192 kb/s
Stream #0.1: Video: mpeg2video, yuv420p, 696x480 [PAR 320:261 DAR 16:9], 980
0 kb/s, 23.98 tb(r)
Output #0, mp4, to 'S:\Temp\myfile-Mar 08 09 0645pm{TV}.mp4':
Stream #0.0: Video: libxvid, yuv420p, 320x240 [PAR 4:3 DAR 16:9], q=2-31, 20
0 kb/s, 23.98 tb(c)
Stream #0.1: Audio: libfaac, 48000 Hz, stereo, 64 kb/s
Stream mapping:
Stream #0.1 -> #0.0
Stream #0.0 -> #0.1
Unsupported codec (id=0) for input stream #0.0
Apparently this file had some sort of conversion problem between MKV and DVR-MS. A different file from the same batch had AC3 audio and converted with the command line:
ffmpeg.exe -y -i "C:\Users\Public\Recorded TV\myfile.dvr-ms" -f mp4 -s 320x240 -acodec libfaac -async 4800 -dts_delta_threshold 1 -vcodec libxvid -qscale 7 "S:\Temp\myfile-Mar 08 09 0645pm{TV}.mp4"
I had to remove iPodifier's suggested "threads auto" option as that doesn't seem to work on my relatively recent build of ffmpeg (SVN-r16596-Sherpya). iPodifier was also making a mess of the filename so this was about the point at which I gave up on iPodifier and just used command-line ffmpeg.
The size was still tiny (70MB) and you could tell in iTunes that the quality was much worse than my ~320MB episodes converted from XviD using my old method.
I tried again with qscale 4 (iPodifier's "Best" quality) and the result was a 107MB file in ~44 minutes. I didn't even watch it (because I overwrote it accidentally in the next run).
I tried again with qscale 3, having found this article and noting that it seemed to be a better tradeoff between quality and size. Its result was a 140MB file. On this one, I noticed that in both QuickTime and when synced to the iPod the audio goes out of sync pretty early on, and also that the rendered video size was actually 426x240, not 320x240 as I had asked it to convert to. I guess I don't know what it ought to do in this case, but the program I've been using gives me a letterboxed 320x240. iTunes accepted the import of the video and the iPod showed it properly, but maybe the other program (which typically yields a ~330MB file) is inserting the black bars into the material itself, hence its larger size? Seems silly given that iTunes (or maybe the iPod itself) can handle the conversion.
Pressing on, I tried again with qscale 2, resulting in a 214MB file, and tried again with qscale 1, which created a 721MB file. Both looked great, not noticeably different from each other, but it was with this run that I noticed that both files had out-of-sync audio (though qscale 2 was further off).
So I decided to stick with qscale 2, but needed to dig into the settings to fix the audio. Here was the command line I started with, courtesy of iPodifier (minus the "threads auto" option that didn't work in my relatively recent build of ffmpeg):
ffmpeg.exe -y -i "C:\Users\Public\Recorded TV\myfile.dvr-ms" -f mp4 -s 320x240 -acodec libfaac -async 4800 -dts_delta_threshold 1 -vcodec libxvid -qscale 2 "S:\Temp\myfile-qscale-2.mp4"
My guess was that the "async" parameter was the problem. I also imagined that dts_delta_threshold had something to do with it but I don't actually understand at all what the description ("Timestamp discontinuity delta threshold") means. Somebody fixed something at the bottom of this page by setting it to 0 so that may be worth a try as well. Yeah, I'm not that scientific with this shit. I tried async 1 and dts_delta_threshold 0, but the sound was immediately, horribly off.
Thus I tried async 1 with the dtb_delta_threshold option left off (per this admittedly several-builds-old post on andy vt's forum). I threw in -threads 2 and it went blazingly fast, but the sound was still worse than with the original async 4800.
This post suggested using async 2 so I tried that (with dtb_delta_threshold turned off). No luck.
I tried once more with async 2 and dtb_delta_threshold set to 1, but that didn't work for me, either. I decided that maybe MKV --> DVR-MS --> MP4 was not that efficient. But what converted MKV to MP4?
Third try: XviD4PSP
Good old ViDEOHelp.com. They offered like 8 ways to convert MKV to iPod MP4 and the simplest used XviD4PSP. Though I had no need for any of the formats described in the name of this free product, and was worried that most people are converting for use with newer iPods than mine, I figured it was worth an hour of my time to try given the many frustrating hours I'd spent on the other programs. On my first try, I used these settings:
- Format: MP4 iPod 5.0G
- Denoise/sharpen: disabled
- Brightness/contrast: disabled
- Video encoding: x264 HQ Ultra (yeah, it bothered me that only x264 and XviD options were present, but I gave it a try)
- Audio encoding: AAC-LC ABR 128k
This worked! The result was 320x184 (which I guess makes sense given the 843 x 480 source resolution) and it was only a ~115MB file, but the play was very smooth and the sound was pinpoint-accurate.
I looked at the output in the XviD4PSP window to see if I could reverse engineer the command lines and utilities used, but I'm not sure. Clearly it is using the x264 utility for video encoding, and it does appear to output its settings, but I can't tell what it uses for audio or for muxing video and audio back together.
The major downside of this utility is that it doesn't appear to have a queue -- it's just one video at a time. This is exacerbated by the fact that it is much slower than DVR-MS to MP4 via ffmpeg -- that took 7 or so minutes for the file I was using to test, whereas XviD4PSP takes about 30 minutes for that same file. So, I can't queue up a bunch before I go to sleep and have them waiting for me in the morning. Nor can I schedule them until I figure out the underlying utilities that provide the actual conversion.
Update: I received a comment pointing out that there IS a queue -- you load each file and then click Save, and then you can load another, and so on until you click Encode to convert the whole batch. Frankly I'm using EncodeHD now which supports drag and dropped files, and it does all its work through ffmpeg so it's pretty fast.
But, it works and the quality is good. After a week of trying other methods and failing, that's saying a lot.
--sbreck
Labels: iPod, iTunes, Media Center
Monday, May 29, 2006
playing with iTunes playlists
All of a sudden iTunes lost track of all the files in a playlist I'd created for a car trip a few months ago. I believe I made the playlist in Winamp and then imported it into iTunes (after some frustration, leading to this blog entry/rant), and then synched it to my iPod. I know it's still on my iPod because I listened to it on the plane last week. It's true, however, that I hate and avoid launching iTunes, so maybe it's now gone because iTunes lost it.
Anyway, the point of this blog entry is to document for myself how I fixed it. And because I never blog (except this weekend's flurry of angry entries) and the point of blogging is to put stuff out there for yourself and hope that your audience likes that stuff and your voice and what have you. Or you don't care about your audience, which, given my lack of any sort of traffic tracking, may actually be the case. My audience is, I guess, my wife and one or two coworkers. Anyway, sorry to digress. Most of this fix came from Rob Brooks-Bilson:
1) Backed up My Music\iTunes\iTunes Music Library.xml and iTunes Library.itl to *.iTunesSux (I'm mature like that)
2) Opened iTunes Music Library.xml in TextPad, the world's greatest text editor which I encourage you all to buy
3) Searched for my playlist name
4) Found that iTunes stores songs in the playlist by TrackID, fortunately they were sequentially numbered from 1455-1528
5) Searched for 1455 to find the first track (PROTIP: you will find any given integer string often in the iTunes XML file, prefix your search with [key]Track ID[/key][integer]1455 to be accurate (and also replace the [] brackets with "less than & greater than" XML/HTML brackets))
6) found that it was in the /sounds/mp3 hierarchy as per my playlists (an idea that should allow my wife and my Media Center PC to use the same playlists despite storing MP3s on different drives and in different folders as long as they maintain the hierarchy), but needed to be in the file://localhost/D:/sounds/mp3/ hierarchy because iTunes is too STUPID to choose a default music library location and build its library off of that (or I'm too stupid to find that option :)
7) searched and replaced each of the 80 tracks in the playlist
8) opened up the ITL file, zeroed it out, and saved it (Rob says this causes iTunes to think its library is corrupt, and thus will rebuild it to reflect your changes to the XML file)
9) launched iTunes, and it ran through "Importing iTunes Music Library.XML" for a minute or so before telling my my ITL file was corrupt and that it had replaced it
10) iTunes then opened up and my playlist was fine.
And after all that, I noticed that, in iTunes' defense, the playlists I'd just imported (the Witch self-titled album, which is awesome -- doomy heavy metal with J Mascis on drums and a weird folk singer on vocals; a semi-new Bolt Thrower ("Those Once Loyal"), which is like every other Bolt Thrower album so if you like Bolt Thrower, and I like Bolt Thrower a lot, you'll like "Those Once Loyal"; and the new Cat Power "The Greatest", which is very different from the prior two albums but I'm trying to find new music that I can listen to on car trips when others are in the car with me) were listed with ../ to represent d:/sounds/mp3/, so maybe iTunes does have a default file location.
Anyway, just a quick blog entry for myself into which I snuck a music review. For those who are left hanging by the prior entry, I have OneNote 2007 working fine, and have an entry I'm working on with a review. In fact, I'm going to try and blog from OneNote now…
Labels: iTunes
Tuesday, March 21, 2006
iTunes frustration
Here is what we do with music in my apartment. I download MP3s from various sources or rip my own CDs and store them in a directory structure where it is vaguely categorized by genre and then artist - album name. I make sure that the music is tagged properly and includes album pictures by using the awesome Tag & Rename. For many years I have made playlists out of this music in Winamp, and this is what I've tended to use as a home stereo and synchronize to my iPod via ephPod. I never used iTunes because when I got my first iPod ("for Windows") in 2002 you had to use MusicMatch, which sucked, and I never tried iTunes when it came out. ephPod worked great because you could load the playlist, it would load all the songs if they were not already on the iPod, and then they would appear in the Playlists section of the iPod. So this was how I grew accustomed to using my iPod, and when my wife got one, I taught her to do the same as well.
I had always backed up my MP3s to a machine called NEMESIS (which, since we're talking about music, is named after a song on the High on Fire album "Surrounded by Thieves" -- in fact, all my machines are named after songs on this album except the Media Center PC, which is named HESSIAN after the song "Cometh Down The Hessian" on the "Blessed Black Wings" album). Before I had HESSIAN, NEMESIS was supposed to be the media/backup server for the apartment with its whopping 60GB RAID1 array. (I later added a single 120GB drive to it.) So I set up a lot of scripts to back up MP3s, My Documents, digital pictures, and my Outlook PST to this box. It happens to have an LTO2 tape drive in it now, but for whatever reason I cannot get it to consistently back up without throwing a million event log errors (I think mostly related to the ancient Compaq PCI SCSI card and/or the fact that it's a 400MHz PII that I bought in 1998). Anyway, when HESSIAN showed up on the scene, it became clear that I should also be storing MP3s and pictures on it to play or show in my living room, so thus it has come to pass that I have three copies of all my MP3s and digital pictures.
So for my wife, I had always figured that whatever solution she used to load MP3s onto her iPod would cleverly whichever of the stored MP3s on the network she wanted. This scheme had worked well when she was using my original iPod for Windows (5GB 1st gen), as all I had to do was make sure my M3U playlists were non-locational (e.g. they referred to relative directory paths rather than absolute like d:\mp3\ or whatever) and she could just load them with ephPod. However, when she got an iPod Mini, ephPod no longer recognized its directory structure, so we had no choice but to go with iTunes.
Having no desire to make it so that every computer in the house had its own copy of my 30GB MP3 collection, I assumed I could easily add songs from their network location to her iTunes library, and then she could sync them. However, this was not quite as easy as I would have hoped:
1) Her computer is connected to the wireless network. 802.11g, 54Mbit "Excellent" connection, but it's really only about 5Mbit in practice. iTunes chews on the songs for about 15 minutes every time you sync because of this. It just really sucks when the songs are remote. Not Apple's fault that my wireless network uses a repeater and also has to compete with 12 other visible networks in my building and the one across the back yard, but I would think that it does not need to exhaustively examine files that are already added to the library.
2) How do you add an album as a playlist in one easy step with iTunes? You cannot do this. The easiest way I've found is to create the playlist manually, name it, copy the directory locally, and then drag the directory from Explorer onto the playlist. This is what you'd think File-->Import Playlist, or dragging an M3U file to the playlist area, would do, but that's not the case.
3) I find the synchronization process to be somewhat random and annoying. I know I'm the only fool who knows that the "bing-bong" of the New Device sound and appearance of the system tray icon means that my device is recognized and I can now (manually) sync it, but it seems that it takes a long time for iTunes to recognize this. Further, I can see that, for less technically savvy individuals, nothing is simpler than connecting the iPod to the computer, which launches iTunes, synchs, and disconnects the iPod. Maybe I'm just used to my prior (manual) process and because ephPod only works when the iPod is connected, but usually I connect my iPod because I want to add music to it, and I don't think to launch iTunes (or whatever) beforehand to add the music. So inevitably I connect my wife's iPod to her computer in order to begin the process of adding some music, and then it starts synching really slowly over the wireless network, and I have to sync it again when it finally completes. If she wanted something on it quickly before running out to the gym, well, that isn't going to happen unless we remember to launch iTunes first.
4) In an optimistic attempt to have the Media Center PC be the center of our living room entertainment system (and also to not have to sync everything wirelessly), I had switched my wife to sync using iTunes on that PC. However, the DRM stuff on purchased iTunes quickly became a pain in the ass. (And I think that, as DRM goes, iTunes is fairly benign.) Here's the deal: my wife wanted to use her computer, with her mouse and her monitor, to buy music rather than the Media Center PC at couch distance with an HP remote keyboard/trackball combo. So the problem with that was that I couldn't come up with a good way to copy from her My Documents\My Music\iTunes\
All these add up to having it be a big pain for us to use iTunes, whereas everyone else in the world thinks iTunes is really easy to use. I agree. I am willing to admit that perhaps I have lost sight of how complicated my setup is, and/or how simple everyone else in the world's setup may be. At some point, I will wire my apartment for Ethernet, and the slow sync issues will presumably disappear. But this is just one of these things where everyone in the world thinks something is great, and I just can't imagine it. All I want is a fairly simple way to share MP3s with my wife while allowing her to buy music that she can sync to her iPod. And it's not like we're entirely small potatoes Apple customers -- I owned a 5GB "iPod for Windows", a 15GB G3, and now a 30GB Video, and my wife has a 4GB Mini and has her eye on a 4GB Nano. That's like $1700 in iPod purchases throughout the years, so that's no joke. I like the devices very much, and I also like my method of synching with ephPod. So if somehow you could purchase music through ephPod, maybe that would be the best option overall…
--sbreck
Technorati tag(s): iTunes, ephPod, iPod
Labels: iTunes
