HDTV to DVD using DGIndex/AviSynth losing frames

[gadgetguy]

Initially I thought I had a synch problem, but the more I try to fix this, the more I realize that's not the case. Let me start at the beginning...

I've switched to OTA capture using a MyHD MDP-130 with the included MyHD software. It captures the transport stream as a .tp file. This plays normally, in synch, no problems. But I don't have HD equipment so I want to cut commercials and convert to standard DVD. I read through several threads and tried several suggested tools for cutting the commercials, but none of them offered frame accurate cutting and all of them left the resulting audio and video out of synch.

I decided that since I would be re-encoding to DVD anyway, I can do frame accurate cutting, resizing, and framerate conversion using my tool of choice for most things video, AviSynth. After trying several DirectShow codecs with varying results, none of them completely successful, and after doing some more reading here, I decided to give DGIndex a try. DGIndex seemed to work, but the further I got into the video, the further the audio became out of synch. Comparing the length of the audio file to the video file showed the audio file was several minutes longer. I could have time compressed the audio, but it didn't make sense to me that there should be such a large difference and when I looked back at the original .tp file, the duration matched the audio file.

So I went back to DGIndex and demuxed the video as well as the audio. The resulting m2v consisted of 330520 frames (opened directly in Virtualdub) and the duration was correct. However, the d2v file opened through AviSynth, mpeg2Source() only shows 313464 frames, and the same m2v opened through DirectShowSource() shows a framecount somewhere between the two (I didn't write that one down). The only way I could get all of the frames and therefore maintain synch throughout was to open the m2v in VirtualDub and frameserve to the AviSynth script, but of course this causes colorspace transitions that I would like to avoid since the original is in the correct colorspace to begin with (YV12). I've scanned the files looking for any obvious missing chunks, but everything looks like it's there, so the dropouts appear to be occasional and the gradual nature of the synch loss seems to support that.

I've tried the beta AviSynth 2.5.8 rc3 and the last stable 2.5.7 with the same results. I also tried updating to the latest DGIndex, but the problem remains. I see there is a new AviSynth out today that I haven't tried yet, but I doubt that there will be any difference since there is nothing in the changes list that seems to address my problem.

Can anyone explain why I'm losing frames through DGIndex? Or offer an alternative method that won't lose frames?

[poisondeathray]

I don't know why you are losing frames, but maybe you could post this in the DGIndex thread over at Doom9 for Donald Graft to see; he will probably want a sample (unless there is some simple setting that is being overlooked)

Did you try DirectShowSource() for the .tp file?

An alternative method might be using avidemux which could open your m2v file natively and encode to MPEG2 as well. (You might even be able to open your .trp file directly). I think it uses FFMpeg as the MPEG2 engine, and I am unaware of any intermediate colorspace conversions with avidemux. You can do your edits and it even has simple filters.

Perhaps you can try tsmuxer to demux audio & video, then check framecount?

Could your problem be related to your OS Win2000?

[Guest34343]

It's probably pulldown related. What field operation did you use in DGIndex?

Please open the stream in the latest version of DGIndex and set Video/Field Operation to Honor Pulldown Flags. Then run a preview (F5 or F6) over a fairly lengthy section of the video. Then post the counts you get in the Coded #, Playback #, Field Rpts, and Frame Rpts fields of the info dialog.

Assuming your stream has irregular frame pulldown (not field pulldown), which is becoming more common for 720P streams, you'll need to use the latest DGIndex with Honor Pulldown Flags to get the correct video length. If that still does not yield the correct length, please provide a sample of the source stream that I can use to duplicate your issue.

[gadgetguy]

Thanks. I'm actually doing the conversion on my laptop which has WinXP SP3, 1.73 GHz Centrino Duo, and 2 GB ram. My primary Win2K machine, listed in my specs, chokes on playing anything HD. It captures ok, but can't play.
I did try DirectShowSource on the .tp file and it also seems to lose frames. I tried various codecs including ffdshow, but they all seem to have the same problem, so I think it has less to do with decoding since it seems to happen regardless of the ?Source method. I don't really think it's a DGIndex problem, per se, because it was able to demux and retain all of the frames, it's just when it's served to AviSynth that it loses frames, just like all of the other methods seem to.
TXMuxer doesn't recognize the .tp file ("Can't detect stream type"). I did try HDTVtoMPEG2 and without even processing it shows the duration of the video to be nearly 25 minutes shorter than it should be. The file contains about an hour and a half of video and when I scan through with HDTVtoMPEG2 I can see scenes from all parts of the video, so it's not like it's reaching a premature end-of-file. It does report the correct framerate (59.94), so the only explanation I can think of is that it's not seeing all of the frames.

[Guest34343]

I am the author of DGIndex and trying to help you, but you totally ignored my post. Are you not interested in my help?

I don't really think it's a DGIndex problem, per se, because it was able to demux and retain all of the frames, it's just when it's served to AviSynth that it loses frames, just like all of the other methods seem to.

When served through Avisynth, it's DGDecode that is working. It is controlled by the Field Operation setting in DGIndex. That's why I asked you about it.

[gadgetguy]

I remember I tried both Honor Pulldown and Ignore Pulldown, and it didn't seem to make any difference, but I didn't actually measure anything to see. I've got the test you suggested running right now and I am seeing Frame Rpts counting up (much slower than the Coded and Playback). It has currently processed about 17000Coded, 18000Playback, and 1100 Rpts

Edit: I didn't ignore you, I was responding to the other post while you posted and didn't see yours until I posted. (It takes me a long time to type.)

[gadgetguy]

Here's a screen cap after about 14 minutes.

[Guest34343]

So it's exactly as I guessed: 720P with irregular frame pulldown. This is why all the apps are having problems; they don't honor frame pulldown. There's one that does, however: DGIndex! Another one that does is fccHandler's MPEG2 plugin for VirtualDub, which explains why things worked for you that way and also supports my theory about the pulldown handling being the cause. So please try everything again with Honor Pulldown Flags and make some measurements this time, i.e, get the frame count.

It simply can't be true that there is no difference between Honor Pulldown Flags and Ignore Pulldown Flags when pulldown flags are actually present.

Something else to explore if this line doesn't yield results is the possibility of corruption in your transport stream. Do you get any decode errors?

[gadgetguy]

I'll have to return to this later, (gotta get to work now), but I agree that this looks like it. Like I said, I didn't actually measure anything when I tried both ways before, so I'm sure there is a difference. But that was after several hours of trying so many different things, my head was spinning and I just didn't see it. I appreciate your help, and will let you know how it goes.

[Guest34343]

Cool. And sorry for the bit about you ignoring me.

[gadgetguy]

(Taking a break)
I'm not getting any errors from the latest DGIndex, but an earlier version warned of something to do with the first few frames of the stream not having the reference I frame, or something like that (I didn't save the error). But in my mind, that didn't account for the vast difference I was seeing, and at worst should have caused a synch off-set that I should have been able to compensate for with DelayAudio().

I let the test run to conclusion and here are the results. Very close to the numbers that I was seeing and definitely explains the difference. I can't wait to try it on the real thing for editing.

[Guest34343]

That open GOP warning in the earlier version is harmless. The orphaned B frames are replaced with copies of the first decodable frame, so AV sync isn't affected.

[gadgetguy]

Yeah, I didn't think it had anything to do with my problem. Setting DGIndex to respect the pulldown flags resolved the problem. The editing goes much smoother now.
Thanks for all your help and for developing and providing DGIndex.

[Guest34343]

My pleasure. Glad to be of assistance.