Recording and streaming emu frames

This is an archive of a topic from NESdev BBS, taken in mid-October 2019 before a server upgrade.
View original topic
Recording and streaming emu frames
by on (#137885)
Hi, i know it's not emu specific topic, but i just want to know the best method to "record" a nes movie comming from emulation playing.

I have used avi files and successfully generated a "nes movie", but as we all know avi files takes TOO MUCH space even that i did it without sound.

So looking for alternatives i found that this could be solved with DirectShow or ffmpeg.

And my question is how to realtime encode to mpeg2 or another format (streming??) so the video is smaller. But, but... how can this be done while the emu is running??, since i think we should have the complete avi file and then the encoder can convert it.

Im kinda lost about this topic and i need assistence.

Thanks in advance.
Re: Recording and streaming emu frames
by on (#137887)
AVI is just a container format; it can contain many different "codecs" that are ultimately what controls how big the file is.

How big is what you've currently got? (for how many pixels/how many seconds?) How much smaller do you want it to be?
Re: Recording and streaming emu frames
by on (#137891)
lidnariq i was making all without compression until i discovered that vfw let me compress videos.

do you know how do i have to set up AVICOMPRESSOPTIONS structure to compress to Microsoft Video 1??

documentation is poor, but there must be a way to do it...

edit: im using DIVX
Re: Recording and streaming emu frames
by on (#137896)
DivX is really bad for retro video game footage, because it's meant for high-color real world images. MS Video 1 is much nicer for the kind of images you're compressing (no blurring), but some pixels here and there will have the wrong color.

There are codecs made specifically for low-color game footage, such as the one that comes with DosBox (I forgot its name). It's lossless (the image is not degraded at all) and designed to take advantage of the kind of redundancy present in old game graphics.
Re: Recording and streaming emu frames
by on (#137897)
Ok im taking note....

Now i have a problem... syncing audio and video.

Im using MS VIDEO 1 for video and PCM for audio (i know its big, but since i have made the work i want to preserve it this way since there are not win32 api for compressed audio).

The thing is that according to AVISTREAMINFO structure MSDN doc it says that the frame rate is obtained dividing dwRate / dwScale.... im using dwScale = 17 and dwRate = 1020 wich gives me 1020/17 = 60hz, but the sound is de-synced... i also tried 16 and 960 pairs but it was worst.

Any idea wich path should i take??
Re: Recording and streaming emu frames
by on (#137902)
I use the Lagarith Codec when making NES videos. It's losless, so there's no quality issue. FCEUX can output directly with the codec of choice, and I'd recommend keeping the sound output, since it will already be synchronized for you.

As for your current synchronization problems, the NES framerate isn't 60hz, it is slightly different than this. A lot of emulators will generate video/sound at the NES hardware rate, so you should use the same video framerate settings as whatever generated the sound.
Re: Recording and streaming emu frames
by on (#137910)
It would be interesting to develop a codec for NES videos specifically, where it has a copy of the game's CHR ROM and just records diffs for final drawn positions of sprites, backdrop tiles, etc per frame, as well as palette writes. Though, I guess at that point you might as well just play back recorded inputs on an emulator, and this wouldn't work for CHR-RAM anyway. I'm sure some people will also jump at the exciting opportunity to talk about copyright issues with distributing the CHR ROM anyway...
Re: Recording and streaming emu frames
by on (#137913)
It would work just as well with CHR RAM, as writes to video memory $0000-$1FFF could be logged just as easily as writes to $2000-$2FFF and $3F00-$3FFF.

As for copyright, the renderer would compile an access log and include only those parts of the CHR ROM accessed by the PPU rendering circuitry in the video itself. If that's not enough, then the CHR ROM's contents could in theory be inferred from the pixels in the video itself. Only a few game publishers get uptight about "Let's Play" videos.
Re: Recording and streaming emu frames
by on (#137915)
Anes wrote:
lidnariq i was making all without compression until i discovered that vfw let me compress videos.

documentation is poor, but there must be a way to do it...



Understatement of the year. Stuff about VfW is horribly hard to find. Especially when you are using an other language than C(++)(#).
I am also currently implementing AVI recording in my emulator. It took me 4 a 6 hours just to get some information that was useful. Then I tried to "compress" the audio-stream also. But it seems that is something else. Even fellow developers think it is b0rked. (COMPRESSOPTIONS for audio that is)

btw.. I have tested Video 1 for recording.. Just let it be.. It is big in size and the results makes you cry blood.. That horrible. It is horribly pixelated. I have also tried the Lagarith Lossless Codec.. The file sizes are around the same with Video 1. But the image is just perfect.

But if you want MP4/h.263/.264 codecs? download the ffdshow directshow/vfw codec. You can configure the bitrate between 512kbs to 10000kbps. around 2000ish is acceptable.. busy games with lots of happening on the screen requires around 3500~4000kbps.
Re: Recording and streaming emu frames
by on (#137942)
Is there any how to developing in lagarith??
I have googled but nothing...
Re: Recording and streaming emu frames
by on (#137945)
There's notthing specific about coding for a particular codec like Lagarith.

If you are using an AVI writing interface that lets you select a codec, you can just choose the Lagarith codec if it's installed on your computer.
Re: Recording and streaming emu frames
by on (#137946)
In a few cases, I noticed that some emulators could output AVI through VfW encoders like Huffyuv (Lagarith's predecessor), but others would crash or return a "failure" when trying to output through those encoders. VirtualDub could encode through just about anything, however. And for some reason, VisualBoyAdvance running PocketNES was more codec-compatible than Nintendulator several years ago. This is why the video in the "Balloon Fever" StepMania simfile (2004) pans up and down: it's actually PocketNES's autoscrolling. So I wonder if some emulators are relying on undefined, unspecified, or implementation-defined behaviors of particular codecs or of VfW.

I've also noticed that FCEUX's AVI output fails in Wine, encoding only three seconds of video or whatever.

One strategy that could work is to output the audio to a wave file, output the video through a pipe to FFmpeg, and then encode the audio while remuxing it with the compressed video. That's what I did in a couple games I made in Pygame.
Re: Recording and streaming emu frames
by on (#137947)
vfw is horrible, but im using it... and im using lame too for sound. But i don't know how to "attach" mp3 sound to avi.
I have spent 2 days on this topic and im still investigating...
Re: Recording and streaming emu frames
by on (#137948)
To "attach" audio to video, you have to "multiplex" (or "mux") them together into a new file. You can mux the compressed audio and compressed video together with FFmpeg.
Re: Recording and streaming emu frames
by on (#137954)
The problem is that i want to codec "in the fly" and there is not good doc for ffmpeg
Re: Recording and streaming emu frames
by on (#137955)
Essentially you have to open a pipe into FFmpeg or avconv that accepts raw RGB and produces AVI in some compressed format. The command line for such a pipe might look like this example that Wrecking Ball Boy and my Pygame remake of FHBG use to encode on the fly:
Code:
avconv -f rawvideo -pix_fmt rgb24 -s "256x208" -r 30 -an -i - -c:v png tmp.avi

I'll break this down for you:
  • avconv: This is a fork of FFmpeg that happens to be the default in Debian.
  • -f rawvideo -pix_fmt rgb24: Input shall be a string of 3-byte RGB pixels, left to right, in rows to bottom.
  • -s "256x208": Each row is 256 pixels (768 bytes) long, and there are 208 rows (159744 bytes) in each picture.
  • -r 30: The video shall be played back at 30 pictures per second.
  • -i -: The input is standard input from a pipe, not a file.
  • -c:v png: The output shall use the FFmpeg "Motion PNG" codec, which is a sequence of PNG images in the same way that Motion JPEG is a sequence of JPEG images. If you have compiled FFmpeg with support for Huffyuv or Lagarith, you can use that instead of PNG.
  • tmp.avi: This is where the video file shall be written.
Re: Recording and streaming emu frames
by on (#137964)
Sorry the noob questions:

- do i have to call externaly "avconv"?
- How to add sound?
Re: Recording and streaming emu frames
by on (#137974)
I think what's been overlooked is Anes is asking how to do all of this within software (e.g. C, some library API, etc.) and not "how do I use an external program to do this". I thought that was apparent from his initial post ("... comming from emulation playing ... realtime encode to mpeg2 or another format ...").

I don't think spawning some external executable is really the way to go for this type of thing.

Nestopia has native code to do all of this already, so that may be a good place to start looking source-code-wise.
Re: Recording and streaming emu frames
by on (#137977)
koitsu wrote:
I think what's been overlooked is Anes is asking how to do all of this within software (e.g. C, some library API, etc.) and not "how do I use an external program to do this". I thought that was apparent from his initial post ("... comming from emulation playing ... realtime encode to mpeg2 or another format ...").


Thats right... i want to make in the software...
for me Windows Video 1 is ok for movies. What i cannot acchive is writing mp3 data to the avi using vfw.
The video part is ok i can record, but the sound part isn't.
Im using lame to encode in realtime sound but i don't know how to use vfw for writing the mp3 data.
I suppouse that using AVIFileCreateStream() in conjuntion with AVIMakeCompressedStream() could help me, but i don't know how to do it.
Re: Recording and streaming emu frames
by on (#137982)
I have already implemented the video recording of nes emulator. I make a project named ffencoder which code is based on ffmpeg for recording nes emulator audio and video into mp4 file encoding by aac and h264.

You can see my project ffnes and ffencoder on github for your reference:
https://github.com/rockcarry/ffnes
https://github.com/rockcarry/ffencoder

ffencoder simply wraps around ffmpeg encoding and muxing functions, and it's easy to use.
Re: Recording and streaming emu frames
by on (#138030)
I know it's an off topic but i need help.

well i installed ubuntu and successfully comiled the ffencoder.dll.

I made the following to make a .lib file from the .dll:

i created a .def file with the four function names and EXPORT in the top of the file.
i ran msvc lib tool to make the .lib and it made it well.
I included the path to the .lib in msvc and put the ffencoder.lib to link to.
No luck, the compiler throw me "unresolved external symbol xxx" whree "xxx" are the functions in the .dll.
Re: Recording and streaming emu frames
by on (#138044)
I think it's good to compile the ffencoder source files to various versions.
For example, Linux, Win32(Visual C++) and Win32(MinGW32).
Like this http://libsdl.org/download-1.2.php
Re: Recording and streaming emu frames
by on (#138046)
Reading msdn doc i could meke it work the ffencoder.dll

The thing is that ffencoder_init() throw me and exception. I don't know if anybody apart of rockcarry uses it, but im doing the following:

Code:
FFENCODER_PARAMS ffpar;
void * ffhandle;

    ffpar.filename = szMovieDir;   //this is the path to the .mp4 movie
    // video params
    ffpar.video_bitrate = 2000;
    ffpar.video_width = 256;
    ffpar.video_height = 240;
    ffpar.frame_rate = 60;
    ffpar.pixel_fmt = 32;
    ffpar.scale_flags = 0;    //i don't know what this member is for
    ffpar.start_vpts = 0;     //i don't know what this member is for
    // audio params
    ffpar.audio_bitrate = 128;
    ffpar.sample_rate = 44100;
    ffpar.channel_layout = 4;    //taken by looking in ffmpeg doc, it is "mono" i think
    ffpar.start_apts = 0;    //i don't know what this member is for
    ffpar.enable_log = 0;   //i don't know what this member is for

    ffhandle = ffencoder_init(&ffpar);



Any help??
Re: Recording and streaming emu frames
by on (#138047)
Boolean wrote:
I think it's good to compile the ffencoder source files to various versions.
For example, Linux, Win32(Visual C++) and Win32(MinGW32).
Like this http://libsdl.org/download-1.2.php


Yes, for someone, it's really diffcult to build ffmpeg for Win32/MSVC. You need ubuntu, mingw32-gcc, many many other related tools and environment.
So now I upload my recently ffencoder build binary for usage.
Re: Recording and streaming emu frames
by on (#138049)
Anes wrote:
Reading msdn doc i could meke it work the ffencoder.dll

The thing is that ffencoder_init() throw me and exception. I don't know if anybody apart of rockcarry uses it, but im doing the following:

Code:
FFENCODER_PARAMS ffpar;
void * ffhandle;

    ffpar.filename = szMovieDir;   //this is the path to the .mp4 movie
    // video params
    ffpar.video_bitrate = 2000;
    ffpar.video_width = 256;
    ffpar.video_height = 240;
    ffpar.frame_rate = 60;
    ffpar.pixel_fmt = 32;
    ffpar.scale_flags = 0;    //i don't know what this member is for
    ffpar.start_vpts = 0;     //i don't know what this member is for
    // audio params
    ffpar.audio_bitrate = 128;
    ffpar.sample_rate = 44100;
    ffpar.channel_layout = 4;    //taken by looking in ffmpeg doc, it is "mono" i think
    ffpar.start_apts = 0;    //i don't know what this member is for
    ffpar.enable_log = 0;   //i don't know what this member is for

    ffhandle = ffencoder_init(&ffpar);



Any help??


I will explain more how to use ffencoder (also you could check test2.c of ffencode project):

1. first thing is call ffencoder_init to get a ffencoder context
you can call it like this:

Code:
    void *encoder = NULL;
    encoder = ffencoder_init(NULL);


if you pass NULL to ffencoder_init function, it will using the default encoding params:

Code:
    static FFENCODER_PARAMS DEF_FFENCODER_PARAMS =
    {
        "test.mp4",         // filename
        128000,             // audio_bitrate
        48000,              // sample_rate
        AV_CH_LAYOUT_STEREO,// audio stereo
        0,                  // start_apts
        512000,             // video_bitrate
        256,                // video_width
        240,                // video_height
        30,                 // frame_rate
        AV_PIX_FMT_BGRA,    // pixel_fmt
        SWS_FAST_BILINEAR,  // scale_flags
        0,                  // start_vpts
    };


if you want use your own params, you can write code like this:

Code:
    FFENCODER_PARAMS params = {0};
    params.frame_rate = 60;
    nes->encoder = ffencoder_init(&params);


this means you set video frame rate to 60Hz which is for NES NTSC. The other members of params are set to 0, ffencoder_init will treat then as default values as DEF_FFENCODER_PARAMS showing above.


2. explaintion of DEF_FFENCODER_PARAMS:

Code:
typedef struct
{
    // output filename
    char *filename;

    // audio params
    int audio_bitrate;   // audio encoding bitrate
    int sample_rate;     // input audio simple rate, such as 44100 or 48000
    int channel_layout;  // audio channel layout, usually we use 16bit stereo, you should use constant defined in ffmpeg header files

    int start_apts;      // start_apts & start_vpts are used to make audio & video synchronous, usually set to 0
                         // if you find that after encoding to mp4 file audio or video is running fast than each other
                         // you could try to adjust these
    // video params
    int video_bitrate;   // video encoding bitrate
    int video_width;     // width of video picture size
    int video_height;    // height of video picture size
    int frame_rate;      // frame rate of video
    int pixel_fmt;       // input pixel format, such as 16bit RGB565, 24bit RGB888, 32bit ARGB8888, please check ffmpeg header files
    int scale_flags;     // i think no need to care this
    int start_vpts;      // see above

    // for debug
    int enable_log;      // don't care this, only for debug.
} FFENCODER_PARAMS;


3. when you meet a audio want to encode, you could write code like this:

Code:
    //++ ffencoder encode audio
    NES  *nes     = container_of(apu, NES, apu);
    void *data[8] = { apu->audiobuf->lpdata };
    ffencoder_audio(nes->encoder, data, apu->mixer_counter);
    //-- ffencoder encode audio


first param of ffencoder_audio is context value which returned by ffencoder_init
second param type should like void *data[8];. but only data[0] is used to store audio buffer address.
third param is the byte length of audio buffer.


4. when you meet a video want to encoder, you could write code like this:

Code:
    //++ ffencoder encode video
    NES  *nes         = container_of(ppu, NES, ppu);
    void *data    [8] = { ppu->draw_buffer - 240 * ppu->draw_stride };
    int   linesize[8] = { ppu->draw_stride * 4 };
    ffencoder_video(nes->encoder, data, linesize);
    //-- ffencoder encode video


first param of ffencoder_video is context value which returned by ffencoder_init
second param type should like void *data[8];. most case only data[0] is used.
third param is the byte length of scanline stride.


5. after all things done, you could simply call:

Code:
    // free ffencoder
    ffencoder_free(nes->encoder);



ffencode is now work fine in my ffnes emulator. but for your project, you need debugging.
To analyze specific issues in specific ways.

I hope these could help you.
Re: Recording and streaming emu frames
by on (#138050)
Anes wrote:
I know it's an off topic but i need help.

well i installed ubuntu and successfully comiled the ffencoder.dll.

I made the following to make a .lib file from the .dll:

i created a .def file with the four function names and EXPORT in the top of the file.
i ran msvc lib tool to make the .lib and it made it well.
I included the path to the .lib in msvc and put the ffencoder.lib to link to.
No luck, the compiler throw me "unresolved external symbol xxx" whree "xxx" are the functions in the .dll.


Yes, if you create a .def and only add four function names to .def, there will have some problems.
I meet the same issue when I try to create a .lib using the same method as you.
Then I use pexports tool to create a .lib from a .dll.

pexports ffencoder.dll > ffencoder.def
lib /MACHINE:X86 /DEF:ffencoder.def /OUT:ffencoder.lib

using this method, ffencoder.dll works fine in vs2005 msvc++.

I suggest you try pexports.
Re: Recording and streaming emu frames
by on (#138064)
First i had problems with the .lib file. So reading msdn i found that putting:

Code:
extern "C" {
#include <ffencoder.h>
}


solved my issue of "unresolved external xxxx symbol".

Now for the implementation, when i call:

Code:
    void *encoder = NULL;
    encoder = ffencoder_init(NULL);


At runtime it tells me "exception at xxxxx, cannot write to xxxx".

Any idea why it's failing?
Re: Recording and streaming emu frames
by on (#138076)
Anes wrote:
First i had problems with the .lib file. So reading msdn i found that putting:

Code:
extern "C" {
#include <ffencoder.h>
}


solved my issue of "unresolved external xxxx symbol".

Now for the implementation, when i call:

Code:
    void *encoder = NULL;
    encoder = ffencoder_init(NULL);


At runtime it tells me "exception at xxxxx, cannot write to xxxx".

Any idea why it's failing?


Are you sure you are linking against it properly? Trying to execute a function that hasn't been linked will make it upset.
Re: Recording and streaming emu frames
by on (#138078)
mikejmoffitt wrote:
Are you sure you are linking against it properly? Trying to execute a function that hasn't been linked will make it upset.


i don't know. How do you link it? How would you link it?
Re: Recording and streaming emu frames
by on (#138079)
I solved it... thanks
Re: Recording and streaming emu frames
by on (#138081)
what is a "scanline stride"??

What i have is all the pixels retriving a pointer to my D3d9 texture, wich acttually contain the RGB info (from botton to up) wich i actually invert the image by code.
Re: Recording and streaming emu frames
by on (#138082)
Anes wrote:
what is a "scanline stride"??

I assume the number of bytes between the start of one scanline and the start of the next.
Re: Recording and streaming emu frames
by on (#138109)
Anes wrote:
I solved it... thanks


what's the problem, and how do you fix it ?
Re: Recording and streaming emu frames
by on (#138113)
tepples wrote:
Anes wrote:
what is a "scanline stride"??

I assume the number of bytes between the start of one scanline and the start of the next.

Yes, you are right.
Re: Recording and streaming emu frames
by on (#138132)
rockcarry wrote:
Anes wrote:
I solved it... thanks


what's the problem, and how do you fix it ?


No, no it was a noob thing from my part. I realized that i had the .def empty.
Then what i did was to add to the .def file:

Code:
EXPORTS
ffencoder_audio
ffencoder_free
ffencoder_init
ffencoder_video


and made:
Code:
lib /def:ffencoder.def /OUT:ffencoder.lib


from the Visual Studio 2010 command prompt.

and then inside the code:
Code:
extern "C" {
#include <ffencoder.h>
}


having the path of "ffencoder.h" in my lib path.

And solved it :lol:
Re: Recording and streaming emu frames
by on (#138134)
So the encoder works by "scanline"??
Re: Recording and streaming emu frames
by on (#138138)
I'll be honest, i don't know how to encode the video.
If i have a ptr to all the ARGB pixels of frame how do i do?
thanks.
Re: Recording and streaming emu frames
by on (#138176)
Anes wrote:
I'll be honest, i don't know how to encode the video.
If i have a ptr to all the ARGB pixels of frame how do i do?
thanks.


Code:
BYTE  vbuf[256 * 240 * 4];
void *vdata   [8] = { vbuf };
int   linesize[8] = { 256 * 4 };
ffencoder_video(encoder, vdata, linesize);


vbuf stored the 32bit ARGB data, then vdata[0] = vbuf, linesize[0] = 256 * 4
Re: Recording and streaming emu frames
by on (#138179)
rockcarry wrote:
Anes wrote:
I'll be honest, i don't know how to encode the video.
If i have a ptr to all the ARGB pixels of frame how do i do?
thanks.


Code:
BYTE  vbuf[256 * 240 * 4];
void *vdata   [8] = { vbuf };
int   linesize[8] = { 256 * 4 };
ffencoder_video(encoder, vdata, linesize);


vbuf stored the 32bit ARGB data, then vdata[0] = vbuf, linesize[0] = 256 * 4


Why use arrays? (vdata[8], linesize[8])

why not just

void* vdata
int* linesize

or

int linesize =256 * 4;
ffencoder_video (encoder, &vbuf, &linesize);

why the arrays of 8 elements?
Re: Recording and streaming emu frames
by on (#138180)
ffmpeg using many many different pixel and audio formats, some format is using multi-paneled or multi-channel data.
Re: Recording and streaming emu frames
by on (#138185)
rockcarry wrote:
Anes wrote:
I'll be honest, i don't know how to encode the video.
If i have a ptr to all the ARGB pixels of frame how do i do?
thanks.


Code:
BYTE  vbuf[256 * 240 * 4];
void *vdata   [8] = { vbuf };
int   linesize[8] = { 256 * 4 };
ffencoder_video(encoder, vdata, linesize);


vbuf stored the 32bit ARGB data, then vdata[0] = vbuf, linesize[0] = 256 * 4


Damn!! my video only shows green colour with some garbage. Could it be a codec problem? i mean i have "k lite codec pack" installed.
Re: Recording and streaming emu frames
by on (#138186)
wtf!!! i tried a lot of configurations and still canno't acchive a good .mp4 file...
Re: Recording and streaming emu frames
by on (#138189)
What i fool i was.. it was "pixel format", anyway i the emu turns very slow as it encode the video. it's my emu performance thing.
Re: Recording and streaming emu frames
by on (#138193)
That's another reason for encoding in a separate process: it lets you use all cores of your multi-core CPU.
Re: Recording and streaming emu frames
by on (#138209)
Is it worth to try to encode in a different thread?
I made the following: i created a thread like this:

Code:
DWORD WINAPI ThreadProc(
  _In_  LPVOID lpParameter
)
{
   for (;;)
   {
      if (video_record)
      {
         data[0]    = pixels;
         video_record = 0;
         ffencoder_video(ffhandle, data, linesize);
         //Sleep(1000);
      }
   }
}


so at the end of the frame i set "video_record" to "1".

The emu run fast, but when i play the video runs too fast, i don't know why.
Re: Recording and streaming emu frames
by on (#138210)
Anes wrote:
What i fool i was.. it was "pixel format", anyway i the emu turns very slow as it encode the video. it's my emu performance thing.


No need to install any other video codec, only using ffencoder.dll is enough.
You can try my ffnes, I upload it. After run a rom in ffnes and close it, test.mp4 will be created in the same directory of *.nes file.
Re: Recording and streaming emu frames
by on (#138211)
Anes wrote:
Is it worth to try to encode in a different thread?
I made the following: i created a thread like this:

so at the end of the frame i set "video_record" to "1".

The emu run fast, but when i play the video runs too fast, i don't know why.


I suggest you first write some test code for ffencoder, for example, generate random data in audio & video buffer, or some special data such as color bar, sine wave, etc. And then try to use ffencode to encode audio & video into mp4 file. After this, you will exactly know how to use ffencode.

Then you can try to use ffencoder in you emulator, but I suggest you try it in single thread first. In my ffnes, I am using single thread, so I am not sure it can work when using in a different thread. In theory it can work in multi-thread case, because ffmpeg support multi-thread.

And for the process of encoding thread, if you using multi-thread, I sugguest you using event intead of video_record flag and Sleep().
Re: Recording and streaming emu frames
by on (#138243)
rockcarry wrote:
I suggest you first write some test code for ffencoder, for example, generate random data in audio & video buffer, or some special data such as color bar, sine wave, etc. And then try to use ffencode to encode audio & video into mp4 file. After this, you will exactly know how to use ffencode.

Then you can try to use ffencoder in you emulator, but I suggest you try it in single thread first. In my ffnes, I am using single thread, so I am not sure it can work when using in a different thread. In theory it can work in multi-thread case, because ffmpeg support multi-thread.

And for the process of encoding thread, if you using multi-thread, I sugguest you using event intead of video_record flag and Sleep().


Yeah, but c'mon its just a pointer full of pixels.

I tought that "pixel_fmt" in the structure was the INPUT pixel format and now i guess is the OUTPUT.

what pixel format are you using for outputing in your emu??
Re: Recording and streaming emu frames
by on (#138261)
Anes wrote:
rockcarry wrote:
I suggest you first write some test code for ffencoder, for example, generate random data in audio & video buffer, or some special data such as color bar, sine wave, etc. And then try to use ffencode to encode audio & video into mp4 file. After this, you will exactly know how to use ffencode.

Then you can try to use ffencoder in you emulator, but I suggest you try it in single thread first. In my ffnes, I am using single thread, so I am not sure it can work when using in a different thread. In theory it can work in multi-thread case, because ffmpeg support multi-thread.

And for the process of encoding thread, if you using multi-thread, I sugguest you using event intead of video_record flag and Sleep().


Yeah, but c'mon its just a pointer full of pixels.

I tought that "pixel_fmt" in the structure was the INPUT pixel format and now i guess is the OUTPUT.

what pixel format are you using for outputing in your emu??


For pixel_fmt, I am using AV_PIX_FMT_BGRA which is the default value of ffencoder params, which is also the pixel format of win32 32bit bitmap object. It means the pixel format of video data buffer passed to ffencoder_video(encoder, vdata, linesize) is AV_PIX_FMT_BGRA. In bytes order, RGB datas are stored like this:
high low
A R G B

ffencoder will converts input video data to YUV420 format which will be accepted by x264 video encoder. The color converting is done internally by libswscale module of ffmpeg. the output format of color converting is always YUV420, the input format should specified by pixel_fmt. You should specifiy the correct value of pixel_fmt, otherwise the encoding result file will be problem.
Re: Recording and streaming emu frames
by on (#138266)
Sorry, sorry what a fool i was!!! the "in realtime conversion" was failing cos i was reading from video card memory :oops: . That slowed down things.

ffencoder works like a charm!!
Re: Recording and streaming emu frames
by on (#138276)
rockcarry wrote:
Anes wrote:
Is it worth to try to encode in a different thread?
I made the following: i created a thread like this:

so at the end of the frame i set "video_record" to "1".

The emu run fast, but when i play the video runs too fast, i don't know why.


I suggest you first write some test code for ffencoder, for example, generate random data in audio & video buffer, or some special data such as color bar, sine wave, etc. And then try to use ffencode to encode audio & video into mp4 file. After this, you will exactly know how to use ffencode.

Then you can try to use ffencoder in you emulator, but I suggest you try it in single thread first. In my ffnes, I am using single thread, so I am not sure it can work when using in a different thread. In theory it can work in multi-thread case, because ffmpeg support multi-thread.

And for the process of encoding thread, if you using multi-thread, I sugguest you using event intead of video_record flag and Sleep().


I have a primary emulation thread which isn't connected to the GUI thread at all. All effects/graphics conversion/upscaling is done using a sleeping-dispatch thread. Technically if my emulator finished emulating one frame it dispatches the PPU frame to another thread/threads which takes care of upscaling (2xsai,xbrz, ntsc, 2x, 3x, tv modes, 1:1 -> 8:7 resizing, etc). when it finished the frame it will pause itself and send the data to the GUI thread..

The GUI thread draws the frame in the window and also encodes it to a videofile (using VfW). probably will add your ffencoder.dll to as it also compresses audio!. The reason why I have done this is as follows:

first:
The emulation thread doesn't need to be paused/halted, when the rendering/upscaling/conversion thread is compositing the image. (unless the 2nd frame is ready before the 1st frame has been processed then the emulation thread needs to be paused). After compositing the image the GUI thread takes over and draws it to the screen and if you are recording.. encodes it to a video stream.

And then. while we are encoding to a videostream.. the emulation thread fires another composition request.. meaning we have 3 threads taking care of:

* emulation
* display composition
* and encoding to a video stream.

at any given moment nothing is idle. because when the video encode is complete, the next frame is ready or almost ready from the compositor and the emulation thread is already computing the next frame. Ofcourse using 4xBRZ (1024x672) will bog down the emulator eventually on a old core2quad cpu. At that moment we are just asking too much.
Re: Recording and streaming emu frames
by on (#138280)
mmm... as it should be now im having problems encoding audio.
The audio is in raw format 16 bit mono pcm data

i have tried filling the structure like this:

Code:
    ZeroMemory(&ffpar, sizeof(ffpar));
    ffpar.filename = szMovieDir;
    // video params
    ffpar.video_bitrate = 512000;
    ffpar.video_width = 256;
    ffpar.video_height = 240;
    ffpar.frame_rate = 60;
    ffpar.pixel_fmt = AV_PIX_FMT_BGR0;
    // audio params
    ffpar.audio_bitrate = 128000;
    ffpar.sample_rate = 44100;
    ffpar.channel_layout = 3;   //this is for stereo


and the encode routine...

Code:
void *data    [8];
int   linesize[8] = {256 * 4};

void AddRecordFrame(unsigned char * pixels, char * audio_stream, int audio_len)
{
    data[0] = pixels;
    ffencoder_video(ffhandle, data, linesize);
    data[0] = audio_stream;
    ffencoder_audio(ffhandle, data, audio_len);
}


It sounds very bad and de-sync.
One thing to mention is that if i use "mono" (0x04) in "ffpar.channel_layout" when i ffencoder_audio() is called it throws me "interger divided by 0" exception.

any idea how to encode the audio?
Re: Recording and streaming emu frames
by on (#138286)
Anes wrote:
mmm... as it should be now im having problems encoding audio.
The audio is in raw format 16 bit mono pcm data

i have tried filling the structure like this:

Code:
    ZeroMemory(&ffpar, sizeof(ffpar));
    ffpar.filename = szMovieDir;
    // video params
    ffpar.video_bitrate = 512000;
    ffpar.video_width = 256;
    ffpar.video_height = 240;
    ffpar.frame_rate = 60;
    ffpar.pixel_fmt = AV_PIX_FMT_BGR0;
    // audio params
    ffpar.audio_bitrate = 128000;
    ffpar.sample_rate = 44100;
    ffpar.channel_layout = 3;   //this is for stereo


and the encode routine...

Code:
void *data    [8];
int   linesize[8] = {256 * 4};

void AddRecordFrame(unsigned char * pixels, char * audio_stream, int audio_len)
{
    data[0] = pixels;
    ffencoder_video(ffhandle, data, linesize);
    data[0] = audio_stream;
    ffencoder_audio(ffhandle, data, audio_len);
}


It sounds very bad and de-sync.
One thing to mention is that if i use "mono" (0x04) in "ffpar.channel_layout" when i ffencoder_audio() is called it throws me "interger divided by 0" exception.

any idea how to encode the audio?


Encoding @ 60fps will desync audio eventually as the NES doesn't run perfect 60fps. it is around ~60.098814 frames per second ( source: viewtopic.php?t=492#p3783 )
Re: Recording and streaming emu frames
by on (#138287)
nIghtorius wrote:
Encoding @ 60fps will desync audio eventually as the NES doesn't run perfect 60fps. it is around ~60.098814 frames per second ( source: viewtopic.php?t=492#p3783 )


yes, i know that, but there must be some way.
Re: Recording and streaming emu frames
by on (#138288)
Slow down everything by 0.16%. Frame rate would become exactly 60 Hz; CPU now runs at 1.78683MHz; PPU pixel clock becomes 5.36049MHz, &c.
Re: Recording and streaming emu frames
by on (#138290)
lidnariq wrote:
Slow down everything by 0.16%. Frame rate would become exactly 60 Hz; CPU now runs at 1.78683MHz; PPU pixel clock becomes 5.36049MHz, &c.


You could do: 60.098814/60 = 1,0016469

and then do 44100 * 1,0016469 = 44173 for sample rate. (still might have rounding errors)
Re: Recording and streaming emu frames
by on (#138300)
I seem to remember libavformat (the container layer of FFmpeg) letting the client specify the exact frame rate as the numerator and denominator of a fraction. For example, color NTSC TV runs at 30000/1001 = about 29.97 fps. To calculate the NES's exact frame rate, you need the PPU's dot clock rate and the exact number of dots per frame and the exact dot rate.

  • Dot clock rate is 945000000/176 Hz, or about 5.369 MHz. This is exactly 3/2 times the NTSC color burst frequency, which is defined as 315/88 MHz.
  • Number of dots is 341*262 in "long" frames and 341*261+340 in "short" frames, or an average of 89341.5 frames
  • Dot clock rate divided by number of dots is 945000000/(176*89341.5) = 945000000/15724104. Or reduced to lowest terms, this fraction is 39375000/655171.
Re: Recording and streaming emu frames
by on (#138317)
Don't over-complicate things.

For my ffnes emulator, in roughly video rendering frame rate is about 60Hz, if you are using 48000Hz sample rate for 16bit stereo audio, on each video frame, apu will genrate 48000 * 2 * 2 / 60 = 3200 bytes. It means every video frame, I will draw 1 video frame on screen and render 800 audio samples (3200 bytes audio data) to waveout device. At the same time, if you call ffencoder_audio & ffencoder_video, passing the A/V data buffer and buffer size, the audio and video will be synced very well.

How do you think the recording result of my ffnes, good or bad ?
Re: Recording and streaming emu frames
by on (#138321)
rockcarry wrote:
Don't over-complicate things.

For my ffnes emulator, in roughly video rendering frame rate is about 60Hz, if you are using 48000Hz sample rate for 16bit stereo audio, on each video frame, apu will genrate 48000 * 2 * 2 / 60 = 3200 bytes. It means every video frame, I will draw 1 video frame on screen and render 800 audio samples (3200 bytes audio data) to waveout device. At the same time, if you call ffencoder_audio & ffencoder_video, passing the A/V data buffer and buffer size, the audio and video will be synced very well.

How do you think the recording result of my ffnes, good or bad ?


Audio/Video gets horrible out of sync on your emulator recordings.. As excepted the audio starts to lag. Because your audio runs @ ~60.10Hz and video op 60Hz. This will eventually desync your video.
Re: Recording and streaming emu frames
by on (#138322)
I finally did it!!

I could mux a mp3 into an avi programatically with vfw and libmp3lame.dll. Audio is not desynced in the final avi. 735 16-bit samples per frame 44100 khz MONO.

Using Windows Video 1 at 50% quality and 128 CBR mp3 i got 5 minutes 315mb of video aprox.
I also testet lagarith but there were not too much difference, indeed Windows Video 1 looks better i think.

Wtf!!!
Re: Recording and streaming emu frames
by on (#138326)
sorry, I made a mistake for explanation of ffencoder_audio, the third param of ffencoder_audio is the sample count not the buffer size in bytes.
if you want to encode 800 audio sample, you should pass 800, not 800 * 4.

And for the recording of ffnes, I also found it was desync after taking a long time. I am trying to fix this bug.
On some low price computer, when using ffnes recording video, the cpu usage is high. It causes audio noise when play games.
Due to the complexity of the H264 encoding, it will spend lots of CPU resource. And in ffnes, I do audio/video rendering and encoding in the same single thread, making thread busy, when audio buffers could not rendered coutinous to wave out device, there will be noise.
two ways can improve this:
1. using another video encode type H263, MPEG2, ...
2. using multi-threads, create a new thread for encoding.
3. for desync issue, if set frame rate 60Hz in ffencoder params, should encode 800 sample(48000/60) audio on each frame.