Ffmpeg's threads/cpu usage

Software
#1

I see a peculiar threads/cpu usage when using ffmpeg to convert a video file (on Haiku Beta4). This is the command line:

ffmpeg -i "Test.mp4" -f avi -vcodec mpeg4 -b:v 1000k -r 30 -acodec ac3 -b:a 128k -ar 44100 -ac 2 "Test.avi"

I have an 8 core i7, 2.3 GHz machine. Pulse shows about 40% usage across all 8 cores at the start, but soon drops to maybe 10% with short 50% spikes on individual cores. In general it seems to fluctuate quite a bit (not sure if it depends on the video contents changingā€¦)

Also curious is the thread count and their core usage (all 8 cores on at the left, 4 cores turned off on the right):

ffmpeg-8+4-cores
ffmpeg-8+4-cores496Ɨ682 42 KB

All those lazy ass threads in the middle donā€™t seem to work at allā€¦

A current Linux Mint shows around 80% cpu usage acoss all cores, relatively stable. No idea about thread count, because I have no idea how to get that info on Linux.

Comments?

3 Likes
#2

can you compare encoding time ? i get the suspicion is that the memory Management isnā€™t doing a good job at feeding the CPUs, but itā€™s just a random ballpark guess.

#3

thread number under Mint: in bash you can run
$ ps -o nlwp <PID>
where PID is the ffmpeg process ID (just run ā€˜psā€™ to see this)

#4

Thanks @topolinik, thatā€™s the output under Linux:

$ ps -o nlwp 3441
NLWP
  34

So, 34 threads? Thatā€™s more than I expectedā€¦ The system monitor reports ~70% CPU usage of all my 8 cores.

My ā€œbenchmarkā€ is a 246 MiB mp4 that gets encoded to a 163 MiB avi, with that same ffmpeg commandline posted farther up. The result:

        Haiku:       Linux:

real    1m40,400s    1m18.950s
user    6m16,824s    6m32.181s
sys     0m7,496s     0m5.454s
3 Likes
#5

here is a great example of why haiku is superior to linux distros in general, small difference in encoding time massive difference in cpu utilization. Iā€™m still willing to bet that haiku could easily encode faster with some memory and CPU pipeline feeding optimization.

a 70% increase in cpu usage for what a 20% in encoding speed, thatā€™s wasted electricity and wtf is the Linux version doing with all that cpu utilization ???

can you test both systems against a external clock ? like a high precision watch or something ?

i for one am quiet skeptical about he stability of Linuxs internal clock monitoring for execution time.

#6

Have you tried putting the source and destination files in a RAM drive or RAMFS? It could be that ffmpeg ends up waiting more on IO than on CPU.

1 Like
#7

I dontā€™t have precise timepieces to benchmark with, and frankly, am not really interested in those numbers. I was really just wondering if all those ffmpeg threads under Haiku are also used under Linux, and that seems to be the case.

Also, my initial assessment of a 50% cpu load sinking to 10% may have been a fluke or misreporting of Pulse. Iā€™ve now installed Slayer to keep watch of cpu usage and threads, and it appears to be quite constant at 45-50% (compared to Linux 70%).

I was wondering why have dozens of threads, many of them waiting/idling and the rest not working full speed. After all, I want that encoding done as quickly as possible - why wait 30 minutes at 50% cpu usage when it could be done in 15 minutes at 100% (minus cpu cooling phasesā€¦ :slight_smile: )

IO doesnā€™t appear to be the bottleneck:

Haiku RAM FS:
real    1m42,401s
user    6m20,702s
sys     0m7,777s

ETA: ffmpeg shows a ā€œspeedā€ factor when encoding. Haiku: 9.5x, Linux: 13x.

4 Likes
#8

the IO bottleneck might be the way haiku moves data from disk to ram to cache. Iā€™ll do some thinking on how to figure this out. if i come up with anything Iā€™ll post up and the total speed difference and cpu utilization donā€™t make sense, it should be relatively linear and it doesnā€™t appear to be. obviously the haiku version is not adequately supporting the execution pipeline. also the linux version appears to have a lot of useless execution bloat

2 Likes
#9

It would be interesting a small research in this area with different formats, file sizes, video duration, conversion in parallel with other activities, simultaneous conversion of several videos, etc.

#10

Iā€™m kind of wierded out how you are giving linux that much shit for something you didnā€™t even investigate. Who sais that the speed would be a linear improvement? Also: the usage isnā€™t directly comparable like this.

#11

i might be deeper into the guts of debian than you are aware of, the gui penalty in linux, is not small. It should be linear btw, if ffmpeg is running same code in both operating systems.

i also noticed that you ignored my being critical of the haiku execution pipeline being poorly managed.

thereā€™s plenty of places you can go feel offended for Linux, is Haiku the best place ??

#12

Again, something you have not proven.

What is the point of this attitude?

It would be much easier to look at differences 1:1 and examine that, as this ffmpeg case does instead of extrapolating from almost nothing and going ā€œoh linux used the cpu more effictively, itā€™s clearly bloatedā€

#13

This looks like a job for scheduling_recorder and DebugAnalyzer to determine where the bottlenecks are, if it doesnā€™t appear I/O is the problem. Probably worth opening a ticket, as this looks like something that could be ā€œlow-hanging fruitā€, performance-wise.

3 Likes
#14

Done: #18216 ffmpeg shows little usage of a large number of threads

4 Likes
#15

You know what just came to me? My cpu is an intel i7-4712MQ @ 2.30 ghz that supposedly can ā€˜boostā€™ to 2.9 ghz. Is this boosting done on-chip or does the OS have to trigger it?

If Linux does the boosting and Haiku does not, it may explain the speed difference.
I still donā€™t know why ffmpeg uses dozens of threads, half of them donā€™t perceptibly use the cpu, and still just barely make it to 50% overall usageā€¦ but maybe thatā€™s by design of ffmpegā€¦

#16

You can watch this happening in ActivityMonitor, I have added support for CPU frequency display a few months ago (based on low-level work from other developers to report this to userspace, sorry, I forgot who did that)

#17

Cool! Missed thatā€¦
All 8 cores hover between 900 and 1000 mhz when idle, while encoding all cores show up to 2800 mhz. So thatā€™s good!

1 Like