TuneTracker problem

You mean RS-423?

The switcher has been in use for about 4 months. It is what is recommended by TuneTracker. (From the manual) I have 3 of the S9 adapters. I can try them all.

If changing the component sitting between two devices does not change the outcome, then of the components it-self has an issue.

In theory, Tune Tracker should know how many systems with this motherboard, USB-Serial converter, and switcher combination are around.

Things like comparison configuration file for Tune Tracker, and configuration settings for the switcher to ensure that those of your system are as they are expected by Tune Tracker should have been done a long time ago.

The apparent randomness of the fault suggests an intermittent fault somewhere - and such fault is the most difficult kind to identify and resolve.

One not so obvious possibility - how clean is your power feeding the entire radio system? Electrical noise on the power grid can wreck the best systems…

2 Likes

The SS 4x4 audio switcher manual says it’s a multidrop rs232.

In other words, perhaps a good power conditioner could clear this up, if trying the single port serial card fails to solve this problem.

Another issue could be power out of phase. Many home/commercial wiring setups can have the opposite side of the phase wired adjacent to each other. Grid power is typically transmitted to end users as 240v single phase (here in the US) and is divided into 120v by splitting the phase with the neutral in the main breaker box. This means roughly half of all circuits are 180° out of phase from the rest. This can often be the case even in outlets adjacent to each other, depending on how they are wired. This can cause undesirable consequences in hardware that is connected, especially if different connected devices are on opposite phases. If you’re plugging multiple devices into separate phases, it can cause similar issues to grid noise. Make sure everything is on the same phase. Professional audio, especially touring shows but stationary studios as well, have to consider this on a regular basis.

2 Likes

Reading the switcher manual http://broadcasttools.com/wp-content/uploads/2014/06/Manual_SS4x4.pdf

There are DIP switches that set it’s ID on the multidrop … note the COM port name in Haiku is different from the Switcher ID a completely separate thing independent of Haiku and TT software technically (other than TT should know which ID to send commands to). If the DIP switch is flakey it could cause somethink like if there were a weak solder joint etc… possibly if it had ever been hit as well.

Does it always change between the same ID’s? Such as 0 and 1? One bad solder joint is more likely than several and it would cause it to jump between two numbers sometimes. this would mean the TT software would try to send commands to switcher ID X and the switcher would be set as switcher ID Y due to the errant short or open circuit.

I should have the single port card by Monday. I guess if that doesn’t solve the problem, that might be the next place to go. It always goes from 0 to 1.

So what you are saying now leads to to think the swticher itself. So the thing to look at is page 8 of the manual, You’ll see there are 2 switches that set the ID
ID SW 1 and SW2
0 OFF OFF
1 ON OFF
2 OFF ON
3 ON ON

Then figure out what ID the TT software expects either 0 or 1? You may try turning on SW2 and seeing if it jumps between 2 and 3 that would probably confirm that it is the SW1 DIP switch… in which case you may be able to open up the case and fix it with a soldering iron if you are feeling confident or perhaps if it is still under warranty call up the manufacturer and get it fixed.

I will take a look tomorrow. What is “0 off off 1 on off 2 off on 3 on on”? Is this where they are suppose to be? I am assuming the software expects to see the same numbers, port 0 ss 4.4#0 or port 1 ss 4.4#1. It only seems to work when the numbers are the same.

OK, all switches are OFF, except 8 for the PIP triggers. I have the old 4.2 switcher. Would it work with that one?

Picture%202
The screenshot from yesterday. It has been perfect for 24 hours now.

Thanks Bruce, the screenshot comes up fine

So the “plugged into” combo box always goes “pc_serial0”, no change there, right?
And the bug you are observing, is the “switchers found” label below, which sometimes reports about switcher #0 and sometimes about a (fictive) “switcher #1” ?

2 Likes

That is correct! I am limited as to how many posts/replies I can make!

Sorry about that; I’ve increased your “trust level” which should remove those restrictions.

2 Likes

Those are the settings of the physical switches on the switcher… damage to them could cause the switcher to thing it’s ID was different. Perhaps vibration could be a factor also if the switches are damaged or faulty.

That could be caused by a problem in CC, or in the switcher (or less likely, corruption of the data stream in between).

From the log, I see you’re using CC 5.3.2. Looking at the code for that version, it cannot possibly generate “ss4.4 #1” on its own, it should be preceded by something, then a “,” comma, then “ss4.4… #1”. (in tech lingo, the #0 digit is not a switcher unit number, it’s just a for() loop integer, indexed to zero).

==> question re. label – can you take a screenshot of the same window/tab, when it displays “…#1” ? I want to see what’s before it.

As to suspicions about the unit number,
Let’s activate terminal tracing with the hidden “ttcc_trace_hal” flag. That will display serial traffic, what CC makes of the unit number and so on. I’ll look especially at the beginning and the end (right after the bug/“digt hop” occurs), but we’re still talking of hundreds of KB of text, way too much to post in this forum.

==> Get in touch with me in email (use my nickname, followed by “@netcourrier dot com”).

There are known ways that serial traffic can get troublesome, when e.g. an app is restarted and re-opens the serial port again; but i’m assuming nobody is restarting CC behind your back when the computer is unattended.

(edit) Another pattern to look for is if the bug occurs at the same time every day. That would hint at a particular sequence of ProgramLog commands triggering the bug. From the above discussion this does not quite seem to be the case though.

Reminder to self – when I look at tracing, should check which version of SS4.4 is used; our code was tested against “SS4.4 V:1.7”

There’s a faint chance that with this new PCI setup we can open the serial port with SerialConnect (after closing down CC) and using the probing commands…
*0U
*1U
*POLL
Something to try out later maybe, depending on our findings.


In case it matters later, I see you’ve updated the “care package” .zip; it now mentions the new device:

KERN: e[32mpc_serial:e[0m Skipping port 0 as it is used for kernel debug.
KERN: e[32mpc_serial:e[0m scanning PCI bus (alt)...
KERN: e[32mpc_serial:e[0m found PCI device 15 [7|0|2] 9710:9835 as MosChip 16550 Serial Port
KERN: e[32mpc_serial:e[0m irq line 20, pin 1
KERN: e[32mpc_serial:e[0m card supports up to 2 ports
KERN: loaded driver /boot/system/add-ons/kernel/drivers/dev/ports/pc_serial

It is presumed that you have received the single serial port card and that this has solved the issue?

I remember hearing about one story about diagnosing intermittent issues at Be Inc. I am uncertain how much is fact and how much is folklore, and who was the developer involved (Dominic Giampaolo, Cyril Meurillon, or Benoit Schillings) as it was related to file systems.

As it goes, some file system related code was being debugged using a floppy drive. The code would would run for hours and suddenly crash for apparently no reasons and without clues left. In an attempt to figure it out, the heroic developer stayed up all night by the machine running the code. And it ran, and ran, and run until at dawn the sun shone onto the floppy drive. It so happened that the sun rays caused the media detection sensor to signal that there was no floppy present in the system.

So, finding out the origin of an intermittent fault is a combination of luck and timing - being capable of observing the fault occurring.

It appears that, if solved, this issue will likely become a story which will be remembered for a long time.

3 Likes

No, I just got the single port card this morning. It has run flawlessly for over 48 hours now. Switching to the serial port card has helped. I have had less problems with it and fixing it has taken less time when I have had the problem. I did discover one of the causes in a similar fashion. It would reset or restart at midnight 2-3 times a week. This would cause it to stop seeing the switcher. I saw that happen once and solved that problem.

2 Likes

Glad to hear that the issue has been resolved.

It looks like Tune Tracker needs some refinements to their QA/QC of the packaged systems before they are sent out to customers.

The “midnight restart/reset”, even if now solved, would be an interesting story to hear about.

It had something to do with Steadycast. I saw it a few times when resetting the computer after it would “lose” the switcher. Command Center would come up, and then a few seconds later, it would restart. I would see a box appear, but never really paid attention to it. When I did, it said something like Steadycast didn’t “hear” or “see” something and restarted. This reset everything, including the hot keys at the bottom. I would notice before, that on the days when it was not working, the hot keys had reset.(I would leave them where I could test the switcher relay.) Apparently, this was happening at midnight when the day changed. Not every night, just every 2-3 days. Dane told me how to remove Steadycast, and the midnight mystery ended!