Thread Tools
Apr 21, 2016, 09:41 AM
Registered User
MadmanK's Avatar
Thread OP
Discussion

Blackbox Log Viewer Enhancements


Everyone,

I have recently been developing a project which focuses on the fantastic Blackbox Log Viewer originally developed by theNickDude for baseflight/cleanflight/betaflight. After encouraging comments from members of this forum, I have decided to start this thread to make the project more readily available and to keep my ramblings separate from the main threads to prevent cluttering them up with unrelated comments.

For anyone who uses the blackbox log viewer, I think they will agree that it is an absolutely excellent tool for diagnosing and tuning their craft; but after watching JoshuaBardwells detailed and comprehensive analyses series and trying to use the tool myself to tune my own rather basic quad I found myself wanting more features; seeing Joshua starting other packages to analyze noise, or trying to compare rcCommand and Yaw gyro’s etc gave me the idea to add more functionality into the viewer; after all, the data is there, how can we make it more accessible?

Now I must point out, that anyone who examines the original log viewer code will very quickly come to the conclusion that the original author had/has plans to go further with the logger; the code has been developed to relatively easily allow enhancements and features and a lot of what I have done is just making those features available. I really do not want to take any credit away from theNickDude.

So what features did I want to add?
  1. A spectrum analyser that can display the frequencies on any graph field (but more useful for the gyro’s).
  2. To provide gyro and rcCommand graph fields on the same scale (in deg/s) so that they directly relate to each other (ideally, they should exactly overlap).
  3. To provide a calculated error (difference between gyro and rcCommand) that can so that it is more obvious when the tuning needs attention.
  4. To add events to the timeline that show when I have changed flight modes, this should make it easier compare flight performance when modes change (does the PID perform well in ACRO, but when airmode is selected, does it start to behave differently?).
  5. To be able to customize each graph field, smoothing, linear scaling, zoom level.
  6. To be able to control the graphs using keyboard and mousewheel (for scrolling etc).

To achieve the above, I needed to modify the blackbox log viewer javascript (obviously), but for some of the features, I needed more information that currently is not part of the log; I have added a new dialog box that allows the user to enter the extra information (relating to rates, expo’s etc) manually (see release notes, after changing these at the moment, you need to re-load your log file to allow it to re-calculate the graph fields). However, there is no reason why this information cannot be added to the log file directly by changing the betaflight source code; I have done this and now fly a custom v2.6.1 of betaflight; the source for this is also on github and I have added a pull request to the betaflight project for discussion. I must point out, you don't need to run a custom version of betaflight; you just won't get the new events;

I’ve added some screen grabs that should explain what I have been doing; feel free to download the log viewer and use it on your existing logs; it does not affect them in any way so you can still open them with the original log viewer if you don’t like the enhancements. And by all means, criticize or add suggestions; I’m always open to new idea’s and things can always be improved. At the moment, I have a very limited test platform, actually, only one quad that I can log on and thats a SPRACINGF3MINI, so logs directly to the SDCARD at 1/1. Please let me know if the functions are useful or just plain wrong....

Maybe, one day, if they are useful enough, they may be merged back into the original version but of course that is up to theNickDude and the community.

On the Graph Setup Dialog there are 3 new fields, Smooth, Expo and Zoom
  1. Smooth : By default, many of the fields in the standard logger are smoothed by the viewer before being plotted; this hides the actual raw data and all that bad noise; you can set smoothing to zero here if you really want to see how much noise is in the data.
  2. Expo : Makes the scale of the graph field non-linear; for example gyro is usually set to 25% which exaggerates the gyro values near the centre line so that you can see noise easier. A value of 100% is a linear scale.
  3. Zoom : is the vertical zoom of the scale; 100% is normal, 200% zooms in x2 (but is can overflow the chart bounds).

The setup log viewer dialog box must be set to your betaflight/cleanflight settings that were active when you made the log BEFORE you load the log file into the viewer (or just re-load the log file after you change these). (This will be sorted as soon as I can).

The github folder is https://github.com/GaryKeeble/blackb...iewer/releases

TO INSTALL
Just extract the zip file into a folder and then open the index.html file from within google chrome. It won't interfere with your existing extension; and you can simply delete the extracted folder if you don't want it.

USING THE GOOGLE CHROME STORE TO INSTALL THE EXTENSION DOES NOT WORK FOR THE CUSTOM VIEWER

Regards (and hoping I haven't upset anyone)

MadmanK
Last edited by MadmanK; Apr 22, 2016 at 01:38 AM.
Sign up now
to remove ads between posts
Apr 21, 2016, 09:49 AM
crashandrebuild
Very subbed! I just started looking into trying to do some similar stuff. I have also been looking into how cleanflight does the actual datalogging. Ideally it would do a CLI dump log to the SD card at arm so you'd have all your PIDs and settings right in the log and also log true calculated error (as opposed to just P which is being modified by your coefficient) If you had true error you might be able to do some kind of math to do a ratio of RC command with true error to get an overall flight error average to see if your tune changes are improving or hindering flight accuracy.

I'm going to continue to look into this but if there's any way I can be of help to this project please let me know, this is great!
Apr 21, 2016, 10:02 AM
Registered User
MadmanK's Avatar
Thread OP
Quote:
Originally Posted by hedlesschkn
Very subbed! I just started looking into trying to do some similar stuff. I have also been looking into how cleanflight does the actual datalogging. Ideally it would do a CLI dump log to the SD card ........
Thanks,

I too, considered trying to add a CLI dump to the blackbox log in the header and haven't shelved the idea by any means; my only concerns at the moment are that some people log to memory on their flight controllers which is usually very small, and this idea may use a lot of the memory; so I am considering two options
  1. To make it so that it only logs the CLI dump to the header when going to an external logger or SDCard.
  2. To make some kind of dump changes similar to the dump all feature in the cli commands, which only dumps differences from default values; this would minimise the data needed to be logged but may need more log viewer maintenance to keep it in sync with the defaults.

In truth option 2 would be really helpful for my backing up of settings anyway (regardless of whether they are logged or not); to know exactly what I've changed without having to use an external file comparison would be a brilliant.

Regards.
Apr 21, 2016, 10:43 AM
crashandrebuild
Quote:
Originally Posted by MadmanK
Thanks,

I too, considered trying to add a CLI dump to the blackbox log in the header and haven't shelved the idea by any means; my only concerns at the moment are that some people log to memory on their flight controllers which is usually very small, and this idea may use a lot of the memory; so I am considering two options
  1. To make it so that it only logs the CLI dump to the header when going to an external logger or SDCard.
  2. To make some kind of dump changes similar to the dump all feature in the cli commands, which only dumps differences from default values; this would minimise the data needed to be logged but may need more log viewer maintenance to keep it in sync with the defaults.

In truth option 2 would be really helpful for my backing up of settings anyway (regardless of whether they are logged or not); to know exactly what I've changed without having to use an external file comparison would be a brilliant.

Regards.
Yeah, a lot of the CLI thought was for my own use too. I try to take a CLI dump and name it the same as my log but inevitably I miss something or didn't take one mid change while tuning or wahtever and it becomes a pain in the ass to deal with later. I too was thinking maybe just log tuning stuff (PIDs, filters, modes, rates, etc) But I feel like it would also REALLY help for an external tune helper tool, feed it multiple changes and sometimes you can learn if that oscillation was really a D/P ratio issue (or a whole host of other things obviously)

That said, my full CLI dumps are only 7k-ish in size but you could do a "only to SD external" or "enable/disable dump" checkbox for sure.
Apr 21, 2016, 10:46 AM
Team AlienWarpSquad
I like the ideas and added features.

I really like the idea of having the FC Output the PIDs, rates, etc on the start of a log. Without this all of us have gotten confused as to which PIDs we were running.
Having other 'events' logged like Airmode, Superexpo, etc is also a good idea and handy during analysis.
Do hope the pull request gets into F
Apr 22, 2016, 04:46 AM
Test Pilot
teracis's Avatar
Great ideas here. I will have to give this a try asap. Then I'll be able to give feedback and ideas.

If it can't take a CLI dump on arm for logging, maybe you can set it up to take a CLI dump from a text file or similar so that the functionality can still be implemented, regardless of whether betaflight accepts the changes or not? Just a thought.
Apr 22, 2016, 12:45 PM
Registered User
Downloading now, great work!
Apr 22, 2016, 04:10 PM
Registered User
MadmanK's Avatar
Thread OP
Quote:
Originally Posted by teracis
Great ideas here. I will have to give this a try asap. Then I'll be able to give feedback and ideas.

If it can't take a CLI dump on arm for logging, maybe you can set it up to take a CLI dump from a text file or similar so that the functionality can still be implemented, regardless of whether betaflight accepts the changes or not? Just a thought.
Like this idea a lot, will put it on my wish list.
Apr 23, 2016, 10:44 AM
Registered User
MadmanK's Avatar
Thread OP

New Version released.


Just released some updates to add some minor features and clear some faults that I found.

Smoothing fields now get defaults as per standard log viewer and new "jump to time" feature for quick navigation.

All this has been finalised in v2.2.0
Last edited by MadmanK; Apr 24, 2016 at 05:29 AM. Reason: Superceded
Apr 23, 2016, 07:22 PM
Registered User
Works really well for me! Got a bit confused with the Expo setting... didn't realise that 100% = no expo. I set it to 0 (thinking that was no expo) - the chart goes crazy if you do that.

Absolutely fantastic to be able to dynamically set the scaling - with no smoothing and everything linear in scale you know exactly what's going on.

Thank you very, very much.
Apr 24, 2016, 05:21 AM
Registered User
MadmanK's Avatar
Thread OP

New Feature - Workspaces


I have added a couple of new keyboard shortcuts and features - would appreciate testing and feedback:-

Measure mode and Easy Sync.
  1. Measure Mode 'M' will put a marker on the graph and display the offset from the marker as you scroll left, right through the log. Useful for measuring time's between gyro spikes etc. Press 'M' again to clear the marker. The offset time is shown next to the time entry field on the toolbar.
  2. Easy Video Sync. Find the start of an event in your log (e.g. a Roll) and mark it (using the 'M' key). Now scroll through your video until your video shows the first frame where this event occurs. Now press [Alt][M] to sync the log at this point.

Other
I have changed the keyboard shortcut for jumping to the start/end of log, it's now the [Home] and [End] keys - makes more sense.
It was originally the [PgUp] and [PgDn] keys - what was I thinking?

Workspaces

Workspaces (or Favourites) are user configurable graph setups. Ever find yourself switching between a particular set of graphs; always plotting rcCommand, gyro and PID for Roll, then switching to a layout for Pitch? Now you can simply store your setup to a hotkey [0] through [9] for quick recall later.

So for example, setup a chart showing just the roll fields (and save changes to exit the graph setup dialog), press [Shift][1] to store it; then setup the same chart but for the pitch fields (and save changes to exit the graph setup dialog), store that using [Shift][2].

Now you can quickly switch between roll and pitch setups by just pressing [1] or [2]. So [Shift][0-9] to store, just [0-9] to recall.

Also, want to go back to your last graph setup, press [Ctrl][Z} to undo your graph setup changes. Press again to toggle back again.

New version is v2.2.0. https://github.com/GaryKeeble/blackb...ses/tag/v2.2.0

Regards.
Last edited by MadmanK; Apr 24, 2016 at 05:31 AM.
Apr 24, 2016, 06:06 AM
Test Pilot
teracis's Avatar
Quote:
Originally Posted by MadmanK
I have added a couple of new keyboard shortcuts and features - would appreciate testing and feedback:-

Measure mode and Easy Sync.
  1. Measure Mode 'M' will put a marker on the graph and display the offset from the marker as you scroll left, right through the log. Useful for measuring time's between gyro spikes etc. Press 'M' again to clear the marker. The offset time is shown next to the time entry field on the toolbar.
  2. Easy Video Sync. Find the start of an event in your log (e.g. a Roll) and mark it (using the 'M' key). Now scroll through your video until your video shows the first frame where this event occurs. Now press [Alt][M] to sync the log at this point.

Other
I have changed the keyboard shortcut for jumping to the start/end of log, it's now the [Home] and [End] keys - makes more sense.
It was originally the [PgUp] and [PgDn] keys - what was I thinking?

Workspaces

Workspaces (or Favourites) are user configurable graph setups. Ever find yourself switching between a particular set of graphs; always plotting rcCommand, gyro and PID for Roll, then switching to a layout for Pitch? Now you can simply store your setup to a hotkey [0] through [9] for quick recall later.

So for example, setup a chart showing just the roll fields (and save changes to exit the graph setup dialog), press [Shift][1] to store it; then setup the same chart but for the pitch fields (and save changes to exit the graph setup dialog), store that using [Shift][2].

Now you can quickly switch between roll and pitch setups by just pressing [1] or [2]. So [Shift][0-9] to store, just [0-9] to recall.

Also, want to go back to your last graph setup, press [Ctrl][Z} to undo your graph setup changes. Press again to toggle back again.

New version is v2.2.0. https://github.com/GaryKeeble/blackb...ses/tag/v2.2.0

Regards.
This is great except I can't get the workspaces thing to work. Is it likely to be something I need to set up on my end? Permissions or something?

I've just bookmarked the index file in chrome for easy access for now. I'll create a shortcut to that later once I know it's all working properly.

Cheers, thanks for your work on this!
Apr 24, 2016, 11:15 AM
Registered User
MadmanK's Avatar
Thread OP
Quote:
Originally Posted by teracis
This is great except I can't get the workspaces thing to work. Is it likely to be something I need to set up on my end? Permissions or something?

I've just bookmarked the index file in chrome for easy access for now. I'll create a shortcut to that later once I know it's all working properly.

Cheers, thanks for your work on this!
Think this was a fault on my part; it looked like I didn't update the github properly so the version that was posted did not include the workspaces feature, sorry.

There should not be any extra privileges required; it should just run normally.

I have released version 2.2.1 now; hopefully that will work... There are yet a few more features too...

I am still a bit of a github noob; so please bear with me while I figure out all the merging, releases etc.....
Apr 24, 2016, 02:59 PM
Test Pilot
teracis's Avatar
Quote:
Originally Posted by MadmanK
Think this was a fault on my part; it looked like I didn't update the github properly so the version that was posted did not include the workspaces feature, sorry.

There should not be any extra privileges required; it should just run normally.

I have released version 2.2.1 now; hopefully that will work... There are yet a few more features too...

I am still a bit of a github noob; so please bear with me while I figure out all the merging, releases etc.....
No problem at all mate, I'll check it out now!

EDIT: It works, time to go flying!
Last edited by teracis; Apr 24, 2016 at 03:08 PM.
Apr 24, 2016, 03:54 PM
Registered User
MadmanK's Avatar
Thread OP
Quote:
Originally Posted by teracis
No problem at all mate, I'll check it out now!

EDIT: It works, time to go flying!
Awesome; glad it's OK; enjoy your flying, raining here all day :-(


Quick Reply
Message:

Thread Tools