collapse

Author Topic: Living with A Sick MPU  (Read 3846 times)

BaldwinK

  • Member
  • ****
  • B
  • Posts: 22
Living with A Sick MPU
« on: November 05, 2016, 12:56:58 PM »
This is a discussion on the failure modes of the MPU 6050, especially when used in an electrically noisy application.

Most of us start out sending streams of data to the console, tilting the MPU and seeing the pitch, roll or raw data change.  All seems in order.  However, if you scroll back through the screen dump, perhaps you will notice some odd values every now and then.

I have been trying to develop a compound servo using the MPU as a positional feedback sensor.  Obviously then there are brushed motors in close proximity and these do affect the smooth operation in a random and non-logical manner.  You could assume that this is just an i2c line problem and worry about pull-up resistor values and level converters.  Or you might see it as a problem within a complex microprocessor system being upset and unable to properly output data to the i2c bus.  Either way, and as a minimum, the bad data needs to be weeded out.

With the Quaternion Mimic experiment there were three small motors and one MPU mounted like a pancake. Loading 3k of DMP code and breaking the quaternion output into YPR values, the servo would always centre downwards. The DMP code provided the accelerometer and gyro fusion but is mainly concerned with correcting yaw drift.  I concluded that the mimic concept was ok, but yaw was not useful and the GY-521 breakout board needed to be mounted on its side to be practical.  There were also problems with noise.

In the end a couple of tests picked out most of the rubbish data: the DMP fifo count must be an exact multiple of the message length; and there must be no sign change for values that are not currently ‘near’ zero.  The fifo would then be flushed and DMP reset.  Occasionally the whole system needed a power cycle reset.

Moving on to the next stage of development using four larger motors, noise cannot be avoided.  I have tried enclosing the MPU in an earthed aluminium box, screening cables, adding slugging capacitors here and there etc.  Nothing really helps and so it must be concluded that the MPU6050 has a pretty poor noise immunity in the real world.

Fusion

Not using the DMP means taking raw values from the MPU then swapping the axes to suit mounting the GY-521 on the side of a robot’s ‘thigh’.  To see the rotations reported by the gyros and check their sign, I first mounted the board onto the arm of a 55g servo and swung it in an arc with pauses between reversals.  I followed Starlino’s approach to fusion software and it mostly works - when there isn’t too much noise.

The MPU raw values are available on demand with an i2c burst read.  There is no flag and no obvious reset either.  If the MPU outputs rubbish it can be discarded but the chip cannot be restarted with an i2c command.  Uncomfortable with no data ready flag, I set up the fifo and used the raw data when presented - but the errors were still there.

Writing endless diagnostic routines, the various bad data outputs were studied.  Some would self correct, some needed a reset to recover.  The worst event though is the perfectly working i2c messages of plausible and changing data that are quite unrelated to reality.  The MPU has gone insane – the LED is on but there’s no one at home.

No longer is this just of passing interest watching numbers roll across the screen, this is the plot of a disaster movie!  Here we have motors accelerating to close the gap between the setpoints and their present position and the MPU has gone offline.  The best outcome now is that the motor wires get pulled off before worse destruction occurs.

At this point then there are two basic questions that must be answered:

1. How do we detect the impending insanity of the MPU 6050?
2. How do we reset the MPU so that it recovers?

Probably best to take question 2 first since if it can’t be done my project is dead.

Resetting the MPU

In the last resort when it really is ‘locked up’, the chip must be power cycled.  So perhaps that may also be a way to cure some of the lesser failure modes.
A single output pin on the Nano can source the power needs of one breakout board.  With no extra capacitors fitted, a pulse of around 5mS is sufficient to restart the MPU.  However more time will pass before the chip is ready to communicate again.  Calling the i2c routines too soon will hold up all processing until the MPU is ready.

Once the MPU has had time to settle then it will be in sleep mode after power up.  The first messages will be to wake it up and set FSR and other parameters.  It will output zero value data at this time so no point in reading it.

The process is complex enough to warrant a state machine approach.  I use 10mS between data reads and all timing for fusion and PIDs follows from this.  State 9 would be when one bad output was given two retries before reset.  State 8 when one retry remains.  State 7 turns off the MPU power for the rest of the cycle.  State 6 is waiting for the MPU to power up and settle.  State 5 sends the i2c wake up commands.  States 4 to 1 read data as normal but prevent further calls to reset the MPU.  State 0 is back to normal.

In all then the restart process will take three time cycles before the MPU is expected to be back online.  In that time previous good data must be re-used and the gyro rotations aggregated for each time slot.

Error Pattern Testing

Displayed on the console most values will be floating point, possibly to three places.  Equivalence testing floats can give fuzzy results so testing is best carried out on the incoming i2c byte pairs. The burst read of 14 bytes gives x, y, z accel, temperature, x, y, z gyro.  Keeping these values in word form is handy but remember the original order despite the axes names changing because the chip is held vertical.

I had difficulty translating between the original word form and the floating point form.  This is due to the applied scale factors, and conversion from degrees per second to radians per 10 mS.  It was simpler to write a short routine to print to screen all 65,536 hex values with the equivalent acceleration and gyro values to 3 places.  It became a reverse dictionary.

The first error test is for empty values.  Common even without motors turned on this one must be found before attempting any vector calculations as it will cause a divide by zero.  We must test for both plus zero and minus zero: 0.000 ≡ 0x0000; –0.000 ≡ 0xFFFF.  Score two out of three of the accelerations and it will be an error but not a reset yet.

Next test is for +/- FSR and +/- FSR/2 suggesting the sensors are hitting the stops.  The default chip setting would give 2g and 1g, which might lead to faulting a valid output. So setting FSR to 4g: 4.000 ≡ 0x7FFF; –4.000 ≡ 0x8000; 2.000 ≡ 0x3FFF; –2.000 ≡ 0xC000.  Any one such accel value will cause an immediate State 7 reset.

Now for some interesting tests on bit stream patterns not obvious from the float values.  Here we test for a pattern of a single zero bit per byte across 4 consecutive incoming bytes.  Of course the test is actually done word by word but would find a pattern such as 0xBFBFBFBF which is too perfect for simple i2c line noise.  In extreme cases this will cross all 14 incoming bytes.  There are six tests for seven adjacent byte pairs (including temperature).  Similarly six tests for a single one bit per byte such as 0x40404040.

The single bit can be in eight places so 96 tests in all!

-2.008 ≡ 0xBFBF      2.008 ≡ 0x4040
-1.004 ≡ 0xDFDF      1.004 ≡ 0x2020
-0.502 ≡ 0xEFEF      0.502 ≡ 0x1010
-0.251 ≡ 0xF7F7      0.251 ≡ 0x0808         shown for FSR 4g
-0.126 ≡ 0xFBFB      0.126 ≡ 0x0404
-0.063 ≡ 0xFDFD      0.063 ≡ 0x0202
-0.031 ≡ 0xFEFE      0.031 ≡ 0x0101
  3.984 ≡ 0x7F7F    -0.016 ≡ 0x8080

It is possible for there to be two items genuinely equal at the lower values so the test can be expanded to three adjacent if considered necessary.  These faults cause an immediate State 7 reset.

Swing Testing

The human eye is good at spotting incongruities (especially on screen after the event) but programming a machine to make a real time judgement on good or bad data is not so easy.  Make the testing too critical and the MPU will be restarted and/or left using old values too frequently causing hiccups.  This in turn is liable to cause further errors.  Going too leniently on testing might miss the vital clue that from now on the messages may not fail but still be quite wrong. 

We know now that electrical noise is a problem but must also remember that mechanical ‘noise’ can be an issue.  Drone users, for example, decouple their sensors from mechanical shock.  Just tapping the servo assembly causes surprisingly large g-forces.  Cut these out with a simple single threshold test and the system may not report back accurately to the PID logic.

I spent a great deal of time trying different combinations of changes versus memory values – but to no avail.  Even normalised values can shift surprisingly in one hundredth of a second.  I ended up setting the FSR to 16g to keep down errors and concluded that second guessing the fusion code was a waste of time.  Even with tuning, the PID is never perfect, and the MPU may be reporting actual overswing rather than an output error.


Gyro Value Testing

Most failures and tests involve the accelerometers but occasionally a gyro will output large values or even stop reporting for a while.  Not catastrophic but the fusion code is biased in favour of the gyro so it has a greater effect on the output position.

Most gyro values for my servo system lie between 0 and +/- 0.080 radians so setting a threshold at +/- 0.150 seems to make for a simple test.  Gyro errors tend to self-correct so retries are permitted.

There are a few other minor tests that might depend on application but the MPU should be much more usable now.

Quadrants

There remain the usual issues of gimbal lock and sudden sign changes as the sensor moves into another quadrant.  This is not a failing of the MPU but it might appear so if not understood.  Although described by pitch and roll, the mechanics of my robot upper leg will limit movement at times.

Non commutating rotation is a nice mathematical description but not as helpful as a practical demonstration:

Stand with one arm straight down, palm flat against the thigh.   Don’t rotate the wrist or arm from now on.

Lift the arm up to the side.  The palm is horizontal.

Swing the arm to the front in a horizontal plane.  The palm is still horizontal.

Start again.  Lift the arm up to the front.  The palm is vertical.

Swing the arm to the side in a horizontal plane.  The palm is still vertical.

So it does matter in which order you move.  The classic rotational equations used for converting gravity vectors or quaternions to pitch and roll assume a particular order.

Suppose the setpoint of a compound servo system is changed.  Firstly it may not be mechanically possible at times to move one motor without first moving the other.  Secondly the MPU could end up at the same point but rotated 90 degrees – how will we know where it is?

Since the wrong answer could be catastrophic then the ‘babysitting software’ above should also include a control on movement commands.  Perhaps an intelligent servo should answer back and not perform a dangerous manoeuvre.

Post Script

I have been using the fusion and babysitting code for a month now and have growing confidence that the cables will not get wrapped around the test rig.  Currently I have two MPUs serving four motors and functioning.  The choice of motor/gearbox, details of construction and programming still have to be refined but the principle is looking sound.

The two MPUs each require Arduino pins to provide power, SDA and SCL.  Since the pull-up resistors are mounted on the breakout boards, they will still be in circuit when the board is powered off.  If the Vcc line is held low they will pull the i2c lines down; even if tri-state they will link across SDA and SCL.  The i2c bus cannot be used.

I use four rechargeable hybrids for the Nano and other logic; fully charged these provide almost 5.5V.  The Nano can then provide around 4V to the GY-521 voltage stabiliser.  This becomes critical if the batteries lose charge and then the MPU doesn’t reset fully.  There may well be better ways to switch the MPU Vcc power but for the moment KISS works adequately.

Processing two i2c, fusion and babysitting routines, four PIDs with PWMs and control logic occasionally takes longer than 10mS.  Extending the interval to 15mS leaves something in hand for further development.  The hit on pin numbers will be a problem though.

OddBot

  • Member
  • ****
  • Posts: 47
  • Have fun every day!
Re: Living with A Sick MPU
« Reply #1 on: November 05, 2016, 09:02:36 PM »
I am also using an MPU-6050 on a GY-521 PCB. After reading your article and doing my own experiments I can tell you your biggest problem is power supply.
A 16bit ADC working at 3.3V is sensitive to a fluctuation of 3.3V / 65536 = 50uV.
I suggest adding some extra capacitance to Vcc and Gnd of the GY-521 (at least 10uF).

As your environment is very noisy, you definitely need to shield your PCB. Wrap it with a layer of copper or aluminium foil.
Depending on the electrical noise level you may need several alternating layers of insulation and foil.
Connect all layers of foil to the battery Gnd.

I have also found that the Arduino Wire library automatically enables the internal pullup resistors.
This was pulling my SCL and SDA lines higher than 3.3V and causing the 3.3V supply to fluctuate.

I used digitalWrite() to turn these pullup resistors off after the Wire.begin() command to solve this problem.
This did reduce the fluctuations in my reading to some extent.




 

* Search


* Recent Topics

The unnamed (yet) quatruped spider project by tinhead
[July 01, 2020, 04:22:11 PM]


"1984 Nixie Time" by 1 what
[May 08, 2020, 01:04:18 AM]


2D Side Scroller Cyberpunk themed by Killer Angel
[February 06, 2020, 06:39:40 AM]


A new wing design for model aircraft / drones by OddBot
[February 06, 2020, 04:42:06 AM]


SDR (Software Defined Radio) by Gareth
[February 02, 2020, 06:15:42 AM]


Circuit Math by ZeroMax
[January 31, 2020, 01:50:18 PM]


NanOMeter by Protowrxs
[January 01, 2020, 12:59:44 PM]


Investigating the VL53L0X Laser Rangefinder by erco
[December 30, 2019, 10:45:44 PM]


PS4 Single Handed Controller Deployed (part 7 of 7) by Gareth
[December 30, 2019, 09:52:29 AM]


"D" -Pad Workio just like Magic (Will Merlin stay or Go) (part 6 of 7) by Gareth
[December 30, 2019, 09:51:27 AM]


PS4 Joystick Digitals 4,5,6,7,10 - Analog's Lx,Ly,Rx,Ry Workio (part 5 of 7) by Gareth
[December 30, 2019, 09:50:37 AM]


Menu Workio ! (part 4 of 7) by Gareth
[December 30, 2019, 09:49:49 AM]


L1 trigger design Workio (Hori controller) (part 3 of 7) by Gareth
[December 30, 2019, 09:48:50 AM]


Hori aka PS4 Joystick Mappings (part 2 of 7) by Gareth
[December 30, 2019, 09:47:17 AM]


PS4 Single Left-Handed Controller (part 1 of 7) by Gareth
[December 30, 2019, 09:44:58 AM]

* Recent Posts

Re: The unnamed (yet) quatruped spider project by tinhead
[July 01, 2020, 04:22:11 PM]


Re: The unnamed (yet) quatruped spider project by jinx
[July 01, 2020, 04:06:19 PM]


Re: "1984 Nixie Time" by 1 what
[May 08, 2020, 01:04:18 AM]


Re: "1984 Nixie Time" by tomasp
[April 13, 2020, 06:03:28 PM]


Re: 2D Side Scroller Cyberpunk themed by Killer Angel
[February 06, 2020, 06:39:40 AM]


A new wing design for model aircraft / drones by OddBot
[February 06, 2020, 04:42:06 AM]


Re: "1984 Nixie Time" by Gareth
[February 02, 2020, 06:23:01 AM]


Re: SDR (Software Defined Radio) by Gareth
[February 02, 2020, 06:15:42 AM]


Re: SDR (Software Defined Radio) by ZeroMax
[January 31, 2020, 01:54:21 PM]


Re: "1984 Nixie Time" by ZeroMax
[January 31, 2020, 01:52:29 PM]


Circuit Math by ZeroMax
[January 31, 2020, 01:50:18 PM]


Re: 2D Side Scroller Cyberpunk themed by ZeroMax
[January 31, 2020, 01:45:33 PM]


NanOMeter by Protowrxs
[January 01, 2020, 12:59:44 PM]


Re: Investigating the VL53L0X Laser Rangefinder by erco
[December 30, 2019, 10:45:44 PM]


PS4 Single Handed Controller Deployed (part 7 of 7) by Gareth
[December 30, 2019, 09:52:29 AM]