2023-02-19 - RCA for Frosted Flakes Crash Feb 19th

Update the title following the format: Jan 01, 2023 - RCA Name

Incident overview

Postmortem owner

@Daniel Puratich , Everyone at flight test and @Nathan Green online

Airframe

FF

Related incidents

 

Incident date

Feb 19, 2023

Approx. Damage Costs

 

Report Date

Feb 19, 2023

 Executive summary

  • Crash Photos

  • Crash Logs

  • These need to be parsed

  • During this debrief we are reviewing the crash footage & Ardupilot telemetry logs

    • "batteries sag to <3v per cell during transition"

    • Had trouble maintaining fixed wing attitude

      • Looking at logs is that the elevator input makes almost no change in attitude

        • Could’ve been caused by airspeed being too low

    • Roll input is weak as well

      • Possibly all of our control surfaces are weak due to low airspeed ?

        • As we get faster servos require more power to work

        • Consider making controls out of something else, maybe a different

    • From logs it started falling as soon as the quad motors shutoff

      • Prime theory is we weren't going fast enough when quad motors shutoff

      • yaw is a bit weird but possibly not the issue

      • We should check pixhawk parameters and that would give us more information

  • There was only one servo on the elevator

    • Two servos total were ripped off the drone

    • Both rudders were ripped

    • Confirmed from photos, elevator one exists, both rudder ones were ripped off.

      • If the control input was mapped in reverse and the servo was reversed the acro mode on the ground is correct, but when the autopilot does something it is reversed.

        • quad mode it worked, but when ardupilot controlled it, it was the other way around

        • We can check this, but later

          • Anni has never had to reverse a servo in ardupilot

  • There was some roll and battery sag that could’ve caused stuff

  • This seems like the most likely theory from the logs

Graph shows control input in red resulting in the green turn down in the opposite direction (was presented and shown during RCA)
  • We can’t really confirm because the servo was ripped off, but hopefully we have pictures of the photos before it was damaged

  • Damage was 2 motors, 2 props, entire airframe, at least one wing

    • Other electronics are okay

    • One battery is GGed (puffed out), other is probably bad but needs to be replaced anyways

      • these were old batteries anyways

      • We have more motors coming in, but we are out for now

      • Both have been clearly sharpie labelled

  • Design airframe to protect critical components in the future

    • We need some way to protect the motor

    • This is more about protecting the motor than the prop, Mech to reassess

  • We will need to retest all electronics following this even the ones that aren't visually destroyed.

  • We weren't able to successfully transition back to quad during the crash

    • From logs it seems like it was trying to pull out of nose dive before transitioning to quad

    • It cant transition to quad when at such a steep angle - it may want to turn all on at once

      • Possibly the aircraft was fully inverted

      • Data cuts off at 40m altitiude.

        • We crashed above where we took off, but no way it was 40 meters

        • We need to take photos as we unplug and disassemble the cabin to see if the Pixhawk was unplugged

          • We should check things sooner rather than later to assess if we need to reorder stuff.

  • Someone needs to take this mess of notes and properly fill out this RCA lmao

 

Review of the elevator servo motor, as well as the flight controller configuration reveals that the autopilot sent elevator commands inverted to expected (pitch up requests were actually pitch down). This is likely the primary cause of the dramatic pitch down events which lead to the crash.

 

 Incident timeline

Describe (approximate) timelines, cross-correlate with telemetry/video recordings/etc if any.

 

Pilot inputs as perceived by pilot on sortie 3:

  • arm

  • took off in QLoiter

  • switched to FBWA

  • switched to QLoiter

  • switched to FBWA

  • spammed the button to switch to QLoiter

 

 Postmortem report

Instructions

Report

Instructions

Report

 Leadup

List the sequence of events that led to the incident.

  • Transmitter was configured with an inverted pitch input to correct for inverted pitch configured on Houston.

  • The aircraft was configured with the elevator inverted to desired input

  • Tests of control surface response were done for manual input, but not checked against autopilot input

  • During second flight, it was observed that the pitch of the drone in quad mode was inverted

  • After completing a successful transition from quad (QLOITER) to fixed (FBWA) modes, the aircraft quickly began pitching downwards uncontrollably. The aircraft was recovered by returning to QLOITER quickly

  • A second attempt of transitioning to fixed from quad resulted in the same pitch down event. Transition back to QLOITER was unable to recover the aircraft before impact with the ground.

 Fault

Describe what didn't work as expected. If available, include relevant data visualizations.

 

 Detection

Report when the team detected the incident and how they knew it was happening. Describe how the team could've improved time to detection.

 

 Response

Report who responded to the incident and describe what they did at what times. Include any delays or obstacles to responding.

 

 Recovery

Report how the user impact was mitigated and when the incident was deemed resolved. Describe how the team could've improved time to mitigation.

 

Five whys root cause identification

Run a 5-whys analysis to understand the true causes of the incident.

 

Related records

Check if any past incidents could've had the same root cause. Note what mitigation was attempted in those incidents and ask why this incident occurred again.

 

 Lessons learned

Describe what you learned, what went well, and how you can improve.

 

 Recommendations for future

Actionable Recommendation

Reasoning

Actionable Recommendation

Reasoning