Robots run by the wiring of a fly's brain

It learned to flick the ball over the walls (fixed with a glass lid), and to hover in and out of the goal to collect reward (fixed by taking the resting reward back if the ball leaves).

Without seeing the ball it fails at once, so it really steers by what comes through its eye. Its weak spot is a noisy picture, which it never saw while learning.

It steers by what it sees, but it memorised this one maze rather than learning to read walls. Training on a new random maze every episode stalled, which is one reason the project moved to a real robot arm with more varied but simpler tasks.

A first version painted the floor green: the yellow ball was camouflaged against it, much as a flower can be in grass, so the room is now grey. A penalty for jerky commands punished the network's own exploration and stalled learning, so it was removed. The drone is paused to focus on the arm.

Corner misses end about 30 cm away, drifting towards the table centre: at the extreme tips the block is a small, dim dot for the head eye and outside the downward wrist eye, so the brain falls back to where blocks usually are. Decoys hardly matter (82–83 % with 0, 1 or 2).

On the way: a success rule that also demanded a nearly still hand could not be met under the network's own exploration jitter (even a perfect pilot with that jitter passed 27 %), so it became "stay within 3 cm for 0.2 s". The first run peaked at 90 % and then declined, so the best brain was kept and fine-tuned at a lower learning rate, with more practice blocks near the table edges.

It still works only by eye (0 % with the block invisible). Lighting is the weak spot: it never saw lighting changes while practising. One training run.

Round 1 the teacher drives every step; then the brain drives more and more (50 %, 25 % …) while the teacher keeps saying what it would do. The brain alone solved 0 % after rounds 1–2, up to 56 % after round 7, then fell to 4 % by round 10: once the brain drove almost all the time, the examples filled with its own near-misses, and it lost the final lift while keeping a clean reach and grasp. Practice with trial and error then took it from 4 % to about 90 %. During practice a second network that judges how well things are going (the critic) saw the true positions; it is used only for learning, never for acting.

For 2 million steps it found the block, went down to it and learned to close its fingers, but never completed a lift. At 2.15 million steps the first full lifts appeared, and by 3.2 million it solved about one attempt in five, often swinging the block sideways while raising it.

Every step towards the goal pays more, so the reward cannot be gamed by stopping halfway.

On 200 layouts it solved 84 %, against 92 % on the 50-episode test: the honest figure is somewhere in the high 80s. The most common miss (7 %): it holds the block 2–3 cm above the bin floor and never lowers it (in the case examined, the block was gripped high and hung tilted between the fingers). Then: never grasping the block (6 %), and ending with the block in the bin but still held (1–2 %). Lighting remains the weak spot, as for the lift.

Every block that landed in a bin landed in the right one: the colour decides the bin. The 14 misses (5 %) never reached a bin at all: dropped on the way (6), still holding it at the end (5), never grasped (3). Blue, the decoy colour it was taught to ignore in earlier tasks, is the weakest. Placed blocks end a median 1.3 cm from the bin centre; sorting takes a median 2.8 s.

Brightness changes hurt colour sorting the most (50 %), probably because brightness is one of the three channels the colours differ in (not yet checked per colour). A first design, with bins coloured like the blocks and at random places, failed in the teacher rounds (0 %): a big yellow bin looked like "the yellow thing" too, so the brain could no longer find the block.

Real fly photoreceptors adapt to the light level: each receptor type adjusts its sensitivity to the light it currently receives, so the brain gets contrast rather than absolute brightness. The eye model now does the same (with photon noise that grows in dim light). The brain then practised under normal light only, so every other lighting below is new to it.

Practising under many brightness levels (v13) only helped inside the practised range. Adapting eyes made overall brightness irrelevant (largely by design: adaptation cancels a global change), corrected a colour cast (like colour constancy), and handled moderately uneven light it never saw. At normal light it also sorted better than ever: 98 % of 300 new layouts, none in a wrong bin. Strongly uneven light remains the weak spot.

Without smell it still picks up and carries every fruit, but always to the ripe bin: "ripe" is its default, and the smell of rot is what sends a fruit to the other bin. Nothing else in the scene gives the answer away. Honest limits: the fruit recipes are simplified, some receptor responses were never measured (counted as none), the odour plume is smooth rather than turbulent, and the mushroom body (odour learning) is not included yet.

It finds the tomato by sight (moving the smell aside does not move the hand), but uses how strong the smell gets as the hand closes in to time the grasp: without smell it stops 4 cm short and never closes. Nobody taught it this; the teacher only ever moved by the true tomato position. Honest limits: the plant the eyes and physics see is simple (a stem, a branch, the stalk, three leaves; the leafy plant in the videos is drawn for viewers only), stems do not bend, and the hand brushes the stalk briefly in most picks. Strongly uneven light breaks it, as in the sorting task.

1. The eye was too coarse from a distance, so the teacher now first brings the hand close to the truss, the same move whatever is ripe, then goes for the ripe one.
2. A bug in the eye model: a thin stalk hid the whole tomato from the wrist eye at grasping distance. Fixed.
3. The teacher had a memory the brain lacks (it kept its earlier choice): in 12 % of situations its choice was not one the brain could make from what it senses. Now it decides from the present only.
4. One step at a time: one ripe tomato among three first, two ripe next.

Each fix was found by measuring rather than guessing: a linear read-out of "is this tomato ripe" from each sense showed where the information was missing. The camera then made it 100 %. Honest limits: the camera is an engineering part, not a fly eye; it has no white balance yet, so tinted light (12 %) and uneven light (0 %) break it; the brain still uses smell to time its grasp.

The zoom was found from the videos: the hand went to the orange tomato first, looked, then took the red one. At 34° a tomato covered about 1.4 pixels of the front camera; at the tomato's pixel, red and orange were told apart 93 % of the time, at 26° 99 %.

1. Memory made learning blow up: carrying all of each neuron's activity over scrambled the copied brain. Now each neuron learns how much to keep, starting at 5 %.
2. The second tomato was never taken: the arm would have had to remember where it was for ~3 s of carrying. Real harvesters look again; so does this one now, from a pose aimed by its camera.
3. The way back knocked tomatoes (up to 80 N): now it rises out of the basket first.
4. Practice slowly made it worse (90 % → 60 %): learning rate lowered.
5. The physics engine blew up after a knocked tomato flew into the table: a flung tomato now ends the try as a lost fruit.

Honest limits: bruises still fail 10 % of layouts, mostly a finger hitting the ripe tomato on the final approach (an impact within one 40 ms control step; stopping the push on contact did not help); a dropped tomato 4 %; tinted or uneven light breaks the cameras (no white balance); one training run.

Ending the try at the first bruise halved the bruising, but the arm learned to stop after its first tomato: following the hand showed it standing back for the rest of the time, never trying again, and more time did not help. The reason was the reward: every tomato already in the basket kept paying a little each moment, and a bruise would end all of it, so not trying was the safe choice. Paying each tomato once, when it lands, banks it; finishing the truss pays the rest.

Bruises still fail 14 % of layouts (mostly the fingers closing on the ripe tomato, or knocking a neighbour on the way in); in 10 % the arm still stops before the last ripe tomato; tinted or uneven light breaks the cameras (no white balance); one training run. Training on a GPU (MuJoCo MJX, Kaggle T4) was measured and was slower than the training PC for this contact-rich scene, so it stays on the CPU.

The goal afterwards was a near-perfect teacher and a fly brain at 95 %. The teacher got there; the brain did not.

The cameras are not the problem: a standard machine-learning model reads the teacher's “close now” moment from exactly the brain's inputs almost perfectly, and the distance to 0.3 mm. Watching the copying brain step by step found why it froze: closing or opening the hand happens on one step per tomato, and it learned the average, a hand that stays open. Weighting those moments removed the freeze (5 to 17 of 40 trusses), but never brought copying past the brain's own practice. On this task the fly brain plateaus at 67–72 %, and the arm series ends here.