MicroPython and the ESP8266

My most recent interests have involved IoT devices and separately getting better at writing Python. So here are a few thoughts and notes from my experiences combining these. Hopefully someone will find this helpful (I will probably find it helpful in a few months when I’ve forgotten everything I did).

I first heard about the ESP8266 chip a few years ago in one of the many technology related conversations I was having with my father-in-law. He was just raving about this tiny chip that could connect to wifi AND also run instructions all on the same chip. It sounded cool but at the time I really wasn’t too sure how to get from that little chip to something I could actually interact with.

When using MicroPython on the ESP8266 there are a couple of ways to get your code loaded and running. One of the easiest ways is through the web or serial REPLs. Through these you can very easily test your code, make quick modifications and really work out specifics about how to implement the program you want to write. This is great until you want to restart the ESP8266. When writing code in the REPL your wireless SSID and passwords get saved but upon reboot anything else you wrote is gone. So then the question becomes how to we persist our program across reboots.

One thing to know about booting the ESP8266 with MicroPython installed is every boot follows a set path to get going.
First it looks for _boot.py this file should normally hold the very basic initialization code for your board.
Next boot.py is loaded followed by main.py.
Generally you will want to either put the main part of your program in main or if you are building your program in multiple files you can simple include the file that kicks of your program and call what you need to get it started.

So now you are probably wondering, just how do I get my main.py file loaded?
There are a couple of answers to this but the first and probably easiest is to load your code through the WebREPL.
If you are using this you probably noticed to upload button and this is precisely what you want to use.

Yak Shaving

So of course, having these options available I decided to do something completely different to get my files onto my ESP8266 board.

After looking through a lot of the MicroPython documentation, I realized that another option to get files onto my board is to load them directly into a custom firmware and then flash that to the board. While it IS overkill for making a couple lights blink, I’d eventually like to do a few more things that just that.

Enter, frozen code!
No, the code isn’t actually cold but it is stuck in place (at least until the next time you flash your device).
If you pull down the whole micro python repo from GitHub you’ll find 2 folders inside the ESP8266 directory, scripts and modules.
These are the default locations we are going to put code that needs to be part of the custom build firmware we are going to be putting on our device.

So what is the difference?
scripts holds regular Python files.
They get added to the framework and run through the Python interpreter like normal on execution.
The modules directory gets a little extra special treatment.
Files in this directory get compiled down to byte specific for the device you are using which in this case is my ESP8266.
Pre-compiling this code helps to save a bit of RAM on load and possibly help things run a little quicker.

Nitty Gritty Details

To get this all working together I decided to use Docker to manage all my build tools for this. The first step is get the latest esp-open-sdk, then build and compile all the tools included with it. I first looked around to see what other people had been doing with Docker, MicroPython and the ESP8266. There are quite a few versions and implementations different people are using but for my personal use I had a few main goals: 1) Not have to rebuild the esp-open-sdk every time I need a new firmware version. 2) Store my Python files locally and be able to include them as needed into the firmware. 3) Install as few as possible dependencies on my local machine.

The closest (and newest) Dockerfile I found was at GitHub - ubergesundheit/micropython-build. At first I thought this was exactly what I was wanting but as I was working through getting it working I found what I considered a few shortcomings. The main one being the MicroPython repository was checked out into the docker container which made accessing files that were created a little difficult. So I decided to change it up just a little.

In my version I am only building the esp-open-sdk once and using a local checkout of the MicroPython repo so I can make changes if needed when building the actual firmware. My Dockerfile:

FROM debian:jessie

RUN apt-get update \
  && apt-get install -y build-essential make unrar-free autoconf automake libtool gcc g++ gperf \
  flex bison texinfo gawk ncurses-dev libexpat-dev python python-serial sed \
  git unzip bash wget libtool-bin help2man python-dev \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/* \
  && useradd --no-create-home esp

RUN git clone --recursive https://github.com/pfalcon/esp-open-sdk.git \
  && chown -R esp:esp ../esp-open-sdk

USER esp

RUN cd esp-open-sdk && make STANDALONE=y

ENV PATH=/esp-open-sdk/xtensa-lx106-elf/bin:$PATH

WORKDIR /opt

VOLUME ["/opt"]

CMD ["make", "-C" "esp8266"]

Once you’ve created the file Dockerfile and saved it you can run: docker build . -t esp-open-sdk.

After you have your docker image build you can checkout the MicroPython repo using the commands git clone https://github.com/micropython/micropython.git && cd micropython && git checkout v1.8.7 && git submodule update --init
You can replace the v1.8.7 with the current version of MicroPython or an older version if needed.
Now that you have the MicroPython repo it is time to finally start building our custom firmware.

For reference, I checked out my MicroPython repo to ~/Code/micropython so I will be using that path in my commands.

First you’ll need to build the cross-compiler so that your frozen modules can be correctly compiled down to the ESP8266 byte code.

docker run --rm -v ~/Code/micropython:/opt esp-open-sdk make -C mpy-cross

Then we need to build the rest of the MicroPython files:

docker run --rm -v ~/Code/micropython:/opt esp-open-sdk make -C esp8266 axtls
docker run --rm -v ~/Code/micropython:/opt esp-open-sdk

You should now be able to go to the esp8266/build directory and find the file firmware-combined.bin. This is the file containing the full firmware to be installed on your device.

The final steps are to flash this new firmware to your device and away you go!

esptool.py --port /dev/tty.usb* erase_flash
esptool.py --port /dev/tty.usb* --baud 115200 write_flash --verify --flash_size=detect -fm dio 0 firmware-combined.bin