Parsing XML, JSON and YAML into Python

What it is

Parsing turns text in a data format into Python objects that your code can search and change.

Serialising is the reverse step. It turns Python objects back into text so you can send them or save them.

Once data is parsed, you work with ordinary dictionaries, lists, strings, numbers and booleans. You no longer think about braces or indentation.

How it works

Each format has a library, and each library maps the text onto Python types in a predictable way.

How JSON maps to Python

The built in json module needs no installation.

Four functions matter. json.loads() parses a string, and json.load() parses an open file.

json.dumps() makes a string, and json.dump() writes to a file. The trailing letter s stands for string. The indent argument makes the output easier to read, and sort_keys=True puts the keys in order.

import json

text = '{"hostname": "R1", "interfaces": [{"name": "Gi0/0", "ip": "10.0.0.1"}]}'
data = json.loads(text)

print(data["hostname"])                  # R1
print(data["interfaces"][0]["ip"])       # 10.0.0.1

data["hostname"] = "R2"
print(json.dumps(data, indent=2))

You read a nested value by chaining keys and list positions. A missing key raises KeyError, and data.get("key") returns None instead.

Parsing YAML

YAML needs the third party PyYAML package, which you import as yaml. Install it with pip install pyyaml.

import yaml

with open("devices.yaml") as f:
    devices = yaml.safe_load(f)

print(devices["interfaces"][0]["name"])
print(yaml.safe_dump(devices))

Use safe_load rather than load for data you did not write yourself. It builds only plain Python types and does not run code. Mappings become dictionaries and sequences become lists.

Parsing XML

Python has xml.etree.ElementTree in the standard library. It builds a tree of elements rather than a dictionary.

import xml.etree.ElementTree as ET

tree = ET.parse("devices.xml")
root = tree.getroot()

for device in root.findall("device"):
    print(device.find("hostname").text)
    print(device.attrib)

Each element has a tag, a text value and an attrib dictionary. Use find() for the first matching child and findall() for every match. ET.fromstring() parses a string instead of a file.

Many people prefer the third party xmltodict package. Its xmltodict.parse() function returns a dictionary, so you can use the same key lookups as with JSON.

Worked example with an API reply

The requests library has a shortcut. The response method .json() parses a JSON body into Python objects.

import os
import requests

BASE_URL = os.environ["API_BASE"]    # scheme and host, set outside the script
resp = requests.get(BASE_URL + "/devices")
devices = resp.json()
for d in devices["items"]:
    print(d["name"], d["status"])

Walking a parsed structure

Parsed data is often a dictionary that holds a list of dictionaries. A for loop visits each item in the list, and a key lookup picks the field you want.

for iface in data["interfaces"]:
    if iface["ip"].startswith("10.0.0."):
        print(iface["name"])

A common pattern is to loop, test a field, and collect the matches into a new list.

Parsing errors

A parser raises an error when the text is invalid.

• json.JSONDecodeError for bad JSON, such as single quotes or a trailing comma.
• yaml.YAMLError for bad YAML, such as a tab used for indentation.
• ET.ParseError for XML that is not well formed.