• Home

  • Custom Ecommerce
  • Application Development
  • Database Consulting
  • Cloud Hosting
  • Systems Integration
  • Legacy Business Systems
  • Security & Compliance
  • GIS

  • Expertise

  • About Us
  • Our Team
  • Clients
  • Blog
  • Careers

  • VisionPort

  • Contact
  • Our Blog

    Ongoing observations by End Point Dev people

    Making sense of XML/JSON items in the shell

    Muhammad Najmi bin Ahmad Zabidi

    By Muhammad Najmi bin Ahmad Zabidi
    December 31, 2019

    a shell

    Working as a system administrator means I have to spend quite some time during my work (and even during casual surfing) with the terminal. Sometimes I feel that certain information I want could just be fetched and parsed through the terminal, without having to use my mouse and point to the browser.

    Some of the websites I visit use XML and JSON, which we could parse with Bash scripting. Previously I wrote a Ruby script to call Nokogiri to parse the XML elements until I found a Bash tool that could do the same thing.

    These tools have already been around for quite a while—I’d just like to share what I did with them. The tools I used are xmlstarlet for XML parsing and jq for JSON.


    I have the following XML elements, and I’ll save them to a file called data.xml:

    <rss version="2.0"
            <title>eSolat JAKIM : Waktu Solat Hari Ini</title>
            <link>Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam</link>
            <description>Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam</description>
            <dc:rights>Copyright JAKIM</dc:rights>
            <dc:date>26-12-2019 00:37:31</dc:date>
            <admin:generatorAgent rdf:resource="expressionengine" />

    I’ll use xmlstarlet, together with a bunch of the related flags to parse these elements into something which is more eye-friendly.

    xmlstarlet sel -T -t -n \
      -o "------------------------------" -n \
      -o "Area: " \
      -m "//channel" -v "link" -n  \
      -o "Date: " \
      -m "//channel" -v "dc:date" -n  \
      -o "------------------------------" -n \
      -t -m "//item" -o "Time: " -v "title" \
      -o " -- " \
      -o "Time: " -v "description" -n  \

    The output looks like this:

    Area: Gombak,Petaling,Sepang,Hulu Langat,Hulu Selangor,Rawang,S.Alam
    Date: 26-12-2019 00:37:31
    Time: Imsak -- Time: 05:53:00
    Time: Subuh -- Time: 06:03:00
    Time: Syuruk -- Time: 07:14:00
    Time: Zohor -- Time: 13:16:00
    Time: Asar -- Time: 16:39:00
    Time: Maghrib -- Time: 19:13:00
    Time: Isyak -- Time: 20:28:00

    I’ll put this in a Bash script, and call it xmlstarlet-time.sh.

    if [[ -z $1 ]]; then
      echo "Put the location code"
      echo "$0 <location code>"
      echo -n
    lynx -source "https://www.e-solat.gov.my/index.php?r=esolatApi/xmlfeed&zon=$1" > $XMLPATH
    xmlstarlet sel -T -t -n \
      -o "------------------------------" -n \
      -o "Area: " -m "//channel" -v "link" -n  \
      -o "Date: " -m "//channel" -v "dc:date" -n  \
      -o "------------------------------" -n \
      -t -m "//item" -o "Time: " -v "title" -o " -- " -o "Time: " -v "description" -n  \

    Now, after making it executable with chmod +x xmlstarlet-time.sh, I can just run the script whenever I need the info. In my case, I would type ./xmlstarlet-time.sh SGR01 in order to get the above information. I got the code (in my case) from the XML URL above. Your use case will likely be different.


    Let’s say I want to grab the latest currency exchange, taking the base of USD from exchangeratesapi.io. I can use curl to get the data.

    $ curl -s 'https://api.exchangeratesapi.io/api/latest?base=USD'

    Which will return:


    Using jq, we can format the information more readably:

    $ curl -s 'https://api.exchangeratesapi.io/api/latest?base=USD' | jq
      "rates": {
        "CAD": 1.3160649819,
        "HKD": 7.7879061372,
        "ISK": 122.3826714801,
        "PHP": 50.8402527076,
        "DKK": 6.7429602888,
        "HUF": 299.4223826715,
        "CZK": 23.0009025271,
        "GBP": 0.7719584838,
        "RON": 4.3131768953,
        "SEK": 9.4361913357,
        "IDR": 13985.0180505415,
        "INR": 71.2567689531,
        "BRL": 4.0835740072,
        "RUB": 62.0877256318,
        "HRK": 6.719765343,
        "JPY": 109.3772563177,
        "THB": 30.155234657,
        "CHF": 0.9817689531,
        "EUR": 0.9025270758,
        "MYR": 4.1364620939,
        "BGN": 1.7651624549,
        "TRY": 5.9561371841,
        "CNY": 7.0074909747,
        "NOK": 8.94566787,
        "NZD": 1.5086642599,
        "ZAR": 14.1935018051,
        "USD": 1,
        "MXN": 18.9626353791,
        "SGD": 1.3553249097,
        "AUD": 1.4457581227,
        "ILS": 3.4714801444,
        "KRW": 1162.1931407942,
        "PLN": 3.8445848375
      "base": "USD",
      "date": "2019-12-24"

    Next, I can make use of the tool in my shell script.

    #!/bin/bash -l
    if [ -z `which jq` ]; then
      printf "You need to install jq, a JSON parsing tool \n"
    sourcemoney=$(echo $2 | tr '[:lower:]' '[:upper:]')
    target=$(echo $3 | tr '[:lower:]' '[:upper:]')
    sumber=$(curl -s "https://api.exchangeratesapi.io/api/latest?base=$sourcemoney" | jq . | grep -i $target | awk -F ':|,' '{ print $2 }')
    jumlah=$(printf "%f*%f\n" $1 $sumber | bc)
    printf "Price per unit: ${GREEN}1 $sourcemoney${NC} = ${YELLOW}$target %.2f${NC}\n" $sumber
    echo -e "Source money: ${YELLOW}$sourcemoney $1${NC}"
    echo -n
    printf "Total money after the conversion: ${YELLOW}$target %.2f ${NC}\n" $jumlah

    Then I can save the script into a file called moneychanger-with-api.sh and make it executable with chmod +x moneychanger-with-api.sh.

    And now the script will do the parsing for me, without the need for a browser.

    $ ./moneychanger-with-api.sh 100 usd eur
    Price per unit: 1 USD =  EUR 0.90
    Source money: USD 100
    Total money after the conversion: EUR 90.25
    $ ./moneychanger-with-api.sh 100 eur sgd
    Price per unit: 1 EUR =  SGD 1.50
    Source money: EUR 100
    Total money after the conversion: SGD 150.17

    shell json