• Home

  • Custom Ecommerce
  • Application Development
  • Database Consulting
  • Cloud Hosting
  • Systems Integration
  • Legacy Business Systems
  • Security & Compliance
  • GIS

  • Expertise

  • About Us
  • Our Team
  • Clients
  • Blog
  • Careers

  • CasePointer

  • VisionPort

  • Contact
  • Our Blog

    Ongoing observations by End Point Dev people

    Getting realtime output using Python Subprocess

    Kannan Ponnusamy

    By Kannan Ponnusamy
    January 28, 2015

    The Problem

    When I launch a long running unix process within a python script, it waits until the process is finished, and only then do I get the complete output of my program. This is annoying if I’m running a process that takes a while to finish. And I want to capture the output and display it in the nice manner with clear formatting.

    Using the subprocess and shlex library

    Python has a “batteries included” philosophy. I have used 2 standard libraries to solve this problem.

    import subprocess
    import shlex
    
    • subprocess—​Works with additional processes.
    • shlex—​Lexical analysis of shell-style syntaxes.

    subprocess.popen

    To run a process and read all of its output, set the stdout value to PIPE and call communicate().

    import subprocess
    process = subprocess.Popen(['echo', '"Hello stdout"'], stdout=subprocess.PIPE)
    stdout = process.communicate()[0]
    print 'STDOUT:{}'.format(stdout)
    

    The above script will wait for the process to complete and then it will display the output. So now we are going to read the stdout line by line and display it in the console untill it completes the process.

    output = process.stdout.readline()
    

    This will read a line from the stdout.

    process.poll()
    

    The poll() method will return

    • the exit code if the process is completed.
    • None if the process is still running.
    while True:
            output = process.stdout.readline()
            if output == '' and process.poll() is not None:
                break
            if output:
                print output.strip()
        rc = process.poll()
    

    The above will loop and keep on reading the stdout and check for the return code and displays the output in real time.

    I had one more problem in parsing the shell commands to pass it to popen when I set the shell=False. Below is an example command:

    rsync -avzXH --delete --exclude=*.swp --exclude=**/drivers.ini /media/lgisos/lg.iso root@42-a:/isodevice
    

    To split the string using shell-like syntax I have used shlex library’s split method.

    Here is the final code looks like

    def run_command(command):
        process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE)
        while True:
            output = process.stdout.readline()
            if output == '' and process.poll() is not None:
                break
            if output:
                print output.strip()
        rc = process.poll()
        return rc
    

    shell python


    Comments