Python

Índex

General

Python 3

Python
  • The Pyhton tutorial
  • PEP
  • Python (Raspberry Pi)
  • PyPI: the Python package index
  • Python Documentation contents
  • Unicode
    • Solving Unicode Problems in Python 2.7
      • UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xd1 in position 1: ordinal not in range(128) (Why is this so hard??)
    • Django i18n
      • Use format instead of '%'
        • do not forget trailing 'u'
          • u'{} {} '.format(...)
      • Use 'u' before literals
      •     string = u'{:02d}:{:02d}:{:02d}'.format(hours, minutes, seconds)
            if days:
                if days==1:
                    string = u'{} {} '.format(days,_("day")) + string
                    #string = u"%d %s %s" % ( days, _("day"), string)
                else:
                    string = u'{} {} '.format(days,_("days")) + string
                    #string = u"%d %s %s" % ( days, _("days"), string)
    • File writing
      • import codecs
        filename = os.path.join(...)
        f = codecs.open(filename,mode='w',encoding='utf-8')
        f.write(content)
        f.close        ()
      • import codecs
        filename = os.path.join(...)
        f = codecs.open(filename,mode='w',encoding='utf-8')
        fitxer = File(f)
        fitxer.write(content)
        fitxer.close
  • Eines / Tools
    • virtualenv
    • pip (*) (package manager) (also installed when virtualenv is installed)
      • Installation of pip itself
        • From distribution
          • CentOS
            • sudo yum install python-pip
        • From source
          • download it:
          • install it:
            • # python setup.py install
      • Installation of packages
        • pip install package_name
      • Installation of a precise version of a package
        • pip install djangorestframework==0.4.0
      • alpha version
        • pip install -pre package_name
      • upgrade
        • pip install -U package_name
      • Problems
        • error fatal: Python.h: El fitxer o directori no existeix
          • Solució / Solution
            • Python 2.7
              • Mageia
                • urpmi lib64python-devel
            • Python 3
              • Mageia
                • urpmi lib64python3-devel
        • Download error on https://pypi.python.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765) -- Some packages may not be found!
          • pip install fails with “connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)”
          • Solution
            • openssl s_client -connect pypi.python.org:443
            • curl -sO http://cacerts.digicert.com/DigiCertHighAssuranceEVRootCA.crt
            • sudo cp DigiCertHighAssuranceEVRootCA.crt /etc/pki/ca-trust/source/anchors/
            • sudo update-ca-trust
          • Alternative solution?
            • sudo yum install ca-certificates
        • pip install --upgrade -r pip_requirements.txt
          • Could not find .egg-info directory in install record for setuptools from https://pypi.python.org/packages/25/4e/1b16cfe90856235a13872a6641278c862e4143887d11a12ac4905081197f/setuptools-28.8.0.tar.gz#md5=43d6eb25f60e8a2682a8f826ce9e3f42 in /home/.../env/lib/python2.7/site-packages
          • see also problems with Google API and httplib2
        • error: Installed distribution setuptools 0.9.8 conflicts with requirement setuptools>=17.1
          • Solució / Solution
            • pip install --upgrade setuptools
        • Error: pg_config executable not found.
          • Solució / Solution
            • Install postgresql devel
              • Mageia
                • urpmi postgresql9.4-devel
      • list of installed packages (it gives the version number):
        • pip freeze
    • easy_install
      • urpmi python-setuptools
      • sudo apt-get install python-setuptools
    • Invoke
    • Fabric
      • Instal·lació / Installation
        • pip install fabric
      • Fabric 2.x
        • See also: Invoke
        • Upgrading from 1.x
          • 1.x
            		 2.x
            put(local_file, remote_dir, use_sudo=True)
            		 from os.path import basename

            def sudo_put(c, local_path, remote_dirname):
                """
                Upload a local file to a remote dir, with sudo privileges
                """
                filename = basename(local_path)
                remote_tmp_path = filename
                remote_path = '{}/{}'.format(remote_dirname, filename)

                print 'sudo_put: {} -> {}'.format(local_path, remote_path)
               
                c.put(local_path, remote=remote_tmp_path)
                c.run("sudo sh -c 'mv {} {}'".format(remote_tmp_path, remote_path))

            ...
            sudo_put(c, local_file, remote_dir)
            text = """
            FCGI_EXTRA_OPTIONS="-M 0770"
            """
            append('/etc/sysconfig/spawn-fcgi', text, use_sudo=True)
            		 text = """
            FCGI_EXTRA_OPTIONS=\\\"-M 0770\\\"
            """
            c.run("sudo sh -c 'echo \"{0}\" >>/etc/sysconfig/spawn-fcgi'".format(text))

            # not working yet:
            #from patchwork import files
            #files.append(c, '/etc/sysconfig/spawn-fcgi', text, sudo=True)
        • Documentation (2.1)
      • Fabric 1.x documentation
      • Utilització / Usage
        • <task name>:<arg>,<kwarg>=<value>,...
      • debug from Eclipse
        • fabfile.py
          • ...
            from fabric.main import main

            if __name__ == '__main__':
                import sys
                sys.argv = ['fab', '-f', __file__, 'my_task']
                main()
      • Exemples / Examples
        • put('toto.sh', '/usr/local/bin/', mode=int('755', 8), use_sudo=True)
      • Context managers
        • with ...
      • Error management
        • ignore
          • with settings(warn_only=True):
        • capture failure
          • result = local('grunt')
            if result.failed:
                print "Grunt is not installed."
                abort('Grunt not found')
        • Python try
          • try:
                sudo('...')
            except Exception as e:
                ...
                abort('...')
      • env definitions
        • fabfile.py
          • from fabric.api import *

            # remote user and group
            env.remote_user = 'myuser'
            env.remote_group = 'mygroup'

            # project
            env.project_name = 'my_project'
            env.project_dir = '/home/%(remote_user)s/%(project_name)s' % env
      • run locally
        • running fabric script locally
        • Optionally avoid using ssh if going to localhost #98
        • fabfile.py
          • # by default, actions are performed remotely.
            # to perform them localy, e.g.: "fab localhost create_virtualenv"
            env.run_as = 'remote'
            env.run = run
            env.sudo = sudo
            env.hosts = ['...',]
            env.key_filename = '~/.ssh/keys/key_for_remote.pem'

            def localhost():
                """
                Set environment for local execution
                """
                env.run_as = 'local'
                env.run = local
                env.sudo = local
                env.hosts = []
          • with lcd()
                ...
      • capture
        • run remotelly and get the value
          • result = sudo(...)
          • result = run(...)
        • run locally and get the value
          • result = local(..., capture=True)
      • crontab
      • files
        • put
        • rsync_project
          • How do I copy a directory to a remote machine using Fabric?
          • Example
            • from fabric.contrib.project import rsync_project

              rsync_project(local_dir='.', remote_dir='/var/www', exclude=('.git','tmp',) )
          • Ignore files indicated by .gitignore
          • Ignore untracked files and files indicated by .gitignore
            • untracked_files_zero = local('git -C .. ls-files -z -o --exclude-standard --directory', capture=True)
              untracked_files = untracked_files_zero.split('\0')
              print "untracked_files: {}".format(untracked_files)
              excluded_files = untracked_files + ['.git','tmp']
              rsync_project(local_dir='.', remote_dir=env.project_dir, exclude=excluded_files, extra_opts="--filter=':- .gitignore'" )
        • append
          • append a line
            • from fabric.contrib.files import append
              ...
                  append('/etc/sysconfig/toto','this line has been appended to the end')
          • append several lines:
            • from fabric.contrib.files import append
              ...
                  text = """
              first_line = "value1"
              second_line = "value2"
              """
                  append('/etc/sysconfig/toto', text, use_sudo=True)
        • sed
          • single quotes in sed
          • replace a line that starts with "host    all             all             127.0.0.1" by "host    all             all             127.0.0.1/32            md5"
            • from fabric.contrib.files import sed
              ...
                  # /var/lib/pgsql/data/pg_hba.conf
                  # host    all             all             127.0.0.1/32            md5
                  sed('/var/lib/pgsql/data/pg_hba.conf',
                      'host    all             all             127.0.0.1/32            ident',
                      'host    all             all             127.0.0.1/32            md5',
                      use_sudo=True )
            •     env.sudo('sudo sed -e "/^host    all             all             127.0.0.1/ c\host    all             all             127.0.0.1/32            md5" -i /var/lib/pgsql/data/pg_hba.conf')
      • Certificat de servidor / Server certificate
        • certificate for https connections (curl will not need to specify --cacert)
          • # add myserver self-signed certificate to the list of trust ca certificates
            put('myserver.example.org.crt', '/etc/pki/ca-trust/source/anchors/', use_sudo=True)
            env.sudo('update-ca-trust')
      • dependencies
        • fabfile.py
          • # list of dependencies to install
            env.install_cmd = 'yum install -y'
            env.dependencies = ['gcc', 'git', 'python-virtualenv', 'mariadb',]

            def install_dependencies():
                """
                Install the system dependencies for the project
                """
                env.sudo(env.install_cmd + " " + "epel-release")
                env.sudo(env.install_cmd + " " + " ".join(env.dependencies))
      • virtualenv
        • Getting virtualenv(wrapper) and Fabric to play nice
        • Activate a virtualenv via fabric as deploy user
        • fabfile.py
          • from contextlib import contextmanager as _contextmanager

            # remote user and group
            env.remote_user = 'my_user'
            env.remote_group = 'my_group'

            # virualenv directory
            env.virtualenv_directory = '/opt/p27'
            env.virtualenv_activate = 'source %(virtualenv_directory)s/bin/activate' % env

            def create_virtualenv():
                """
                Create the virtualenv
                """
                env.sudo('mkdir -p %(virtualenv_directory)s' % env)
                env.sudo('chown %(remote_user)s.%(remote_group)s %(virtualenv_directory)s' % (env) )
                env.run('virtualenv %(virtualenv_directory)s' % env)

            @_contextmanager
            def virtualenv():
                """
                Activate the virtualenv
                """
                with cd(env.virtualenv_directory):
                    with prefix(env.virtualenv_activate):
                        yield

            def install_pip_dependencies():
                """
                Install the pip dependencies
                """
                with virtualenv():
                    if env.run_as == 'remote':
                        put('pip_requirements.txt', 'pip_requirements.txt')
                    env.run('pip install -r pip_requirements.txt')
      • Django
        • fabfile.py
          • def django_setup():
                """
                Django: migrate, createsuperuser, collectstatic
                """
                with virtualenv():
                    with cd(env.project_dir):
                        env.run('python manage.py migrate')
                        env.run('python manage.py createsuperuser')
                        env.run('python manage.py collectstatic')
                        env.run('python manage.py loaddata auth_initial')

            def django_update():
                """
                Django: migrate, collectstatic
                """
                with virtualenv():
                    with cd(env.project_dir):
                        env.run('python manage.py migrate')
                        env.run('python manage.py collectstatic')
      • Nginx
        • fabfile.py
          • def nginx_setup():
                """
                Configure and start nginx
                """
                env.sudo('chmod 755 /home/centos')
                env.sudo('mkdir -p /etc/uwsgi/vassals/')
                if env.run_as == 'remote':
                    # nginx
                    put('nginx-uwsgi/%(project_name)s_nginx.conf' % env, '/etc/nginx/conf.d/', use_sudo=True)
                    # remove default site from nginx.conf
                    put('nginx-uwsgi/nginx.conf', '/etc/nginx/', use_sudo=True)
                    put('nginx-uwsgi/nginx.pp', '/etc/nginx/', use_sudo=True)
                   
                    # uwsgi       
                    put('nginx-uwsgi/uwsgi_params', '/etc/uwsgi/', use_sudo=True)
                    put('nginx-uwsgi/emperor.ini', '/etc/uwsgi/', use_sudo=True)
                    put('nginx-uwsgi/%(project_name)s_uwsgi.ini' % env, '/etc/uwsgi/vassals/', use_sudo=True)
                    put('nginx-uwsgi/emperor.uwsgi.service', '/etc/systemd/system/', use_sudo=True)
                    # socket in /run/ (http://uwsgi-docs.readthedocs.org/en/latest/Systemd.html#putting-sockets-in-run)
                    #put('nginx-uwsgi/emperor.uwsgi.socket', '/etc/systemd/system/', use_sudo=True)
                    #put('nginx-uwsgi/emperor.uwsgi.conf', '/etc/tmpfiles.d/', use_sudo=True)
               
                # custom selinux policy module (http://axilleas.me/en/blog/2013/selinux-policy-for-nginx-and-gitlab-unix-socket-in-fedora-19/)
                env.sudo('semodule -i /etc/nginx/nginx.pp')
                # activate selinux
                env.sudo('setenforce 1')

                # enable and start nginx
                env.sudo('systemctl enable nginx.service')
                env.sudo('systemctl restart nginx.service')
              
                # enable and start uwsgi
                env.sudo('systemctl enable emperor.uwsgi.service')
                env.sudo('systemctl restart emperor.uwsgi.service')
              
                # configure firewall
                #env.sudo('firewall-cmd --permanent --zone=public --add-service=http')
                #env.sudo('firewall-cmd --permanent --zone=public --add-service=https')
                #env.sudo('firewall-cmd --reload')


            def nginx_restart():
                """
                Restart nginx and wsgi
                """

                # restart nginx
                env.sudo('systemctl restart nginx.service')
              
                # restart uwsgi
                env.sudo('systemctl restart emperor.uwsgi.service')
      • Database
        • fabfile.py
          • # database
            env.mysql_host = 'localhost'
            env.mysql_database = 'mydatabase_db'
            env.mysql_user = 'my_user'
            env.mysql_password = 'my_password'
            env.mysql_master_user = 'root'

            def database_setup():
                """
                Setup database service
                """
                env.sudo("systemctl enable mariadb.service")
                env.sudo("systemctl start mariadb.service")
                env.sudo("mysql_secure_installation")


            def database_create():
                """
                Create the sql database
                """
                env.run('echo "CREATE DATABASE IF NOT EXISTS %(mysql_database)s; \
            GRANT ALL ON %(mysql_database)s.* TO \'%(mysql_user)s\'@\'%%\' IDENTIFIED BY \'%(mysql_password)s\'; \
            FLUSH PRIVILEGES;" | \
            mysql -h %(mysql_host)s -u %(mysql_master_user)s -p' % (env) )


            def database_delete():
                """
                Delete the sql database
                """
                env.run('echo "DROP DATABASE %(mysql_database)s;" | \
            mysql -h %(mysql_host)s -u %(mysql_master_user)s -p' % (env) )
      • Git
        • fabfile.py
          • def ssh_config():
                """
                Add fabuser_bitbucket_support to ~/.ssh/config
                """
                text = """
            Host fabuser-bitbucket
                 HostName bitbucket.org
                 IdentityFile ~/.ssh/fabuser_bitbucket
                """
                append('%s/config' % env.ssh_dir, text )
                env.run('chmod 600 %s/config' % env.ssh_dir)

            def git_clone():
                """
                Clone from git
                """
                #git_user = 'francesc_pinyol_margalef'
                with settings(warn_only=True):
                    with settings(warn_only=True):
                        if env.run("test -d %s" % env.project_dir).failed:
                            env.run("git clone git@fabuser-bitbucket:%(bitbucket_account)s/%(project_name)s.git %(project_dir)s" % (env) )


            def git_pull():
                """
                Pull from git
                """
                with cd(env.project_dir):
                    env.run("git pull")
    • Empaquetament / Packaging
  • Multiple platforms
  • Python path:
    • /usr/local/lib/python2.7/dist-packages/...
    • print path:
      • python
        • import sys
          sys.path
    • set path:
      • python
        • import sys
          sys.path.append("/my/path")
      • Django:
        • /etc/apache2/mods-available/wsgi.conf
          • WSGIPythonPath ...
    • recursively create directory if it does not exist:
      • # create the directory if it does not exist
        father_dir = os.path.dirname(filename)
        if not os.path.exists(father_dir):
            os.makedirs(father_dir)
            print "creating directory: %s" % father_dir
    • get home dir:
      • from os.path import expanduser
        home = expanduser("~")
    • get absolute dir inside home:
      • from os.path import expanduser
        my_dir = expanduser("~/my_dir")
  • Python for MS Windows:
  • HTML parsing
  • Logging
    • Django logging
    • Logging Cookbook
    • Exemple / Example:
      • # logger
        import logging

        logger = logging.getLogger('my_program')
        logger.setLevel(logging.DEBUG)
        # create console handler with a higher log level
        ch = logging.StreamHandler()
        ch.setLevel(logging.DEBUG)
        # create formatter and add it to the handlers
        formatter = logging.Formatter('%(asctime)s %(levelname)-8s %(name)-12s %(message)s', '%Y-%m-%dT%H:%M:%SZ')
        ch.setFormatter(formatter)
        # add the handlers to the logger
        logger.addHandler(ch)

        logger.debug("my message")
  • Nginx

Biblioteques / Libraries
  • Useful modules
    • Dades / Data
    • Bases de dades / Databases
    • Imatges / Images
    • Criptografia / Cryptography
    • Autenticació / Authentication
    • Xarxa / Network
      • Twisted
      • stomp.py
        • Problemes / Problems
          • UnicodeEncodeError: 'ascii' codec can't encode character ...
            • Solution
              • /usr/local/lib/python2.7/dist-packages/stomp/protocol.py
                • #body = encode(body)
    • Processos / Processes
      • 17.1 subprocess (2) (3)
        • subprocess.run (>=3.5)
        • Older high-level API
          • old
            		 new
            subprocess.call(...) subprocess.run(...)
            subprocess.check_call(...) subprocess.run(..., check=True)
            subprocess.check_output(...) subprocess.run(..., check=True, stdout=subprocess.PIPE).stdout
      • Exemples / Examples
        • ...
          • import shlex, subprocess

            command = '...'
            parameters = '...'
            command_line = "{0} {1}".format(command, parameters)
            args = shlex.split(command_line)

            try:
                completed_process = subprocess.run(args, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
                print(completed_process.stdout.decode('utf-8'))
                print(completed_process.stderr.decode('utf-8')) except Exception as e:
                print("ERROR: {0}".format(e))
                if e.stdout:
                    print("ERROR stdout: {0}".format(e.stdout.decode('utf-8')))
                if e.stderr:
                    print("ERROR stderr: {0}".format(e.stderr.decode('utf-8')))
        • stdout, stderr to a file:
          • import shlex, subprocess

            command = '/usr/bin/ffmpeg'
            parameters = '-y -i sintel.mp4 -vf "scale=1280:-1" -c:a copy -c:v h264 -f flv /tmp/toto.flv'
            command_line = "{0} {1}".format(command, parameters)
            args = shlex.split(command_line)

            with open("toto.log", "w") as f:
                try:
                    completed_process = subprocess.run(args, check=True, stdout=f, stderr=f)
                except Exception as e:
                    print("ERROR: {0}".format(e))
      • Problemes / Problems
    • SSH
      • Paramiko
        • Documentation
        • Example
          • import paramiko

            client = paramiko.SSHClient()
            #client.load_system_host_keys()
            k = paramiko.RSAKey.from_private_key_file('/home/my_user/.ssh/keys/remoteserver.pem')
            client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
            client.connect('remoteserver.example.org', username='myremoteuser', pkey=k)
            ssh_stdin, ssh_stdout, ssh_stderr = client.exec_command('ls -ltr')

            for line in ssh_stdout:
                print line
    • SVG
      • Creating Simple SVG from Python
      • pySVG
      • svgwrite
        • Documentation
        • mozman / svgwrite (github)
        • utf-8
          • # save svg to file
            from xml.etree.ElementTree import tostring
            dwg.tostring = lambda self=dwg: tostring(self.get_xml()).decode("utf-8")
            dwg.save(pretty=True)
      • convert svg to image
        • Convert SVG to PNG in Python
        • CairoSVG (Python3)
          • install Python3 and virtualenv
          • pip3 install cairosvg
          • as command:
            • cairosvg image.svg [--output_width 200] -o image.png
          • as library:
            • import cairosvg
              cairosvg.svg2pdf(url='image.svg',               output_width=200, write_to='image.pdf')
        • pyrsvg (librsvg)
          • pyrsvg How to use librsvg from Python (Cairo)
          • Installation
            • Mageia
              • urpmi gnome-python-desktop
                • /usr/lib64/python2.7/site-packages/gtk-2.0/rsvg.so
                • /usr/share/doc/gnome-python-desktop/rsvg/rsvg-cairo.py
            • Ubuntu
          • Ús / Usage
            • virtualenv --system-site-packages env
            • souce env/bin/activate
            • python toto.py
            • toto.py
              • import cairo
                import rsvg
                ...
    • FTP
      • libftp (2.7)
        • upload local or remote file to ftp:
          • # python 3
            import os
            import ftplib
            import urllib.request
            from urllib.parse import urlparse

            host = '...'
            username = '...'
            password = '...'

            #origin = '/tmp/toto.txt'
            origin = 'https://...'
            local_basename = os.path.basename(origin)
            remote_dirname = 'toto_dir_2'
            remote_path = os.path.join(remote_dirname, local_basename)

            print("ftp upload: {0} -> {1}".format(origin, remote_path))

            o = urlparse(origin)
            print("sheme: {}".format(o.scheme))

            if o.scheme:
                filehandle = urllib.request.urlopen(origin)
            else:
                filehandle = open(origin, 'rb')


            with ftplib.FTP(host, username, password) as ftp:
                # create dir
                try:
                    print("creating remote dir: {}".format(remote_dirname))
                    ftp.mkd(remote_dirname)
                except Exception as e:
                    print("WARNING when creating dir: {} - {}".format(e, type(e)))

                # upload file
                try:
                    print("uploading: {0} -> {1}".format(origin, remote_path))
                    ftp.storbinary("STOR {0}".format(remote_path), filehandle)
                except Exception as e:
                    print("ERROR when uploading file: {}".format(e))

            print("done")
        • upload remote http file directly to ftp (Python - Transfer a file from HTTP(S) URL to FTP/Dropbox without disk writing (chunked upload)):
          • ...
            import urllib
            ...
            filehandle = urllib.request.urlopen(origin_url)
            ...
    • HTTP / HTTPS
          • post
            • import requests

              address = ...
              payload = {'toto_key':'toto_value',}               r = requests.post(address, payload)
          • jwt + get
            • import requests
              from pprint import pprint

              backend_url = 'http://127.0.0.1:8000'

              # jwt token
              # file toto_admin.txt contains the password
              password = open('toto_admin.txt', 'r').read().strip()
              payload_credentials = {
                                     'username': 'admin',
                                     'password': password
                                     }
              r = requests.post(backend_url+'/api-token-auth/', data=payload_credentials)
              token = r.json()['token']

              r = requests.get('%s/v1/api/totos/' % (backend_url), headers={'Authorization':'JWT %s'%token})
              pprint( r.json() )
          • upload (e.g. with Django Restframework)
            • POST a Multipart-Encoded File
            • Exemple / Example:
              • models.py
                • class MyModel(models.Model):
                      field_1 = models.FileField(...)
                      field_2 = ...
                      field_3 = ...
              • upload_test.py
                • import requests

                  url = 'http...'
                  payload = {'field_2': '...', 'field_3': '...'}
                  headers = {}
                  headers['...'] = '...'                   files = {'field_1': open('/tmp/toto.png', 'rb')}
                  r = requests.post(url, payload, headers=headers, files=files)
          • HTTPS
        • Problemes / Problems
          • empty request.DATA when receiving a put/patch:
            • Solució / Solution
              • check that there is a trailing "/". Otherwise, requests receives a 30x and data is lost on the second url
          • claudàtors en el payload / square brackets in payload
            • Solució / Solution: explicit conversion to json, with json header
              • import requests
                import json

                payload = {
                           'toto':['first element','second element'],
                           }
                r = requests.post(address, headers={'Content-Type':'application/json'}, data=json.dumps(payload))
              • import requests
                import json

                # get token as usual
                ...
                token = ...

                payload = {
                           'toto':['first element','second element'],
                           }
                r = requests.post(address, headers={'Authorization':'JWT %s'%token, 'Content-Type':'application/json'}, data=json.dumps(payload))
      • pycurl (curl)
        • Exemple HHTPS / HTTPS Example
          • import pycurl
            curl = pycurl.Curl()
            curl.setopt(pycurl.CAINFO, "ca.crt")
            curl.setopt(pycurl.SSL_VERIFYPEER, 1)
            curl.setopt(pycurl.SSL_VERIFYHOST, 2)
            curl.setopt(pycurl.URL, "https://server_name/")
            curl.perform()
  • PyGObject (based on GObject)

    		• C
      		 Python
      • import gi
        gi.require_version("Gtk", "3.0")
        from gi.repository import Gtk
      		error when packages are not installed

      		-
      		 PyGObject
      • virtualenv: pip install pygobject
        • dependències / dependencies:
          • urpmi lib64girepository-devel gobject-introspection libcairo-devel [?python3-cairo-devel]
          • yum install gcc gobject-introspection-devel cairo-gobject-devel freetype-devel
          • apt-get ...
      • urpmi [python-gobject] python3-gobject3
      • yum install python34-gobject
      • apt-get ...
      • ModuleNotFoundError: No module named 'gi'
      • Package cairo was not found in the pkg-config search path.
      GObject libraries:
      • /usr/lib[64]/girepository-1.0/
      • GI_TYPELIB_PATH
      		GLib
      • GLib-2.0.typelib


      Gtk
      • Gtk-2.0.typelib
      • Gtk-3.0.typelib


      GStreamer
      • Gst-1.0.typelib
      • Gst...1.0.typelib
      • GES-1.0.typelib
      • ValueError: Namespace Gst not available
      • ValueError: Namespace GstWebRTC not available
      ...

    • Instal·lació / Installation
      • Sistema / System
        • CentOS
          • sudo yum install python34-gobject
        • Mageia
          • urpmi ...
      • Virtualenv + pip
        • Dependències / Dependencies
          • Install virtualenv
          • CentOS
            • sudo yum install gobject-introspection-devel cairo-gobject-devel freetype-devel
          • Mageia
            • urpmi lib64girepository-devel gobject-introspection
        • virtualenv-3.5 env (CentOS: virtualenv-3 --python python36 env)
        • source env/bin/activate
        • [pip install --upgrade pip]
        • pip install pygobject
    • Libraries are retrieved from:
      • standard location:
        • /usr/lib64/girepository-1.0/*.typelib
      • non-standard location (e.g. GStreamer compiled from source and installed into /usr/local)
        • export GI_TYPELIB_PATH=/usr/local/lib/girepository-1.0
    • PyGObject API Reference
    • Exemple / Example
      • import gi
        gi.require_version("Gtk", "3.0")
        from gi.repository import Gtk

        window = Gtk.Window(title="Hello World")
        window.show()
        window.connect("destroy", Gtk.main_quit)
        Gtk.main()
  • Parsing
  • Fulls de càlcul / Spreadsheet

Celery
  • ...
  • Celery with Django
  • Arquitectura / Architecture
  • First steps with Celery
    1. install a message broker (message transport)
      • RabbitMQ (AMQP)
        • RabbitMQ
        • Configuració / Setup (Django: same as in settings.py BROKER_URL)
          1. rabbitmqctl add_user myuser mypassword
          2. rabbitmqctl add_vhost myvhost
          3. rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"
        • check settings:
          • rabbitmqctl list_users
          • rabbitmqctl list_vhosts
          • rabbitmqctl list_permissions -p myvhost
      • Amazon SQS
    2. install celery (dependencies automatically installed: pytz billiard kombu anyjson amqp)
      • pip install celery
    3. check that celery is working
  • command line





    		• example
      celery





             		-b <broker> -A <module>
             		worker (start a single worker) -Q <queue1>,<queue2>
      --hostname=<node_name>@<host> (default: celery@localhost.localdomain)
      multi (start several named workers)
      • start <node_name_1> [<node_name_2>, ...]
        • --pidfile=/var/run/celery/%n.pid
        • --logfile=/var/log/celery/%n%I.log
      • restart <node_name> [<node_name_2>, ...]
      • stop <node_name> [<node_name_2>, ...]
      • stopwait <node_name> [<node_name_2>, ...]
      		-c <number_processes> (default: number of CPUs)

      		inspect
      • active
      		--destination=celery@example.com

      		control
      • enable_events
      • disable_events
      • rate_limit tasks.add 10/m



      		events --dump

      		status
      		celery -b amqp://<celery_user>:<celery_password>@<rabbitmq_server/><celery_vhost> status

      		...


  		• celery.py
    		 tasks.py
    		 worker
    		 usage


    		tasks.py
    • from celery import Celery

      app = Celery('tasks', broker='pyamqp://guest@localhost//')

      @app.task
      def add(x, y):
          return x + y
    		 celery -A tasks worker --loglevel=info
    • from tasks import add

      result = add.delay(4, 4)
      result.ready()
      result.successful()
      result.get()
      result.failed()
      res.state
      res.id



    		proj/rabbitmq.txt
    • amqp://<celery_user>:<celery_password>@<rabbitmq_server/><celery_vhost>
    proj/celery.py
    • from __future__ import absolute_import, unicode_literals
      from celery import Celery

      app = Celery('proj',
                   broker=open('rabbitmq.txt','r').read().strip(),
                   backend='rpc://',
                   include=['proj.tasks'])

      # Optional configuration, see the application user guide.
      app.conf.update(
          result_expires=3600,
      )

      if __name__ == '__main__':
          app.start()
    		 proj/__init__.py

    proj/tasks.py
    • from __future__ import absolute_import, unicode_literals
      from .celery import app

      @app.task
      def add(x, y):
          return x + y

      @app.task
      def mul(x, y):
          return x * y

      @app.task
      def xsum(numbers):
          return sum(numbers)
    		 (from proj parent dir)
    celery -A proj worker -l info
    		 (from proj parent dir)
    • from proj.tasks import add

      result = add.delay(4, 4)
    Django
    mysite/myproject/settings.py
    • # celery
      BROKER_URL = 'amqp://myuser:mypassword@localhost/myvhost'
    		mysite/myproject/celery.py
    • from __future__ import absolute_import
      import os
      from celery import Celery
      from django.conf import settings

      # set the default Django settings module for the 'celery' program.
      os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')

      app = Celery('myproject')

      # Using a string here means the worker will not have to
      # pickle the object when using Windows.
      app.config_from_object('django.conf:settings')

      # access to mysite/*/tasks.py
      app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

      # access to mysite/myproject/tasks.py
      app.autodiscover_tasks(lambda: ('myproject',))

      @app.task(bind=True)
      def debug_task(self):
          print('Request: {0!r}'.format(self.request))
    		mysite/myproject/__init__.py
    • from __future__ import absolute_import

      # This will make sure the app is always imported when
      # Django starts so that shared_task will use this app.
      from .celery import app as celery_app
    mysite/myproject/tasks.py
    • from __future__ import absolute_import
      import logging
      from celery import shared_task

      logger = logging.getLogger(__name__)

      @shared_task
      def add(x, y):
          logger.info("adding...")
          return x + y
    mysite/myapp/tasks.py
    • from __future__ import absolute_import
      import logging
      from celery import shared_task

      logger = logging.getLogger(__name__)

      @shared_task
      def mul(x, y):
          logger.info("multiplying...")
          return x * y
    		 celery -A myproject worker -l info

    		 e.g.: mysite/myapp/views.py
    • from myapp.tasks import mul

      result = mul.delay(3, 4)
  • Next steps
  • User Guide
  • Progrés / Progress
  • ...

Exemples / Examples
  • Python Command Line Arguments Examples


    		• list of arguments
      		 number of arguments
      		 script
      		 argument






      sys
      		 import sys
      		 # including script
      sys.argv      		 # including script
      len(sys.argv)      		 sys.argv[0] sys.argv[1]
      sys.argv[2]
      ...
      getopt
      		 import sys, getopt


      		try:
          myopts, args = getopt.getopt(sys.argv[1:],"i:o:")
      except getopt.GetoptError as e:
          print (str(e))
          print("Usage: %s -i input -o output" % sys.argv[0])
          sys.exit(2)
       
      for o, a in myopts:
          if o == '-i':
              ifile=a
          elif o == '-o':
              ofile=a
      argparse



      		import argparse
      parser = argparse.ArgumentParser(description='ADD YOUR DESCRIPTION HERE')
      parser.add_argument('-i','--input', help='Input file name', required=True)       parser.add_argument('-t','--title', metavar='titol', help='Input file name', required=False) parser.add_argument('-n','--number', type=int, help='Input file name', required=True) parser.add_argument('first_parameter', help='...')

      parser.add_argument('parameter_1')
      parser.add_argument('parameter_2')
      parser.add_argument('--disable', dest='is_enabled', action='store_false', required=False)
      parser.add_argument('-t', '--title', metavar='titol',
                          help='title to be put in field s= in sdp file',
                          required=False)
         
      args = parser.parse_args()
      print(args)
      print("parameter_1: {0}".format(args.parameter_1))
      print("title: {0}".format(args.titol))
  • print
    • print ("nombre_args: %d" % nombre_args)
  • Fusió d'intervals / Merge intervals

IDE
  • Comparison of integrated development environments: Python (wp)
  • IDLE (wp)
  • PyDev (Eclipse)
    • Installation
      • Help -> Install new software -> Add ...:
        • PyDev - http://pydev.org/updates
    • Django
    • New project
      • workspace: ~/src/djcode/
      • project name: nom_projecte
      • will create (according to new directory layout in Django 1.4):
        • ~/src/djcode/
          • nom_projecte/
            • .project
              • <name>nom_projecte</name>
            • .pydevproject
            • sqlite.db
            • nom_projecte/
              • settings.py
              • ...
      • nom_projecte can be renamed to nom_projecte_pare (if you get the error "out of sync with file system", do a Refresh) (but DATABASES in settings.py is not updated):
        • ~/src/djcode/
          • nom_projecte_pare/
            • .project
              • <name>nom_projecte_pare</name>
            • .pydevproject
            • sqlite.db
            • nom_projecte/
              • settings.py
              • ...
      • les noves aplicacions (nom_projecte_pare -> Django -> create new app) es crearan dins de nom_projecte (de fet, dins del directori especificat a .pydevproject com a DJANGO_MANAGE_LOCATION)
    • virtualenv
      • Integrar Virtualenv con Eclipse (PyDev)
      • Pydev and virtualenv
      • passos / steps
        • general configuration:
          • Window -> Preferences -> PyDev -> Interpreter - Python: New...
            • Interpreter Name: python-PYTHON27
            • Interpreter Executable: /opt/PYTHON27/bin/python
        • on your project:
          • Properties -> PyDev - Interpreter/Grammar
            • Interpreter: python-PYTHON27
    • Problems
      • debugging suspends on caught exceptions ("VariableDoesNotExist: Failed lookup for key...") related to Django templates:
        • Solució / Solution
          • PyDev -> Manage exception breakpoints
            • Uncheck: "Suspend on django template render exceptions"
      • Unable to read repository at http://pydev.org/updates/content.xml Transport initialization error..
        • Solution:
          • rm ~/.eclipse
          • eclipse
      • New added library (e.g. by using pip) not detected
        • Solution:
          • Window -> Preferences -> PyDev -> Interpreter - Python Interpreter -> Remove -> AutoConfig
    • Existing code (1.3)
      • djcode/
        • mysite/
          • settings.py
          • mystite.db
          • polls/

GUI
  • tkinter

Frameworks

Google API

http://www.francescpinyol.cat/python.html
Primera versió: / First version: 24 VIII 2015
Darrera modificació: 16 de gener de 2020 / Last update: 16th January 2020

Valid HTML 4.01!

Cap a casa / Back home