Python

Índex

General

Python
  • The Pyhton tutorial
  • PEP
  • Python (Raspberry Pi)
  • PyPI: the Python package index
  • Python Documentation contents
  • Unicode
    • Solving Unicode Problems in Python 2.7
      • UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xd1 in position 1: ordinal not in range(128) (Why is this so hard??)
    • Django i18n
      • Use format instead of '%'
        • do not forget trailing 'u'
          • u'{} {} '.format(...)
      • Use 'u' before literals
      •     string = u'{:02d}:{:02d}:{:02d}'.format(hours, minutes, seconds)
            if days:
                if days==1:
                    string = u'{} {} '.format(days,_("day")) + string
                    #string = u"%d %s %s" % ( days, _("day"), string)
                else:
                    string = u'{} {} '.format(days,_("days")) + string
                    #string = u"%d %s %s" % ( days, _("days"), string)

    • File writing
      • import codecs
        filename = os.path.join(...)
        f = codecs.open(filename,mode='w',encoding='utf-8')
        f.write(content)
        f.close        ()
      • import codecs
        filename = os.path.join(...)
        f = codecs.open(filename,mode='w',encoding='utf-8')
        fitxer = File(f)
        fitxer.write(content)
        fitxer.close
  • Eines / Tools
    • virtualenv
    • pip (*) (package manager) (also installed when virtualenv is installed)
      • Installation of pip itself
        • From distribution
          • CentOS
            • sudo yum install python-pip
        • From source
          • download it:
          • install it:
            • # python setup.py install
      • Installation of packages
        • pip install package_name
      • Installation of a precise version of a package
        • pip install djangorestframework==0.4.0
      • alpha version
        • pip install -pre package_name
      • upgrade
        • pip install -U package_name
      • Problems
        • Download error on https://pypi.python.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765) -- Some packages may not be found!
          • pip install fails with “connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)”
          • Solution
            • openssl s_client -connect pypi.python.org:443
            • curl -sO http://cacerts.digicert.com/DigiCertHighAssuranceEVRootCA.crt
            • sudo cp DigiCertHighAssuranceEVRootCA.crt /etc/pki/ca-trust/source/anchors/
            • sudo update-ca-trust
          • Alternative solution?
            • sudo yum install ca-certificates
        • pip install --upgrade -r pip_requirements.txt
          • Could not find .egg-info directory in install record for setuptools from https://pypi.python.org/packages/25/4e/1b16cfe90856235a13872a6641278c862e4143887d11a12ac4905081197f/setuptools-28.8.0.tar.gz#md5=43d6eb25f60e8a2682a8f826ce9e3f42 in /home/.../env/lib/python2.7/site-packages
          • see also problems with Google API and httplib2
        • error: Installed distribution setuptools 0.9.8 conflicts with requirement setuptools>=17.1
          • Solució / Solution
            • pip install --upgrade setuptools
        • Error: pg_config executable not found.
          • Solució / Solution
            • Install postgresql devel
              • Mageia
                • urpmi postgresql9.4-devel
        • list of installed packages (it gives the version number):
          • pip freeze
      • easy_install
        • urpmi python-setuptools
        • sudo apt-get install python-setuptools
      • Invoke
      • Fabric
        • Instal·lació / Installation
          • pip install fabric
        • Fabric 2.x
          • See also: Invoke
          • Upgrading from Fabric 1.x
            • 1.x
              		 2.x
              put(local_file, remote_dir, use_sudo=True)
              		 from os.path import basename

              def sudo_put(c, local_path, remote_dirname):
                  """
                  Upload a local file to a remote dir, with sudo privileges
                  """
                  filename = basename(local_path)
                  remote_tmp_path = filename
                  remote_path = '{}/{}'.format(remote_dirname, filename)

                  print 'sudo_put: {} -> {}'.format(local_path, remote_path)
                 
                  c.put(local_path, remote=remote_tmp_path)
                  c.run("sudo sh -c 'mv {} {}'".format(remote_tmp_path, remote_path))

              ...
              sudo_put(c, local_file, remote_dir)


          • Documentation (2.1)
        • Fabric 1.x documentation
        • Utilització / Usage
          • <task name>:<arg>,<kwarg>=<value>,...
        • debug from Eclipse
          • fabfile.py
            • ...
              from fabric.main import main

              if __name__ == '__main__':
                  import sys
                  sys.argv = ['fab', '-f', __file__, 'my_task']
                  main()
        • Exemples / Examples
          • put('toto.sh', '/usr/local/bin/', mode=int('755', 8), use_sudo=True)
        • Context managers
          • with ...
        • Error management
          • ignore
            • with settings(warn_only=True):
          • capture failure
            • result = local('grunt')
              if result.failed:
                  print "Grunt is not installed."
                  abort('Grunt not found')
          • Python try
            • try:
                  sudo('...')
              except Exception as e:
                  ...
                  abort('...')
        • env definitions
          • fabfile.py
            • from fabric.api import *

              # remote user and group
              env.remote_user = 'myuser'
              env.remote_group = 'mygroup'

              # project
              env.project_name = 'my_project'
              env.project_dir = '/home/%(remote_user)s/%(project_name)s' % env
        • run locally
          • running fabric script locally
          • Optionally avoid using ssh if going to localhost #98
          • fabfile.py
            • # by default, actions are performed remotely.
              # to perform them localy, e.g.: "fab localhost create_virtualenv"
              env.run_as = 'remote'
              env.run = run
              env.sudo = sudo
              env.hosts = ['...',]
              env.key_filename = '~/.ssh/keys/key_for_remote.pem'

              def localhost():
                  """
                  Set environment for local execution
                  """
                  env.run_as = 'local'
                  env.run = local
                  env.sudo = local
                  env.hosts = []
            • with lcd()
                  ...
        • capture
          • run remotelly and get the value
            • result = sudo(...)
            • result = run(...)
          • run locally and get the value
            • result = local(..., capture=True)
        • crontab
        • files
          • put
            • rsync_project
              • How do I copy a directory to a remote machine using Fabric?
              • Example
                • from fabric.contrib.project import rsync_project

                  rsync_project(local_dir='.', remote_dir='/var/www', exclude=('.git','tmp',) )
              • Ignore files indicated by .gitignore
              • Ignore untracked files and files indicated by .gitignore
                • untracked_files_zero = local('git -C .. ls-files -z -o --exclude-standard --directory', capture=True)
                  untracked_files = untracked_files_zero.split('\0')
                  print "untracked_files: {}".format(untracked_files)
                  excluded_files = untracked_files + ['.git','tmp']
                  rsync_project(local_dir='.', remote_dir=env.project_dir, exclude=excluded_files, extra_opts="--filter=':- .gitignore'" )
            • append
              • append a line
                • from fabric.contrib.files import append
                  ...
                      append('/etc/sysconfig/toto','this line has been appended to the end')
              • append several lines:
                • from fabric.contrib.files import append
                  ...
                      text = """
                  first_line = "value1"
                  second_line = "value2"
                  """
                      append('/etc/sysconfig/toto', text, use_sudo=True)
            • sed
              • single quotes in sed
              • replace a line that starts with "host    all             all             127.0.0.1" by "host    all             all             127.0.0.1/32            md5"
                • from fabric.contrib.files import sed
                  ...
                      # /var/lib/pgsql/data/pg_hba.conf
                      # host    all             all             127.0.0.1/32            md5
                      sed('/var/lib/pgsql/data/pg_hba.conf',
                          'host    all             all             127.0.0.1/32            ident',
                          'host    all             all             127.0.0.1/32            md5',
                          use_sudo=True )
                •     env.sudo('sudo sed -e "/^host    all             all             127.0.0.1/ c\host    all             all             127.0.0.1/32            md5" -i /var/lib/pgsql/data/pg_hba.conf')
          • Certificat de servidor / Server certificate
            • certificate for https connections (curl will not need to specify --cacert)
              • # add myserver self-signed certificate to the list of trust ca certificates
                put('myserver.example.org.crt', '/etc/pki/ca-trust/source/anchors/', use_sudo=True)
                env.sudo('update-ca-trust')
          • dependencies
            • fabfile.py
              • # list of dependencies to install
                env.install_cmd = 'yum install -y'
                env.dependencies = ['gcc', 'git', 'python-virtualenv', 'mariadb',]

                def install_dependencies():
                    """
                    Install the system dependencies for the project
                    """
                    env.sudo(env.install_cmd + " " + "epel-release")
                    env.sudo(env.install_cmd + " " + " ".join(env.dependencies))
          • virtualenv
            • Getting virtualenv(wrapper) and Fabric to play nice
            • Activate a virtualenv via fabric as deploy user
            • fabfile.py
              • from contextlib import contextmanager as _contextmanager

                # remote user and group
                env.remote_user = 'my_user'
                env.remote_group = 'my_group'

                # virualenv directory
                env.virtualenv_directory = '/opt/p27'
                env.virtualenv_activate = 'source %(virtualenv_directory)s/bin/activate' % env

                def create_virtualenv():
                    """
                    Create the virtualenv
                    """
                    env.sudo('mkdir -p %(virtualenv_directory)s' % env)
                    env.sudo('chown %(remote_user)s.%(remote_group)s %(virtualenv_directory)s' % (env) )
                    env.run('virtualenv %(virtualenv_directory)s' % env)

                @_contextmanager
                def virtualenv():
                    """
                    Activate the virtualenv
                    """
                    with cd(env.virtualenv_directory):
                        with prefix(env.virtualenv_activate):
                            yield

                def install_pip_dependencies():
                    """
                    Install the pip dependencies
                    """
                    with virtualenv():
                        if env.run_as == 'remote':
                            put('pip_requirements.txt', 'pip_requirements.txt')
                        env.run('pip install -r pip_requirements.txt')
          • Django
            • fabfile.py
              • def django_setup():
                    """
                    Django: migrate, createsuperuser, collectstatic
                    """
                    with virtualenv():
                        with cd(env.project_dir):
                            env.run('python manage.py migrate')
                            env.run('python manage.py createsuperuser')
                            env.run('python manage.py collectstatic')
                            env.run('python manage.py loaddata auth_initial')

                def django_update():
                    """
                    Django: migrate, collectstatic
                    """
                    with virtualenv():
                        with cd(env.project_dir):
                            env.run('python manage.py migrate')
                            env.run('python manage.py collectstatic')
          • Nginx
            • fabfile.py
              • def nginx_setup():
                    """
                    Configure and start nginx
                    """
                    env.sudo('chmod 755 /home/centos')
                    env.sudo('mkdir -p /etc/uwsgi/vassals/')
                    if env.run_as == 'remote':
                        # nginx
                        put('nginx-uwsgi/%(project_name)s_nginx.conf' % env, '/etc/nginx/conf.d/', use_sudo=True)
                        # remove default site from nginx.conf
                        put('nginx-uwsgi/nginx.conf', '/etc/nginx/', use_sudo=True)
                        put('nginx-uwsgi/nginx.pp', '/etc/nginx/', use_sudo=True)
                       
                        # uwsgi       
                        put('nginx-uwsgi/uwsgi_params', '/etc/uwsgi/', use_sudo=True)
                        put('nginx-uwsgi/emperor.ini', '/etc/uwsgi/', use_sudo=True)
                        put('nginx-uwsgi/%(project_name)s_uwsgi.ini' % env, '/etc/uwsgi/vassals/', use_sudo=True)
                        put('nginx-uwsgi/emperor.uwsgi.service', '/etc/systemd/system/', use_sudo=True)
                        # socket in /run/ (http://uwsgi-docs.readthedocs.org/en/latest/Systemd.html#putting-sockets-in-run)
                        #put('nginx-uwsgi/emperor.uwsgi.socket', '/etc/systemd/system/', use_sudo=True)
                        #put('nginx-uwsgi/emperor.uwsgi.conf', '/etc/tmpfiles.d/', use_sudo=True)
                   
                    # custom selinux policy module (http://axilleas.me/en/blog/2013/selinux-policy-for-nginx-and-gitlab-unix-socket-in-fedora-19/)
                    env.sudo('semodule -i /etc/nginx/nginx.pp')
                    # activate selinux
                    env.sudo('setenforce 1')

                    # enable and start nginx
                    env.sudo('systemctl enable nginx.service')
                    env.sudo('systemctl restart nginx.service')
                  
                    # enable and start uwsgi
                    env.sudo('systemctl enable emperor.uwsgi.service')
                    env.sudo('systemctl restart emperor.uwsgi.service')
                  
                    # configure firewall
                    #env.sudo('firewall-cmd --permanent --zone=public --add-service=http')
                    #env.sudo('firewall-cmd --permanent --zone=public --add-service=https')
                    #env.sudo('firewall-cmd --reload')


                def nginx_restart():
                    """
                    Restart nginx and wsgi
                    """

                    # restart nginx
                    env.sudo('systemctl restart nginx.service')
                  
                    # restart uwsgi
                    env.sudo('systemctl restart emperor.uwsgi.service')
          • Database
            • fabfile.py
              • # database
                env.mysql_host = 'localhost'
                env.mysql_database = 'mydatabase_db'
                env.mysql_user = 'my_user'
                env.mysql_password = 'my_password'
                env.mysql_master_user = 'root'

                def database_setup():
                    """
                    Setup database service
                    """
                    env.sudo("systemctl enable mariadb.service")
                    env.sudo("systemctl start mariadb.service")
                    env.sudo("mysql_secure_installation")


                def database_create():
                    """
                    Create the sql database
                    """
                    env.run('echo "CREATE DATABASE IF NOT EXISTS %(mysql_database)s; \
                GRANT ALL ON %(mysql_database)s.* TO \'%(mysql_user)s\'@\'%%\' IDENTIFIED BY \'%(mysql_password)s\'; \
                FLUSH PRIVILEGES;" | \
                mysql -h %(mysql_host)s -u %(mysql_master_user)s -p' % (env) )


                def database_delete():
                    """
                    Delete the sql database
                    """
                    env.run('echo "DROP DATABASE %(mysql_database)s;" | \
                mysql -h %(mysql_host)s -u %(mysql_master_user)s -p' % (env) )
          • Git
            • fabfile.py
              • def ssh_config():
                    """
                    Add fabuser_bitbucket_support to ~/.ssh/config
                    """
                    text = """
                Host fabuser-bitbucket
                     HostName bitbucket.org
                     IdentityFile ~/.ssh/fabuser_bitbucket
                    """
                    append('%s/config' % env.ssh_dir, text )
                    env.run('chmod 600 %s/config' % env.ssh_dir)

                def git_clone():
                    """
                    Clone from git
                    """
                    #git_user = 'francesc_pinyol_margalef'
                    with settings(warn_only=True):
                        with settings(warn_only=True):
                            if env.run("test -d %s" % env.project_dir).failed:
                                env.run("git clone git@fabuser-bitbucket:%(bitbucket_account)s/%(project_name)s.git %(project_dir)s" % (env) )


                def git_pull():
                    """
                    Pull from git
                    """
                    with cd(env.project_dir):
                        env.run("git pull")
        • Empaquetament / Packaging
      • Multiplpe platforms
      • Python path:
        • /usr/local/lib/python2.7/dist-packages/...
        • print path:
          • python
            • import sys
              sys.path
        • set path:
          • python
            • import sys
              sys.path.append("/my/path")
          • Django:
            • /etc/apache2/mods-available/wsgi.conf
              • WSGIPythonPath ...
        • recursively create directory if it does not exist:
          • # create the directory if it does not exist
            father_dir = os.path.dirname(filename)
            if not os.path.exists(father_dir):
                os.makedirs(father_dir)
                print "creating directory: %s" % father_dir
        • get home dir:
          • from os.path import expanduser
            home = expanduser("~")
        • get absolute dir inside home:
          • from os.path import expanduser
            my_dir = expanduser("~/my_dir")
      • Python for MS Windows:
      • HTML parsing
      • Logging

      Biblioteques / Libraries
              • post
                • import requests

                  address = ...
                  payload = {'toto_key':'toto_value',}
                  r = requests.post(address, payload)
              • jwt + get
                • import requests
                  from pprint import pprint

                  backend_url = 'http://127.0.0.1:8000'

                  # jwt token
                  # file toto_admin.txt contains the password
                  password = open('toto_admin.txt', 'r').read().strip()
                  payload_credentials = {
                                         'username': 'admin',
                                         'password': password
                                         }
                  r = requests.post(backend_url+'/api-token-auth/', data=payload_credentials)
                  token = r.json()['token']

                  r = requests.get('%s/v1/api/totos/' % (backend_url), headers={'Authorization':'JWT %s'%token})
                  pprint( r.json() )
              • upload (e.g. with Django Restframework)
              • HTTPS
            • Problemes / Problems
              • empty request.DATA when receiving a put/patch:
                • Solució / Solution
                  • check that there is a trailing "/". Otherwise, requests receives a 30x and data is lost on the second url
              • claudàtors en el payload / square brackets in payload
                • Solució / Solution: explicit conversion to json, with json header
                  • import requests
                    import json

                    payload = {
                               'toto':['first element','second element'],
                               }
                    r = requests.post(address, headers={'Content-Type':'application/json'}, data=json.dumps(payload))
                  • import requests
                    import json

                    # get token as usual
                    ...
                    token = ...

                    payload = {
                               'toto':['first element','second element'],
                               }
                    r = requests.post(address, headers={'Authorization':'JWT %s'%token, 'Content-Type':'application/json'}, data=json.dumps(payload))
          • pycurl (curl)
            • Exemple HHTPS / HTTPS Example
              • import pycurl
                curl = pycurl.Curl()
                curl.setopt(pycurl.CAINFO, "ca.crt")
                curl.setopt(pycurl.SSL_VERIFYPEER, 1)
                curl.setopt(pycurl.SSL_VERIFYHOST, 2)
                curl.setopt(pycurl.URL, "https://server_name/")
                curl.perform()
      • Parsing
      • Fulls de càlcul / Spreadsheet

      Exemples / Examples

      IDE
      • Comparison of integrated development environments: Python (wp)
      • IDLE (wp)
      • PyDev (Eclipse)
        • Installation
          • Help -> Install new software -> Add ...:
            • PyDev - http://pydev.org/updates
        • Django
        • New project
          • workspace: ~/src/djcode/
          • project name: nom_projecte
          • will create (according to new directory layout in Django 1.4):
            • ~/src/djcode/
              • nom_projecte/
                • .project
                  • <name>nom_projecte</name>
                • .pydevproject
                • sqlite.db
                • nom_projecte/
                  • settings.py
                  • ...
          • nom_projecte can be renamed to nom_projecte_pare (if you get the error "out of sync with file system", do a Refresh) (but DATABASES in settings.py is not updated):
            • ~/src/djcode/
              • nom_projecte_pare/
                • .project
                  • <name>nom_projecte_pare</name>
                • .pydevproject
                • sqlite.db
                • nom_projecte/
                  • settings.py
                  • ...
          • les noves aplicacions (nom_projecte_pare -> Django -> create new app) es crearan dins de nom_projecte (de fet, dins del directori especificat a .pydevproject com a DJANGO_MANAGE_LOCATION)
        • virtualenv
          • Integrar Virtualenv con Eclipse (PyDev)
          • Pydev and virtualenv
          • passos / steps
            • general configuration:
              • Window -> Preferences -> PyDev -> Interpreter - Python: New...
                • Interpreter Name: python-PYTHON27
                • Interpreter Executable: /opt/PYTHON27/bin/python
            • on your project:
              • Properties -> PyDev - Interpreter/Grammar
                • Interpreter: python-PYTHON27
        • Problems
          • debugging suspends on caught exceptions ("VariableDoesNotExist: Failed lookup for key...") related to Django templates:
            • Solució / Solution
              • PyDev -> Manage exception breakpoints
                • Uncheck: "Suspend on django template render exceptions"
          • Unable to read repository at http://pydev.org/updates/content.xml Transport initialization error..
            • Solution:
              • rm ~/.eclipse
              • eclipse
          • New added library (e.g. by using pip) not detected
            • Solution:
              • Window -> Preferences -> PyDev -> Interpreter - Python Interpreter -> Remove -> AutoConfig
        • Existing code (1.3)
          • djcode/
            • mysite/
              • settings.py
              • mystite.db
              • polls/

      GUI
      • tkinter

      Frameworks

      Google API

      http://www.francescpinyol.cat/python.html
      Primera versió: / First version: 24 VIII 2015
      Darrera modificació: 5 de desembre de 2018 / Last update: 5th December 2018

      Valid HTML 4.01!

      Cap a casa / Back home