Internet / WWW

Index

General

3D

Analítica / Analytics

Aplicacions / Applications

Banda ampla / Broadband

CMS (content management) 

Correu electrònic / E-mail 

Dominis / Domains

Allotjament / Hosting


apps
manage
url
capacitat
baixada
comments
Google Drive
-

https://googledrive.com/host/0B42MQgikjmMwT05OV09ONDR4WkE/


Is it possible to have clean URLs for Google Drive items? [closed]
gdriv.es

http://gdriv.es/francescpinyol/


only highest level url
gweb.io




$1.50/month
Dropbox -

https://dl.dropboxusercontent.com/u/8062953/site/index.html
(Dropbox)


dockbox.io



dockbox.io – Host Websites in Dropbox With Custom Domains
harp.io




Harp.io – Host production websites with custom domains and SSL
KISSr
http://www.kissr.com/sites
http://francescpinyol.kissr.com/

50MB
50MB/mes
DropPages




based on given templates
Site44



100MB/mes

allotjament.cat






webgratis.cat


http://francescpinyol.webgratis.cat/
1GB
100GB/mes
PHP, MySQL


alta domini
renovació (¤/any)

emmagatzematge
tràfic
correu
altres
preus






comptes
espai
MySQL PHP
One Click
mensual
anual
active24
8
16,99 8
Basic Hosting
5GB
il·limitat


x
x
x
4,99 1,99

swhosting
8
17,75
Petit L
2GB
5GB/mes


-
x
-
3,50

Don Dominio (.cat)
9,95
22,95 17,95
Pla Mini
100MB
5GB/mes 5
50MB
-
-
-
0,08
1
Pla Basic
1GB
12GB/mes
100
300MB
100MB
x

2,08
25
Dina Hosting
8,95
18,60
Hosting Personal
2GB
30GB/mes
20




4,50

Virtual Pyme
8,95
18,95
Mínim
1GB 5GB/mes 10




2,45

CD Mon
8
19,95
Pla Start
1GB
10GB/mes
1

-
x

2,50
30
Entorno Digital
8
19,99
Pla Presència
2GB
10GB/mes 5

-
-

3,80


Feeds

Geolocalització / Geolocation

Grups de notícies / Newsgroups

Gestors de preferits / Bookmarks managers

Internet per satèl·lit / Internet via satellite

Internet Services Providers (ISP)

Internet TV

Internet of things

P2P 

Protocols

  • TCP and UDP port numbers (Wikipedia) nou
  • Internet Protocol (IP)
  • W3C
    • HTTP
    • HTTPS
      • Securing the web
      • Servidors / Servers
      • Clients
      • Server Name Indicator (SNI) (wp)
        • from nginx docs:
          • "With this configuration a browser receives the default server’s certificate, i.e. www.example.com regardless of the requested server name. This is caused by SSL protocol behaviour. The SSL connection is established before the browser sends an HTTP request and nginx does not know the name of the requested server. Therefore, it may only offer the default server’s certificate."
          • "A more generic solution for running several HTTPS servers on a single IP address is TLS Server Name Indication extension (SNI, RFC 6066), which allows a browser to pass a requested server name during the SSL handshake and, therefore, the server will know which certificate it should use for the connection."
        • SNI in nginx
      • HSTS
      • Marca blanca / White label
      • Let's Encrypt
        • wildcards
        • Protocols
        • Documentation
          • Challenge types
            • ACME
              challenge
              used by pros cons

              certbot plugin

              HTTP-01
              • apache
              • nginx
              • webroot
              • standalone
              • manual
              • It allows hosting providers to issue certificates for domains CNAMEd to them.
              • ...

              DNS-01
              • DNS plugins
                • ...
              • manual






        • ACME Clients
          • certbot
            • Documentation
            • Installation
            • Compilation
              • Requirements
                • CentOS
                  • sudo yum install python2-mock python-zope-interface python-zope-component python-six python-setuptools python2-acme
                • Mageia
                  • Compile certbot (recommended)
                  • or get it:
                    • wget https://dl.eff.org/certbot-auto
                    • chmod a+x certbot-auto
                    • ./certbot-auto -debug
                    • ./certbot-auto certonly
              • Download
                • git clone https://github.com/certbot/certbot.git
              • Compile
                • cd certbot
                • sudo python setup.py install
              • Compile version >=0.22 (for wildcards)
                • cd certbot
                • virtualenv env
                • source env/bin/activate
                • pip install --upgrade pip
                • pip install --upgrade setuptools
                • cd acme
                • python setup.py install
                • cd ..
                • python setup.py install
                • sudo ./env/bin/certbot ...
            • Configuració / Setup
              • serveis amb usuari no root / non-root services
                • How to use certs in non-root services?
                • letsencrypt_change_permissions.sh
                  • #!/bin/bash
                    # file permissions
                    chgrp ssl-cert -R /etc/letsencrypt
                    chmod 2755 /etc/letsencrypt
                    chmod g=rX -R /etc/letsencrypt
                • groupadd --gid 3000 ssl-cert change_letsencrypt_persmissions.sh
                  usermod -a -G ssl-cert nginx
                • crontab
                  • certbot -n renew --deploy-hook /usr/local/bin/letsencrypt_change_permissions.sh
            • Usage

              info
              usage from
              additional setup for running server


              CLI options /etc/letsencrypt/renewal/your.domain.org.conf
              Apache Nginx


              certbot ...
              [renewalparams]
              /etc/httpd/httpd.conf
              /etc/nginx/nginx.conf
              authentication
              when your html pages are on a static server (hosted, AWS S3...)
              --authenticator manual



              when you have no running server, but ports 80 and 443 are available. A mini server will be started just for the process.
              E.g.: Postfix email server
              --authenticator standalone authenticator = standalone


              when you have a running server (certbot will generate some content locally, in /path/to/.well-known/, which must be available by http from letsencrypt servers; this requires additional setup of your server)
              Can also be used to obtain a certificate for Postfix (-d mail.domain.org) when an http server is already running (even if it is  running with server_name your.domain.org). But make sure that you have a DNS entry: mail.domain.org CNAME your.domain.org
              --authenticator webroot -w /path/to
              authenticator = webroot
              [[webroot_map]]
              your.domain.org = /path/to

              (needed?)
              <VirtualHost *:80>
                  Alias /.well-known /path/to/.well-known

                  <Directory "/path/to/.well-known"> 
                      Options Indexes FollowSymLinks MultiViews 
                      AllowOverride All 
                      Require all granted   
                  </Directory>


              server {
                  listen      80;
                 
                  server_name         your.domain.org;

                  # letsencrypt
                  location /.well-known {
                      alias /path/to/.well-known;
                  }

              when you have a running Apache server --authenticator apache authenticator = apache


              when you have a running Nginx server --authenticator nginx authenticator = nginx








              installation
              you do not want to install the obtained certificate
              certonly
              installer = None


              you want to install the obtained certificate in the Apache config
              --installer apache
              installer = apache


              you want to install the obtained certificate in the Nginx config --installer nginx
              installer = nginx


              • Examples
                • Just retrieve a certificate (a nginx server is running):
                  • webroot=/usr/share/nginx/html
                    letsencrypt_port=80
                    key_size=4096
                    email=me@my_company.com
                    domain=my_subdomain.my_company.com
                    sudo certbot certonly -n --agree-tos --webroot -w $webroot --http-01-port $letsencrypt_port --rsa-key-size $key_size --email $email -d $domain
                • just retrieve a wildcard certificate (requires certbot version >=0.22)
                  • install the required dns plugin
                  • sudo -i
                    letsencrypt_port=80
                    key_size=4096
                    email=me@my_company.com
                    domain='*.my_company.com'
                    certbot certonly -n --agree-tos --dns-route53 --http-01-port $letsencrypt_port --rsa-key-size $key_size --email $email -d $domain
                    --server https://acme-v02.api.letsencrypt.org/directory
                  • Problemes / Problems
                    • The currently selected ACME CA endpoint does not support issuing wildcard certificates.
                      • Solució / Solution
                        • --server https://acme-v02.api.letsencrypt.org/directory
                • just retrieve a certificate using AWS Route 53
                  • install the required dns plugin
                  • sudo -i
                    email=me@my_company.com
                    domain='mydomain.my_company.com'

                    certbot certonly [--dry-run] -n --agree-tos --dns-route53 --key-type ecdsa --email $email -d $domain
              • Renewal
                • Renewing certificates
                  • --pre-hook
                  • --post-hook
                  • --deploy-hook
                • all certificates:
                  • certbot -n renew
                • specified certificates:
                  • certbot certonly -n --authenticator webroot --webroot-path /path/to -d your.domain.org
          • Client options
        • Apache
          • first time
            • certbot run --apache --http-01-port 80 --rsa-key-size 4096 --email your@email.org -d your.domain.org
            • non interactive:
              • certbot run --apache -n --agree-tos --http-01-port 80 --rsa-key-size 4096 --email your@email.org -d your.domain.org
          • first renewal (because apache authenticator is not working for certain apache config)
            • certbot certonly -n --authenticator webroot --webroot-path /path/to/ -d your.domain.org
          • next renewals
            • sudo certbot -n renew
            • Problemes / Problems
              • /etc/letsencrypt/options-ssl-apache.conf not found
                • Solució / Solution
                  • sudo yum install python2-certbot-apache
                  • ln -s /usr/lib/python2.7/site-packages/certbot_apache/options-ssl-apache.conf /etc/letsencrypt/options-ssl-apache.conf
              • Could not open configuration file /etc/letsencrypt/options-ssl-apache.conf: Permission denied
                • This happens because /etc/letsencrypt is a symbolic link to /mnt/nfs/letsencrypt, a nfs-mounted point
                • Solució / Solution
        • Nginx (using webroot)
          • first time
            • certbot certonly --webroot -w /path/to --http-01-port 80 --rsa-key-size 4096 --email your@email.org -d your.domain.org
            • non interactive:
              • certbot certonly -n --agree-tos --webroot -w /path/to --http-01-port 80--rsa-key-size 4096 --email your@email.org -d your.domain.org
          • renewal
            • /etc/nginx/conf.d/my_site.conf
              • server {
                    # letsencrypt
                    location /.well-known {
                        alias /path/to/.well-known;
                    }
                    ...
                }
            • check that /etc/letsencrypt/renewal/my.site.org.conf contains:
              • ...
                [renewalparams]
                authenticator = webroot
                webroot_path = /path/to
                installer = None
                ...
            • sudo certbot renew --post-hook "systemctl restart nginx.service"
            • crontab
              • certbot -q -n renew --post-hook "systemctl restart nginx.service"
        • Certificate installation in hosting services
          1. Get only the certificate (follow the instructions)
            • domain=my_subdomain.my_company.org
              sudo certbot certonly --authenticator manual -d $domain
          2. Install it:
            • DonDominio
              • Alojamiento web / Subdominios
                • CERTIFICADO:
                  • xclip -selection clipboard < /etc/letsencrypt/live/${domain}/cert.pem
                  • CERTIFICADO: CTRL-V
                • CLAVE SSL: 
                  • xclip -selection clipboard < /etc/letsencrypt/live/${domain}/privkey.pem
                  • CLAVE SSL: CTRL-V
                • CERTIFICADOS DE AUTORIDAD: 
                  • xclip -selection clipboard < /etc/letsencrypt/live/${domain}/chain.pem
                  • CERTIFICADOS DE AUTORIDAD: CTRL-V
                  • elimineu el segon certificat que acabeu d'enganxar dins de la casella CERTIFICADOS DE AUTORIDAD
                  • 247
              • panel247
        • AWS
          • if you want to use certificates for LoadBalancer or CloudFront, consider using AWS ACM instead
          • EC2
          • Elastic Load Balancer (ELB)
            • Let's encrypt and ELBs?
            • First time
              1. create a load balancer with a port 80 listener
              2. AWS IAM: grant permissions to user/role that will be used in next steps
                • elasticloadbalancing:CreateLoadBalancerListeners
                • iam:GetServerCertificate
                • iam:UploadServerCertificate
                • Example of policy:
                  • {
                        "Version": "2012-10-17",
                        "Statement": [
                            {
                                "Sid": "",
                                "Effect": "Allow",
                                "Action": [
                                    "elasticloadbalancing:DescribeLoadBalancers",
                                    "elasticloadbalancing:CreateLoadBalancerListeners",
                                    "elasticloadbalancing:SetLoadBalancerListenerSSLCertificate"
                                ],
                                "Resource": [
                                    "*"
                                ]
                            },
                            {
                                "Sid": "",
                                "Effect": "Allow",
                                "Action": [
                                    "iam:ListServerCertificates",
                                    "iam:GetServerCertificate",
                                    "iam:UploadServerCertificate"
                                ],
                                "Resource": [
                                    "*"
                                ]
                            }
                        ]
                    }
              3. from an instance (e.g. from an instance behind the load balancer)
                1. get a LE certificate (IMPORTANT: keys with 4096 bytes are not allowed; only 2048 or 1024 (not allowed by Let's Encrypt))
                  • certbot ... --rsa-key-size 2048 ...
                2. upload the certificate to IAM (Upload the Certificate):
                  • aws iam upload-server-certificate --server-certificate-name ${domain}.cert \
                    --certificate-body file:///etc/letsencrypt/live/${domain}/cert.pem \
                    --private-key file://
                    /etc/letsencrypt/live/${domain}/privkey.pem \
                    --certificate-chain file://
                    /etc/letsencrypt/live/${domain}/chain.pem
                3. Get the ARN of the uploaded certificate
                  • ARN=$(aws --output json iam get-server-certificate --server-certificate-name ${domain}.cert | jq '.ServerCertificate.ServerCertificateMetadata.Arn')
                4. Add an HTTPS Listener Using the AWS CLI
                  • aws elb create-load-balancer-listeners --load-balancer-name my-load-balancer --listeners Protocol=HTTPS,LoadBalancerPort=443,InstanceProtocol=HTTPS,InstancePort=443,SSLCertificateId=$ARN
                5. ...
            • Renewal
              • letsencrypt-aws
                • Installation
                  • git clone https://github.com/alex/letsencrypt-aws.git
                  • cd letsencrypt-aws
                  • virtualenv env
                  • source env/bin/activate
                  • pip install -r requirements.txt
                  • deactivate
                • First time
                  • ask ACME for a private key
                    • source env/bin/activate
                    • python letsencrypt-aws.py register email@host.com
                    • deactivate
                    • save it as email_host_com.privkey.acme.pem (locally or on AWS S3)
                  • create an IAM policy (name it e.g. LetsencryptAWS)
        • Problemes / Problems
          • The requested nginx plugin does not appear to be installed? #1736
          • Solution: manual request and installation
            • cd letsencrypt
            • ./letsencrypt-auto certonly -a manual --rsa-key-size 4096 --email your@email.org -d your.domain.org
            • /etc/nginx/nginx.conf (or /etc/nginx/conf.d/your_site.conf)
              • ssl_certificate     /etc/letsencrypt/live/www.example.org/fullchain.pem;
                ssl_certificate_key /etc/letsencrypt/live/www.example.org/privkey.pem;
            • Note: if you specify cert.pem instead of fullchain.pem, browsers will work, but curl nor openssl s_client won't:
    • Content negotiation
    • JPEG JFIF
    • SOAP (Simple Object Access Protocol)
    • Platform for Privacy Preferences (P3P) (press release)
    • Resource Description Framework (wp) (metadades/metadata)
    • Media Fragments URI (MFWG)
    • WebVTT (Web Video Text Tracks)
    • Cross-Origin Resource Sharing (CORS)
      • Access-Control-Allow-Origin: http://foo.example/
    • W3C TV
  • IETF RFC
  • Protocols de missatgeria / Messaging protocols

    • broker implementation (MOM) clients


      Android
      AMQP
      MQTT
      STOMP
    • AMQP (Advanced Message Queuing Protocol) (wp)
    • MQTT (Message Queue Telemetry Transport) (wp)
    • STOMP (Streaming Text Oriented Messaging Protocol) (wp)
      • to connect to message-oriented middleware
      • STOMP Protocol Specification, Version 1.2
      • Frames and headers

        frame standard headers
        request headers response headers


        content-length content-type receipt

        connection frames CONNECT MAY

        • accept-version
        • host
        • login
        • passcode
        • heart-beat

        CONNECTED MAY


        • version
        • heart-beat
        • session
        • server
        client frames SEND MAY SHOULD MAY
        • destination
        • transaction

        SUBSCRIBE MAY
        MAY
        • destination
        • id
        • ack: [auto, client, client-individual]

        UNSUBSCRIBE MAY
        MAY
        • id

        BEGIN MAY
        MAY
        • transaction

        COMMIT MAY
        MAY
        • transaction

        ABORT MAY
        MAY
        • transaction

        ACK MAY
        MAY
        • id
        • transaction

        NACK MAY
        MAY
        • id
        • transaction

        DISCONNECT MAY
        MAY
        • receipt

        server frames MESSAGE MAY SHOULD

        • destination
        • message-id
        • subscription
        • ack
        RECEIPT MAY


        • receipt-id
        ERROR MAY SHOULD

        • receipt-id
        • message
      • Server
      • Client
        • CLI
          • stomp (from stomp.py)
            • stomp --verbose --ssl -H b-xxxx-1.mq.eu-south-2.amazonaws.com -P 61614 --user=myuser --password=mypassword
              • subscribe /queue/my_queue
              • ...
            • stomp --verbose -H vis.musicradio.com -P 61613
              • subscribe /topic/fm/ce1/c479/09580/image
              • ...
          • telnet/ncat client (producer: SEND; consumer: SUBSCRIBE)
            • RadioDNS
            • Notes
              • to end every frame:
                • ^@ = Ctrl+@
              • to end communication:
                • when using ncat:
                  • (just DISCONNECT)
                • when using telnet:
                  • ^] = Ctrl+]
            • frame (you must put an empty line between header and body and end the frame with ^@)
              • COMMAND
                header1:value1
                header2:value2

                Body^@
            • telnetncat vis.musicradio.com 61613
              • Trying 81.20.48.151...
                Connected to 81.20.48.151.
                Escape character is '^]'.

              • CONNECT

                ^@
              • SUBSCRIBE
                destination: /topic/fm/ce1/c479/09580/image

                ack: auto

                ^@
              • DISCONNECT
                receipt:77

                ^@
        • Python
  • HTTP headers
  • HTTP request methods (wp)
    • method
      description
      note
      usage
      related HTML tag



      safe
      idempotent

      HEAD
      same as GET, but without the body




      GET
      requests a representation of the resource
      only to retrieve
      y
      y
      href
      POST
      submits data to be processed

      n
      n
      form method="post"
      PUT
      uploads a representation of the resource

      n
      y

      DELETE
      deletes the resource




      TRACE
      echoes the received request




      OPTIONS
      http methods supported by the server




      CONNECT





      PATCH
      apply partial modifications to a resource



    • Ús / Usage
    • curl -X http_method
    • Django
    • POST
      • generation
        received by CGI
        html form
        curl
        content-type
        as standard input
        <form action="my_cgi.sh" method="post">
        <input type="text" name="tata" value="tata_value">
        <input type="text" name="titi" value="titi_value">
        <input type="submit">
        </form>
        • curl -i -X POST --data 'tata=tata_value' --data 'titi=titi_value'
        • curl -i -H "Content-Type: application/x-www-form-urlencoded" -d 'tata=tata_value&titi=titi_value'
        • curl -i -X POST --data-binary 'tata=tata_value' --data-binary 'titi=titi_value'
        application/x-www-form-urlencoded
        tata=tata_value&titi=titi_value
        <form action="my_cgi.sh" method="post" enctype="multipart/form-data">
        <input type="text" name="tata" value="tata_value">
        <input type="text" name="titi" value="titi_value">
        <input type="submit">
        </form>
        • curl -i -X POST -F tata=tata_value -F titi=titi_value
        • curl -i -X POST -H 'Content-Type: multipart/form-data' -F tata=tata_value -F titi=titi_value
        multipart/form-data; boundary=------------------------6e0b5ea0578a6399
        --------------------------6e0b5ea0578a6399
        Content-Disposition: form-data; name="tata"

        tata_value
        --------------------------6e0b5ea0578a6399
        Content-Disposition: form-data; name="titi"

        titi_value
        --------------------------6e0b5ea0578a6399--


        • curl -i -X POST -H 'Content-Type: application/json' --data '{"toto":"toto_value","tata":"tata_value"}'
        • url="https://127.0.0.1:8000/"
          my_var="toto value";
          json_code="{\"toto\":\"${my_var}\"}";
          curl_options="
          -H 'Content-Type: application/json' --data '$json_code'";
          curl_command="curl -v -X POST $curl_options '$url'";
          eval "$curl_command"
        • my_var="toto l'educat"; my_escaped_var=$(echo "${my_var}" | sed "s/'/'\\\''/g");
          json_code="{\"toto\":\"${my_escaped_var}\"}";
          curl_options="
          -H 'Content-Type: application/json' --data '$json_code'";
          curl_command="curl -v -X POST $curl_options '$url'";
          eval "$curl_command"
        application/json

  • HTTP status codes
    • Response Status Codes (RFC 7231, Section 6)
    • Hyper Text Coffee Pot Control Protocol (HTCPCP/1.0) (RFC 2324)
      • 418
    • List of HTTP status codes (wp)
    • Setting HTTP Status Codes (Servlet tutorial)
    • family
      description
      code
      1xx
      Informational
      • 100 Continue
      • 101 Switching Protocols
      • 102 Processing
      • 103 Early Hints
      2xx
      Success
      • 200 OK
      • 201 Created
      • 202 Accepted
      • ...
      3xx
      Redirection
      • 300 Multiple Choices
      • 301 Moved Permanently (POST is switched to GET)
      • ...
      • 308 Permanent Redirect (do not switch methods)
      4xx
      Client error
      • 400 Bad Request
      • 401 Unauthorized
      • 402 Payment Required
      • 403 Forbidden
      • 404 Not Found
      • 405 Method Not Allowed
      • ...
      • 408 Request Timeout
      • ...
      • 409 Conflict
      • ...
      • 415 Unsupported Media Type
      • ...
      • 418 I'm a teapot
      • ...
      • 429 Too Many Requests
      • ...
      • 451 Unavailable For Legal Reasons
      • ...
      • 460 The load balancer received a request from a client, but the client closed the connection with the load balancer before the idle timeout period elapsed.
      5xx
      Server error
      • 500 Internal Server Error
      • 501 Not Implemented
      • 502 Bad Gateway
      • 503 Service Unavailable
      • 504 Gateway Timeout
      • 505 HTTP Version Not Supported
      • ...
    • 4xx
  • DOCTYPE / DTD 
  • Emmagatzematge / Storage
  • Same origin policy (wp)
  • JSON-RPC
    • Info
    • Basics
      • request frame:
        • method attribute (mandatory)
        • params attribute (optional)
      • if request frame has an optional id attribute, then a server must send a response frame
      • frames without id are considered notification frames; server does not response on notification frames
      • communication is bidirectional
    • Over
  • Push
    • Comet / Ajax push (wp)
    • WebSocket (wp) (HTML5)
      • RFC 6455
      • WebSocket in Django
      • Websockets load-balancing with HAProxy
      • 9 Websocket Servers for Reliable Real-time Applications
      • WebSocket (Full Stack Python)
        • arquitectura
      • Escalabilitat / Scalability
      • Eines / Tools
        • nginx
        • ...
          format server client
          raw CLI
          Languages
          CLI
          • wscat -c ws://127.0.0.1:3000
          • websocat ws://127.0.0.1:3000
          • iocat ws://127.0.0.1:3000
          • Weasel
          Languages
          JSON-RPC
          Languages
          socket.io CLI
          • iocat --socketio -l -p 3000
          Languages
          CLI
          • iocat --socketio ws://127.0.0.1:3000
            • as iocat is using ?EIO=3, error "Unsupported protocol version" is returned; to solve this, new (v4) socket.io server has to be started with option:
              • js:
                • const io = new Server(server, {
                      allowEIO3: true // false by default
                  });
              • python: ...
          • iocat --emit-key "chat message" --socketio ws://127.0.0.1:3000/
          Languages

        • info instal·lació / installation docs servidor / server client
          Firefox



          Responses
          • raw
          • JSON
          • Socket.IO
          • ...
          Weasel




          websocat
          • Rust toolchain (Mageia)
            • curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
            • cargo install --features=ssl websocat
          • ...
          • websocat -s 1234
          • websocat ws://127.0.0.1:1234/
          • websocat ws://ws.vi-server.org/mirror
          • websocat --jsonrpc ...
          wscat
          npm install -g wscat

          • wscat -c ws://websocket-echo.com
          ws
          • plain




          socket.io
          • bash
          • JavaScript (Node.js)
            • socket.io
              • import { Server } from "socket.io";

                const io = new Server(3000);

                io.on("connection", (socket) => {
                    // send a message to the client
                    socket.emit("hello", "world");

                    // receive a message from the client
                    socket.on("howdy", (arg) => {
                        console.log(arg); // prints "stranger"
                    });
                });
              • import { createServer } from "http";
                import { Server } from "socket.io";

                const httpServer = createServer();
                const io = new Server(httpServer, {
                  path: "/my-custom-path/"
                });
              • const express = require('express');
                const app = express();
                const http = require('http');
                const server = http.createServer(app);
                const { Server } = require("socket.io");
                const io = new Server(server);

                app.get('/', (req, res) => {
                  res.sendFile(__dirname + '/index.html');
                });

                io.on('connection', (socket) => {
                    console.log('a user connected');
                    socket.on('disconnect', () => {
                        console.log('user disconnected');
                    });
                    socket.on('chat message', (msg) => {
                        io.emit('chat message', msg);
                    });
                });

                server.listen(3000, () => {
                  console.log('listening on *:3000');
                });
              • ...
          • Python
            • python-socketio
              • from aiohttp import web
                import socketio

                sio = socketio.AsyncServer()
                app = web.Application()
                sio.attach(app)

                async def index(request):
                    """Serve the client-side application."""
                    with open('index.html') as f:
                        return web.Response(text=f.read(), content_type='text/html')

                @sio.event
                def connect(sid, environ):
                    print("connect ", sid)

                @sio.event
                async def chat_message(sid, data):
                    print("message ", data)
                    await sio.emit("chat_message", data)

                @sio.event
                def disconnect(sid):
                    print('disconnect ', sid)

                app.router.add_static('/static', 'static')
                app.router.add_get('/', index)

                if __name__ == '__main__':
                    web.run_app(app)
            • deployment creation
              aiohttp from aiohttp import web
              import socketio

              sio = socketio.AsyncServer()
              app = web.Application()
              sio.attach(app)

              async def index(request):
                  """Serve the client-side application."""
                  with open('index.html') as f:
                      return web.Response(text=f.read(), content_type='text/html')

              @sio.event
              ...

              app.router.add_static('/static', 'static')
              app.router.add_get('/', index)

              if __name__ == '__main__':
                  web.run_app(app)


              WSGI import socketio

              sio = socketio.Server()
              app = socketio.WSGIApp(sio)

              ASGI import socketio

              sio = socketio.AsyncServer(async_mode="asgi")
              app = socketio.ASGIApp(sio)
              # not needed when using Daphne
              # uvicorn.run(app, host='127.0.0.1', port=5000)


            • ...
          • ...

          • bash
          • JavaScript (browser, Node.js, React Native)
            • socket.io-client
              • <!DOCTYPE html>
                <html>
                  <head>
                  </head>
                  <body>
                    <script src="https://cdn.socket.io/4.5.4/socket.io.min.js"></script>
                    <script>
                      import {io} from "socket.io-client";

                      const socket = io("ws://localhost:3000")

                      // receive a message from server
                      socket.on("hello", (arg) => {
                      console.log(arg); // prints "world"
                      });

                      // send a message to the server
                      socket.emit("howdy", "stranger");
                    </script>
                  </body>
                </html>
              • import { io } from "socket.io-client";

                const socket = io("https://example.com", {
                  path: "/my-custom-path/"
                });
              • ...
                  <body>
                    <ulid="messages"></ul>
                    <form id="form" action="">
                      <input id="input" autocomplete="off" /><button>Send</button>
                    </form>
                   
                    <script src="https://cdn.socket.io/4.5.4/socket.io.min.js"></script>
                    <script>
                      var socket = io();

                      var messages = document.getElementById('messages');
                      var form = document.getElementById('form');
                      var input = document.getElementById('input');

                      form.addEventListener('submit', function(e) {
                          e.preventDefault();
                          if (input.value) {
                              socket.emit('chat message', input.value);
                              input.value = '';
                          }
                      });

                      socket.on('chat message', function(msg) {
                          var item = document.createElement('li');
                          item.textContent = msg;
                          messages.appendChild(item);
                          window.scrollTo(0, document.body.scrollHeight);
                      });
                    </script>   
                  </body>
                </html>
              • ...
          • Python
            • python-socketio
              • import asyncio
                import socketio

                sio = socketio.AsyncClient()

                @sio.event
                async def connect():
                    print("Received event: connect")
                    print("my sid is", sio.sid)

                @sio.event
                def connect_error(data):
                    print("The connection failed!")

                @sio.on('*')
                async def catch_all(event, data):
                   print(f"Received event {event} with data: {data}")

                @sio.event
                async def my_message(data):
                    print("Received event: my_message, with data: ", data)
                    await sio.emit('my response', {'response': 'my response'})

                @sio.on("chat message")
                async def on_missatge(data):
                    print("Received event: chat message, with data: ", data)

                @sio.event
                async def disconnect():
                    print("Received event: disconnect")

                async def main():
                    await sio.connect('http://localhost:3000')
                    await sio.wait()

                if __name__ == '__main__':
                    asyncio.run(main())
            • ...
          • ...
  • Media types (ftp)
  • Byte articles
  • Cache
  • vCard and vCalendar
  • MIME-Types
  • Imatges / Images

Proxy

  • Info
    • Proxy vs reverse proxy
      • [direct] proxy: una petició iniciada des d'un client en una subxarxa privada cap a Internet pot passar per un proxy, que va a buscar la resposta a Internet i li retorna cap al client
        • exemples: Squid, ...
      • reverse proxy: una petició iniciada des d'un client a Internet cap a un servidor, passa per un proxy invers, que va a buscar la resposta en un altre servidor (normalment en una subxarxa interna) i li retorna la resposta cap al client a Internet
        • tasques: balanceig de càrrega, autenticació, desxifratge, cache
        • exemples: nginx, ...
  • General
  • Articles
  • Programari / Software
  • Server
  • Client
    • configuració / setup
      • bash environment variable
        • export http_proxy=http://server-ip:port/
        • export http_proxy=http://USERNAME:PASSWORD@server-ip:port/
      • GUI
        • Centre de control de Mageia
          • Xarxa i internet
            • Servidor intermediari
    • ús / usage
      • CLI
        • wget http://toto.org/page.html
        • wget --proxy-user=USERNAME --proxy-password=PASSWORD http://toto.org/page.html
        • curl http://toto.org/page.html
        • curl --proxy-user user:password http://toto.org/page.html
      • aplicacions / applications
        • Firefox
          • Preferències -> Paràmetres de xarxa
        • Chrome
          • Configuració -> Sistema -> Obre la configuració del servidor intermediari de l'ordinador
  • Discover

Servidors HTTP / HTTP servers

  • Comparison of web servers (wp)
  • Comparison of lightweight web servers (wp)
  • Best Practices for Speeding Up Your Web Site (Steve Sounders)
  • Servidors HTTPS / HTTPS server
  • Balanceig de càrrega / Load balancing
  • Cache on servers
  • Servidors de prova / Test servers
  • Nginx (nginx.com) (wp)
    • Compilació / Compilation
      • Init scripts
      • show compilation options
        • nginx -V
      • check config
        • nginx -t
    • Instal·lació / Installation
      • Mageia
        • urpmi nginx
        • sytemctl enable nginx
        • sytemctl start nginx
      • Ubuntu
      • CentOS
        • How To Install Nginx on CentOS 7
        • LEMP server on CentOS 7 with FastCGI
        • sudo yum install epel-release
        • sudo yum install nginx
        • Official nginx repositories for CentOS
          • /etc/yum.repos.d/nginx.repo
        • sudo systemctl start nginx.service
        • Firewall (only needed if firewalld.service is running)
          • sudo firewall-cmd --permanent --zone=public --add-service=http
            sudo firewall-cmd --permanent --zone=public --add-service=https
            sudo firewall-cmd --reload
        • SELinux
          • SELinux on HTTP servers
          • sudo setsebool -P httpd_read_user_content 1
          • if only using nginx (not uwsgi)
          • if using also uwsgi (unix socket permission denied)
            • SELinux policy for nginx and GitLab unix socket in Fedora 19
            • disable Selinux:
              • setenforce 0
            • detect the problem
              • access your website, to generate logs
              • sudo yum install policycoreutils-python
              • sudo grep nginx /var/log/audit/audit.log | audit2allow
            • solve the problem (do not use -M option, as you will not be able to edit the te text file to add rules)
              • sudo -i
              • grep nginx /var/log/audit/audit.log | audit2allow -m nginx > nginx.te
              • checkmodule -M -m -o nginx.mod nginx.te
              • semodule_package -o nginx.pp -m nginx.mod
              • semodule -i nginx.pp
              • setenforce 1
            • example
              • nginx.te
                • module nginx 1.0;

                  require {
                          type httpd_t;
                          type init_t;
                          type nfs_t;
                          type user_home_t;
                          type var_lib_t;
                          type unlabeled_t;
                          class unix_stream_socket connectto;
                          class file {read getattr open};
                          class lnk_file read;
                          class sock_file write;
                  }

                  #============= httpd_t ==============
                  allow httpd_t init_t:unix_stream_socket connectto;
                  allow httpd_t nfs_t:file {read getattr open};
                  allow httpd_t nfs_t:lnk_file read;
                  allow httpd_t user_home_t:file {read open};
                  allow httpd_t var_lib_t:sock_file write;
                  allow httpd_t unlabeled_t:lnk_file read;
                  allow httpd_t unlabeled_t:file {read open};
          • Problems / Problemes
      • Django wsgi
    • Documentation (.org)
    • Service setup
      • NGINX systemd service file
      • nginx.service
        • # https://www.nginx.com/resources/wiki/start/topics/examples/systemd/

          [Unit]
          Description=The NGINX HTTP and reverse proxy server
          After=syslog.target network.target remote-fs.target nss-lookup.target cloud-init.service

          [Service]
          Type=forking
          PIDFile=/run/nginx.pid
          TimeoutStartSec=200s
          ExecStartPre=/usr/sbin/nginx -t
          PrivateTmp=true

          # other ExecStartPre scripts
          ExecStartPre=/bin/bash -c 'for script in /etc/nginx/execstartpre.d/*; do $script; done'

          ExecStart=/usr/sbin/nginx
          ExecReload=/bin/kill -s HUP $MAINPID
          ExecStop=/bin/kill -s QUIT $MAINPID

          [Install]
          WantedBy=multi-user.target
      • /tmp
        • /usr/lib/systemd/system/nginx.service
        • real /tmp dir for nginx is:
          • /tmp/systemd-private-...-nginx.service-.../tmp
    • Configuració / Configuration
      • How To Optimize Nginx Configuration
        • Timeouts
      • Modificació del fitxer de configuració / Modification of config file
      • Variables
      • Modular




        • worker_processes number | auto;


          events { ... } worker_connections number;

          # ngx_http_core_module
          http { ... }
          include file | mask; include mime.types;
          default_type application/octet-stream;
          sendfile on;
          keepalive_timeout  65;
          gzip on;



          # ngx_http_upstream_module
          upstream name { ... }
          # round-robin
          server address [parameters];

          server 127.0.0.1:8001;
          server 127.0.0.1:8002;


          # other server{} sections in files:
          include /etc/nginx/conf.d/*.conf;



          server { ... } listen address[:port] ...;
          listen port ...;
          listen unix:path ...;

          server_name name ...; #
          see: regex
          listen 80;
          server_name localhost;
          access_log
          /var/log/nginx/proxy.access.log;
          error_log /var/log/nginx/proxy.error.log;



          # ngx_http_ssl_module
          ssl_certificate file;
          ...

          ssl_certificate     /etc/letsencrypt/live/www.example.org/fullchain.pem;
          ssl_certificate_key /etc/letsencrypt/live/www.example.org/privkey.pem;
          ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
          ssl_ciphers         HIGH:!aNULL:!MD5;



          # nginx-vod-module: vod settings
          vod_mode ...;
          vod_upstream_location ...;
          vod_fallback_upstream_location ...;
          vod_last_modified ...;
          vod_last_modified_types ...;

          # nginx-vod-module: vod caches
          ...

          # nginx-vod-module: file handle caching / aio
          ...



          # other location{}
          include ...*.conf;



          location [ = | ~ | ~* | ^~ ] uri { ... }
          • = exact matching
          • ~ case-sensitive matching regex
          • ~* case-insensitive matching regex
          • ^~ ...
          location @name { ... }

          location / { ... }
          root ...;

          root html;



          alias ...;



          # ngx_http_proxy_module
          proxy_pass ...;
          proxy_read_timeout ...;
          proxy_send_timeout ...;
          proxy_http_version ...;
          proxy_set_header ...;
          proxy_redirect ...;



          # ngx_http_fastcgi_module
          fastcgi_pass ...;
          fastcgi_param ...;



          # ngx_http_uwsgi_module
          uwsgi_pass ...;


          location = /50x.html {} root html;
          # nginx-rtmp-module
          rtmp{}
          include /etc/nginx/rtmp.conf.d/*.conf;


          server{} listen 1935;


          application ... {} live on;
          wait_video on;
          wait_key on;
        • Regex (PCRE - Perl regular expressions)


          • examples
            server Wildcard names
            • server_name .example.org;
              • matches: example.org, *.example.org
            asterisk only at the begin or end:
            • server_name *.example.org;
            • server_name my.example.*;
            asterisk not at the begin nor end is not allowed:
            • www.*.example.org (use regex instead: ~^www\..+\.example\.org$)
            • w*.example.org (use regex instead: ~^w.*\.example\.org$)
            Regular expressions names
            (start with ~)
            • server_name  ~^www\d+\.example\.net$;
            • a regular expression containing the characters “{” and “}” should be quoted:
              • server_name  "~^(?<name>\w\d{1,3}+)\.example\.net$";
            • digital captures
              • usage:
                • server {
                      server_name   ~^(www\.)?(.+)$;

                      location / {
                          root   /sites/$2;
                      }
                  }
            • named regular expression
              • syntax:
                • ?<name> Perl 5.10 compatible syntax, supported since PCRE-7.0
                • ?'name' Perl 5.10 compatible syntax, supported since PCRE-7.0
                • ?P<name> Python compatible syntax, supported since PCRE-4.0
              • usage:
                • server {
                      server_name   ~^(www\.)?(?<domain>.+)$;

                      location / {
                          root   /sites/$domain;
                      }
                  }
            • ...
            location Regular expressions
            • (empty) starting with
            • = exact matching
            • ~ case-sensitive matching regex
            • ~* case-insensitive matching regex
            • ^~ ...
            • location = / {
                  [ configuration A ]
              }
            • location / {
                  [ configuration B ]
              }
            • location /documents/ {
                  [ configuration C ]
              }
            • location ^~ /images/ {
                  [ configuration D ]
              }
            • location ~* \.(gif|jpg|jpeg)$ {
                  [ configuration E ]
              }
          • ...
        • /etc/nginx/
          • nginx.conf
            • user  nginx;
              worker_processes  1;

              http {
                  include       /etc/nginx/mime.types;
                  default_type  application/octet-stream;
                  # Load modular configuration files from the /etc/nginx/conf.d directory.
                  # See http://nginx.org/en/docs/ngx_core_module.html#include
                  # for more information.
                  include /etc/nginx/conf.d/*.conf;

              }


              # nginx-rtmp-module

              rtmp {
                  # Load modular configuration files from the /etc/nginx/rtmp.conf.d directory.
                  include /etc/nginx/rtmp.conf.d/*.conf;
              }
          • execstartpre.d/
            • script1_execstartpre.sh
            • ...
          • conf.d/
            • http_server_1.conf
              • server {
                    listen       80;
                    server_name  localhost;

                    # Load modular configuration files from the /etc/nginx/conf.d/server_1.d/ directory.
                    include /etc/nginx/conf.d/http_server_1.d/*.http_server.conf;
                }
            • http_server_1.d/
              • dir_1_a.http_server.conf
                • location /dir_1_a {
                      alias /path/to/dir_1_a;
                  }
              • rtmp_server_1.http_server.conf (if using nginx-rtmp-module statistics and control)
                • # rtmp statistics location /stat {
                      rtmp_stat all;
                      rtmp_stat_stylesheet stat.xsl;
                  }

                  location /stat.xsl {
                      root /path/to/parent/dir/of/stat/xsl/file;
                  }

                  # rtmp control
                  location /control {
                      rtmp_control all;
                  }
            • https_server_2.conf
              • server {
                    # https
                    listen              443 ssl;
                    server_name         www.example.org;

                    ssl_certificate     /etc/letsencrypt/live/www.example.org/fullchain.pem;
                    ssl_certificate_key /etc/letsencrypt/live/www.example.org/privkey.pem;
                    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
                    ssl_ciphers         HIGH:!aNULL:!MD5;
                    ...
                }
            • https_server_2.d/
              • ...
          • rtmp.conf.d
            • rtmp_server_1.conf
              • server {
                    listen 1935;
                    ping 30s;
                    notify_method get;

                    # compatibility with GStreamer:
                    publish_time_fix off;

                    # Load modular configuration files from the /etc/nginx/rtmp.conf.d/wct-rtmp/ directory.
                    include /etc/nginx/rtmp.conf.d/rtmp_server_1.d/*.rtmp_server.conf;
                }
            • rtmp_server_1.d/
              • app_1_a.rtmp_server.conf
                • application app_1_a {
                      live on;
                      wait_video on;
                      wait_key on;
                  }
            • rtmp_server_2.conf
              • ...
            • rtmp_server_2.d/
              • ...
      • HTTPS
    • Autenticació / Authentication
    • Debug
      • A debugging log (nginx.org)
        • Debugging log for selected clients
        • Logging to a cyclic memory buffer
      • Debugging NGINX (nginx.com)
      • when building
        • ./configure --with-debug ...
        • check it:
          • nginx -V 2>&1 | grep -- '--with-debug'
      • nginx.conf
        • error_log /path/to/log debug;
        • error_log /var/log/nginx/error.log debug;
      • level possible values: debug, info, notice, warn, error, crit, alert, emerg
      • tail -n 200 -f /var/log/nginx/error.log
    • Logs
      • Logging and monitoring
      • Error log
      • Access log
        • log_format
          • default nginx.conf
            • http {
              ...
                  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                                    '$status $body_bytes_sent "$http_referer" '
                                    '"$http_user_agent" "$http_x_forwarded_for"';
              ...
              }
          • add request_time:
            • http {
              ...
                  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                                    '$status $body_bytes_sent "$http_referer" '
                                    '"$http_user_agent" "$http_x_forwarded_for" $request_time';
              ...
              }
        • access_log
      • access.log columns when using awk BEGIN {FPAT="([^ ]+)|(\"[^\"]+\")"}; :
        • $0 all line
          $1 IP address
          $2 -
          $3 -
          $4 datetime
          $5 timezone
          $6 request
          $7 response code
          $8 bytes
          $9 referer
          $10 user_agent
          $11
          x_forwarded_for (real origin ip address)
          $12
          (request_time)
      • count the most requested urls:
        • awk 'BEGIN {FPAT="([^ ]+)|(\"[^\"]+\")"}; {print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn
      • count the most requested urls inside a specified time period:
        • awk 'BEGIN {FPAT="([^ ]+)|(\"[^\"]+\")"}; $4 ~ /14\/Dec\/2021:13:5/ {print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -n
      • get requests with code 4xx and print time, code, request:
        • awk 'BEGIN {FPAT="([^ ]+)|(\"[^\"]+\")"}; $4 ~ /17\/Dec\/2021/ && $7 ~ /^4/ {print $4,$7,$6}' /var/log/nginx/access.log
    • Monitoratge / Monitoring
      • ngxtop
        • Instal·lació / Installation
          • pip install ngxtop
        • Ús / Usage
          • only last entries (live)
            • ngxtop
          • all log content
            • ngxtop --no-follow -i 'status >= 500' print request status http_referer
    • Headers
      • your.conf
        •     location /toto {
                  add_header x-peticio $request;
                  add_header x-temps $request_time;
                  add_header x-longitud $request_length;
                  return 200 'tot bé\n';
              }
    • Reverse proxying to:
      • Django deployment
      • Nginx: the High-Performance Web Server and Reverse Proxy
      • nginx config backend server
        gateway interface apps

        memcached



        # round robin
        upstream python_servers {
          server 127.0.0.1:8001;
          server 127.0.0.1:8002;
        }
        server {
          listen   127.0.0.1:8000;
          server_name  proxy;
          access_log  /var/log/nginx/proxy.access.log;
          error_log /var/log/nginx/proxy.error.log;
          location / {
            proxy_pass http://python_servers;
          }
        }
        ngx_http_proxy_module

        • server.py
          • import sys,BaseHTTPServer as B
            class Handler(B.BaseHTTPRequestHandler):
              def do_GET(self):
                self.wfile.write("Served from port %s" % port)
              def log_message(self, *args):
                pass
            if __name__ == '__main__':
              host,port = sys.argv[1:3]
              server = B.HTTPServer((host,int(port)), Handler)
              server.serve_forever()
        • # python /tmp/server.py 127.0.0.1 8001 &
          # python /tmp/server.py 127.0.0.1 8002 &


        location /mywebsocket {
            proxy_read_timeout 1d;
            proxy_send_timeout 1d;
            proxy_pass http://localhost:8123;
            #proxy_pass http://unix:/tmp/unixsocket_websocat;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }

        websocket
        • websocat -s 8123
        • socket.io
          • node index.js
          • python server.py
            • sio = socketio.Server()
            • sio = socketio.AsyncServer()
        • ...



        SCGI



        upload_pass ... upload-module




        location ~ \.php$ {
            include /etc/nginx/fcgi_params; #or whatever you named it
            fastcgi_pass  127.0.0.1:9000;
        }
        FastCGI
        PHP
        • php-cgi -b 127.0.0.1:9000
        CGI
        server {
            ...
            location /myscript {
                fastcgi_pass unix:/var/run/fcgiwrap.socket;

                # Fastcgi parameters, include the standard ones
                include /etc/nginx/fastcgi_params;

                fastcgi_param SCRIPT_FILENAME /usr/local/bin/myscript.sh;
            }
        }
        FCGIWrap
        • systemctl start spawn-fcgi


        # the upstream component nginx needs to connect to
        upstream django_mysite {
            server unix:///var/lib/uwsgi/mysite.sock;
        }

        # configuration of the server
        server {
            ...
            # Finally, send all non-media requests to the Django server.
            location / {
                uwsgi_pass django_mysite;
                include /etc/uwsgi/uwsgi_params
            }
        }
        UWSGI
        • bin/uwsgi
        WSGI
        • Django
        • Socket.IO WSGI
          • python wsgi_server.py
            • import socketio

              sio = socketio.Server()
              app = socketio.WSGIApp(sio)


        Gunicorn

        upstream channels-backend {
             # use 127.0.0.1 instead of localhost to avoid errors related to ipv6
            server 127.0.0.1:9000;
        }

        server {
            ...
            location / {
                try_files $uri @proxy_to_app;
            }
          
            location @proxy_to_app {
                proxy_pass http://channels-backend;

                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";

                proxy_redirect off;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Host $server_name;
            }
        }

        Daphne
        • daphne -p 9000 myproject.asgi:application
        • daphne -b 0.0.0.0 -p 9000 myproject.asgi:application
        • daphne -e ssl:443:privateKey=key.pem:certKey=crt.pem my_project.asgi:application
        • systemctl start my_django_project.daphne.service
          • [Unit]
            Description=Daphne daemon (my_project)
            After=network.target

            [Service]
            PIDFile=/run/daphne/pid
            User=root
            Group=root
            WorkingDirectory=/path/to/my_project
            ExecStart=/path/to/my_project/env/bin/daphne --bind 0.0.0.0 --port 9000 --verbosity 0 my_project.asgi:application
            ExecReload=/bin/kill -s HUP $MAINPID
            ExecStop=/bin/kill -s TERM $MAINPID
            Restart=on-abort
            PrivateTmp=true
            LimitNOFILE=32768

            [Install]
            WantedBy=multi-user.target
        • daphne -p 3000 asgi_server:app
        • daphne -b 0.0.0.0 -p 3000 asgi_server:app
        • systemctl start asgi_server.daphne.service
          • ...
        ASGI
        • Django
          • my_project/my_project/asgi.py
            • import os
              import django
              from channels.routing import get_default_application

              os.environ.setdefault("DJANGO_SETTINGS_MODULE", "my_project.settings")
              django.setup()
              application = get_default_application()
        • Socket.IO ASGI
          • python asgi_server.py
            • import socketio

              sio = socketio.AsyncServer()
              app = socketio.ASGIApp(sio)

        Uvicorn

      • AWS CloudFront
      • FastCGI (CGI)
        • fastcgi module (ngx_http_fastcgi_module)
        • Understanding and Implementing FastCGI Proxying in Nginx
        • FastCGI example
        • fastcgi_params Versus fastcgi.conf – Nginx Config History
        • nginx - How to run a shell script on every request?
        • FCGIWrap (github)
          • Documentation
          • Installing FcgiWrap and Enabling Perl, Ruby and Bash Dynamic Languages on Gentoo LEMP
          • Installation
            • Ubuntu
              • sudo apt-get install fcgiwrap
              • (no need to install spawn-fcgi, as fcgiwrap already comes with a service)
              • sudo systemctl status fcgiwrap.service
            • Mageia
              • from repo
                • ...
              • from source
                • install dependencies
                  • sudo dnf install libfcgi-devel
                • download and compile
                  • git clone http://github.com/gnosek/fcgiwrap.git
                  • cd fcgiwrap
                  • autoreconf -i
                  • ./configure
                  • make
                  • sudo make install
                    • /usr/local/sbin/fcgiwrap
              • install and setup spawn-fcgi (needed?)
                • install
                  • sudo dnf install ...
                • setup
                  • ...
            • CentOS
              • from repo
                • sudo dnf install fcgiwrap
              • from source
              • install and setup spawn fcgi
                • install
                • setup
                  • /etc/sysconfig/spawn-fcgi
                    • # You must set some working options before the "spawn-fcgi" service will work.
                      # If SOCKET points to a file, then this file is cleaned up by the init script.
                      #
                      # See spawn-fcgi(1) for all possible options.
                      #
                      # Example :
                      #SOCKET=/var/run/php-fcgi.sock
                      #OPTIONS="-u apache -g apache -s $SOCKET -S -M 0600 -C 32 -F 1 -P /var/run/spawn-fcgi.pid -- /usr/bin/php-cgi"

                      FCGI_SOCKET=/var/run/fcgiwrap.socket
                      # if fcgiwrap was compiled from source
                      #FCGI_PROGRAM=/usr/local/sbin/fcgiwrap
                      # if fcgiwrap was installed from repo

                      FCGI_PROGRAM=
                      /usr/sbin/fcgiwrap
                      FCGI_USER=nginx
                      FCGI_GROUP=nginx
                      FCGI_EXTRA_OPTIONS="-M 0770"
                      # IMPORTANT: -F 1 means only one worker!
                      OPTIONS="-u $FCGI_USER -g $FCGI_GROUP -s $FCGI_SOCKET -S $FCGI_EXTRA_OPTIONS -F 1 -P /var/run/spawn-fcgi.pid -- $FCGI_PROGRAM"
                    • to redirect stderr logs to /var/log/nginx/error.log (they will be shown as "FastCGI sent in stderr:")
                      • ...
                        OPTIONS="-u $FCGI_USER -g $FCGI_GROUP -s $FCGI_SOCKET -S $FCGI_EXTRA_OPTIONS -F 1 -P /var/run/spawn-fcgi.pid -- $FCGI_PROGRAM -f"
                • start
                  • systemctl enable spawn-fcgi
                  • systemctl start spawn-fcgi
                  • systemctl status spawn-fcgi
                  • ...
          • Problemes / Problems
          • Security issues
          • Example to execute a script specified in the url
            • setup
              • /etc/nginx/default.d/toto.conf
                • location /cgi-bin/ {
                       # Disable gzip (it makes scripts feel slower since they have to complete
                       # before getting gzipped)
                       gzip off;

                       # Set the root to /usr/lib (inside this location this means that we are
                       # giving access to the files under /usr/lib/cgi-bin)
                       root  /usr/share/nginx;

                       # Fastcgi socket
                       fastcgi_pass  unix:/var/run/fcgiwrap.socket;

                       # Fastcgi parameters: include the standard ones
                       include /etc/nginx/fastcgi_params;
                       #
                  Fastcgi parameters: adjust non standard parameters (e.g. SCRIPT_FILENAME)
                       fastcgi_param SCRIPT_FILENAME  $document_root$fastcgi_script_name;
                       #
                  Fastcgi parameters: user-defined parameters
                      
                  fastcgi_param TOTO_PARAM $request_length;
                  }
              • sudo systemctl reload nginx.service
            • scripts
              • mkdir -p /usr/share/nginx/cgi-bin
              • perl script
                • /usr/share/nginx/cgi-bin/hello_world.cgi
                  • #!/usr/bin/perl -w
                         # Tell perl to send a html header.
                         # So your browser gets the output
                         # rather then <stdout>(command line
                         # on the server.)
                    print "Content-type: text/html\n\n";
                         # print your basic html tags.
                         # and the content of them.
                    print "<html><head><title>Hello World!! </title></head>\n";
                    print "<body><h1>Hello world</h1></body></html>\n";
                • chmod 755 /usr/share/nginx/cgi-bin/hello_world.cgi
              • bash script
                • /usr/share/nginx/cgi-bin/toto.sh
                  • #!/bin/bash

                    # as PATH is set in /etc/init.d/functions, you may need to complete it
                    # when you are calling other scripts (e.g. in /usr/local/bin) from here
                    PATH=${PATH}:/usr/local/bin
                    export $PATH

                    # call a script in /usr/local/bin
                    my_script.sh

                    # content type must be present, followed by an empty line

                    # all lines must be ended with \r\n (CR LF)
                    echo -e "Content-type: text/plain\r"
                    echo -e "\r"

                    # your text
                    echo "123"

                    echo "server software: $SERVER_SOFTWARE"
                    echo "script filename: $SCRIPT_FILENAME"

                    echo  "toto param: $TOTO_PARAM"
                    # print all available environment variables
                    env
                • chmod 755 /usr/share/nginx/cgi-bin/toto.sh
                • Debug
                  • /usr/share/nginx/cgi-bin/toto_debug.sh
                    • #!/bin/bash -x
                      # adding -x will make bash xtrace output (by default written to stderr) to appear in /var/log/nginx/error.log,
                      # if option -f was specified in spawn config

                      echo -e "Content-type: text/plain\n"

                      # log to stderr instead of response body
                      (1>&2 echo "[hello]")


                      ...
                • Parse GET and POST parameters (using cgi_functions.sh)
                  • See also CGI bash examples with POST
                  • /usr/share/nginx/cgi-bin/toto_with_params
                    • #!/bin/bash
                      # if cgi_functions.sh is installed in /usr/local/bin

                      PATH=/usr/local/bin:${PATH}
                      export PATH=$PATH

                      # register all GET and POST variables
                      source cgi_functions.sh
                      cgi_getvars BOTH ALL 1>&2


                      # content type must be present, followed by an empty line
                      # all lines must be ended with \r\n (CR LF)
                      echo -e "Content-type: text/plain\r"
                      echo -e "\r"

                      # variables from QUERY and POST are available as shell variables
                      echo "var1: $var1"
                      echo "var2: $var2"
                      echo "var3: $var3"
                  • GET
                    • curl -X GET http://127.0.0.1/cgi-bin/toto_with_params?var1=value1
                  • POST
                    • application/x-www-form-urlencoded
                      • curl -X POST --data 'var3=value3' http://127.0.0.1/cgi-bin/toto_with_params
                    • multipart/form-data
                      • curl -X POST -F var2=value2 http://127.0.0.1/cgi-bin/toto_with_params?var1=value1
                      • curl -X POST -F var2=value2 http://127.0.0.1/cgi-bin/toto_with_params
                  • from local:
                    • REQUEST_METHOD=POST QUERY_STRING="var1=value1&var4=value4" var2=value2 var3=value3 /usr/share/nginx/cgi-bin/toto_with_params
                    • echo "var2=value_a&var2=value_b&var3=value3" | (REQUEST_METHOD=POST CONTENT_TYPE=application/x-www-form-urlencoded CONTENT_LENGTH=1000 ./multi_cgi.sh)
                • Allow only POST:
                  • /usr/share/nginx/cgi-bin/toto_only_post
                    • #!/bin/bash # register all GET and POST variables
                      source cgi_functions.sh
                      cgi_getvars BOTH ALL 1>&2

                      if [[ $REQUEST_METHOD != "POST" ]]
                      then
                          echo "Status: 405 Method Not Allowed"
                          echo -e "Content-type: text/plain\r"
                          echo -e "\r"

                          echo "Method not allowed"
                          exit 0
                      fi
                      # content type must be present, followed by an empty line # all lines must be ended with \r\n (CR LF)
                      echo -e "Content-type: text/plain\r"
                      echo -e "\r"

                      ...
            • Client
            • Problemes / Problems
              • SELinux
              • 403 Forbidden
                • chmod +x toto.sh
              • 502 Bad gateway
                • check that script returns as first lines (optionally a Status line, containing return code number and reason phrase, see: ngx_http_fastcgi_hide_headers) (note the empty line):
                  • Content-type: text/plain
                  • Status: 202 OK
                    Content-type: text/plain
          • Example to execute a fixed script when accessing a url:
            • nginx - How to run a shell script on every request?
            • ...conf with standard parameters
              • location /do_something/ {
                     # Disable gzip (it makes scripts feel slower since they have to complete
                     # before getting gzipped)
                     gzip off;
                     # Set the root to /usr/lib (inside this location this means that we are
                     # giving access to the files under /usr/lib/cgi-bin)
                     root  /usr/share/nginx;
                     # Fastcgi socket
                     fastcgi_pass  unix:/var/run/fcgiwrap.socket;
                     # Fastcgi parameters, include the standard ones
                     include /etc/nginx/fastcgi_params;
                     # Adjust non standard parameters (SCRIPT_FILENAME)
                     fastcgi_param SCRIPT_FILENAME /path/to/fixed_script.sh;
                     # user-defined parameters
                    
                fastcgi_param TOTO_PARAM $request_length;
                }
            • fixed_script.sh (see also simple_cgi.sh for an example with POST)
              • #!/bin/bash
                # content type must be present, followed by an empty line
                # all lines must be ended with \r\n (CR LF)
                echo -e "Content-type: text/plain\r"
                echo -e "\r"

                # execute some script
                result=$(/path/to/do_something_script.sh)
                echo "result:"
                echo $result

                # your text
                echo "request method:  $REQUEST_METHOD"

                echo "server software: $SERVER_SOFTWARE"
                echo "script filename: $SCRIPT_FILENAME"

                echo  "toto param: $TOTO_PARAM"
            • fixed_script_with_specified http response code
              • FastCGI and Nginx - Return HTTP Status
              • #!/bin/bash                                                                                                                                                                                                                       
                echo "Status: 404 Not Found"
                echo -e "Content-type: text/plain\r"
                echo -e "\r"

                echo "Uep, com anam?"
            • conf with parameters from url
              • location /fast {
                    # Disable gzip (it makes scripts feel slower since they have to complete
                    # before getting gzipped
                    gzip off;

                    # script will be under
                    root /usr/share/nginx;

                    # Fastcgi socket
                    fastcgi_pass  unix:/var/run/fcgiwrap.socket;
                    # Fastcgi parameters, include the standard ones
                    include /etc/nginx/fastcgi_params;

                    # script to be executed
                    fastcgi_param SCRIPT_FILENAME /usr/share/nginx/cgi-bin/fixed_script.sh;

                    # parameters for the script
                    fastcgi_param REQUEST_URI $request_uri;

                    # parameters from url
                    # /fast/<param1>/<param2>/
                    if ($request_uri
                ~* "/fast/([^/]*)/([^/]*)/$" ) {
                        set $param1 $1;
                        set $param2 $2;
                    }
                    fastcgi_param PARAM1 $param1;
                    fastcgi_param PARAM2 $param2;
                }
            • fixed_script with parameters
              • #!/bin/bash

                # content type must be present, followed by an empty line
                # all lines must be ended with \r\n (CR LF)
                echo -e "Content-type: text/plain\r"
                echo -e "\r"

                echo "REQUEST_URI: ${REQUEST_URI}"
                echo "PARAM1: ${PARAM1}"
                echo "PARAM2: ${PARAM2}"

                exit 0

          • Example to get upload time:
            • .../upload.conf
              • location /up {
                    # allow 100 Continue
                    client_max_body_size 20M;

                    # allow upload
                    sendfile on;

                    # Disable gzip (it makes scripts feel slower since they have to complete
                    # before getting gzipped)
                    gzip off;
                    # Set the root to /usr/lib (inside this location this means that we are
                    # giving access to the files under /usr/lib/cgi-bin)
                    root  /usr/share/nginx;
                    # Fastcgi socket
                    fastcgi_pass  unix:/var/run/fcgiwrap.socket;
                    # Fastcgi parameters, include the standard ones
                    include /etc/nginx/fastcgi_params;

                    # Adjust non standard parameters:
                    fastcgi_param SCRIPT_FILENAME /path/to/upload_time.sh;
                    # user-defined parameters:
                    fastcgi_param REQUEST_LENGTH $request_length;
                    fastcgi_param REQUEST_TIME $request_time;

                    # user-defined headers
                    add_header x-temps $request_time;
                    add_header x-longitud $request_length;
                }
            • upload_time.sh
              • #!/bin/bash
                # content type must be present, followed by an empty line

                # all lines must be ended with \r\n (CR LF)
                echo -e "Content-type: text/plain\r"
                echo -e "\r"

                # execute some script
                result=$(/path/to/do_something_script.sh)
                echo "result:"
                echo $result

                echo "request method:  $REQUEST_METHOD"
                echo "request length:  $REQUEST_LENGTH"
                echo "request time:    $REQUEST_TIME"
            • Client
              • dd if=/dev/zero of=megabyte.dat  bs=1M  count=1
              • curl -i --data-binary '@megabyte.dat' http://<server>/up
    • Cache
    • CORS
    • Upload
      • Test upload speed
      • Nginx direct file upload without passing them through backend
      • nginx upload client_max_body_size issue
        • Expect: 100-Continue
      • client
      • client_body_in_file_only
        • files uploaded to
          • /tmp/...
        • ...conf
          •  server {
                listen       80;
                server_name  localhost;

                error_log /var/log/nginx/error.debug.log debug;

                #sendfile on;

                location /up {
                  #auth_basic                 "Restricted Upload";
                  #auth_basic_user_file       basic.htpasswd;
                  #limit_except POST          { deny all; }

                  #client_body_temp_path      /tmp/;
                  #client_body_in_file_only   on;
                  #client_body_buffer_size    128K;
                  # allow 100 Continue
                  client_max_body_size 20M;

                  #proxy_pass_request_headers on;
                  #proxy_set_header           X-FILE $request_body_file;
                  #proxy_set_body             off;
                  #proxy_redirect             off;
                  #proxy_pass                 http://www.example.org/;

                    sendfile on;

                    add_header x-peticio $request;
                    add_header x-temps $request_time;
                    add_header x-longitud $request_length;

                    #root html;
                    #return 200 '$request_body_file';

                    return 200 'tot bé\n';
                }
        • Client (POST)
          • curl -i --data-binary '@myfile.png' http://192.168.1.29/upload
        • Problemes / Problems
      • nginx-upload-module
        • Upload
        • Resumable upload
        • nginx-upload-module 2.2 (github)
        • Nginx upload module (v 2.2.0)
          • nginx.service
            • [Service]
              ...
              # create hashed dirs for upload-module (level1=1)
              ExecStartPre=/bin/sh -c '/bin/mkdir -p /var/www/uploads/{0..9}; chmod 777 -R /var/www/uploads/{0..9}'
          • Example configuration
              •     # allow 100 Continue.
                    # if not specified here, as general value, defaults to 1M, and limits upload,
                    #   even if a greater vallue is specified inside a location section
                    #   (which only controls return of 100 Continue message)
                    client_max_body_size 20M;

                    location /upload {
                        # this value would response 100 for files under 30M, but general client_max_body_size 20M
                        # would not allow upload of files between 20 and 30M:
                        #
                client_max_body_size 30M;

                        # Pass altered request body to this location
                        upload_pass   @new_upload;

                        # Store files to this directory
                        # The directory is hashed:
                        # level1=1 (digit) : subdirectories 0 1 2 3 4 5 6 7 8 9 should exist
                        # (they can be created in nginx.service)
                        upload_store /var/www/uploads 1;
                       
                        # Allow uploaded files to be read only by user
                        upload_store_access user:r;

                        # Set specified fields in request body
                        # They are passed to backend as POST multipart/form-data
                        upload_set_form_field "${upload_field_name}_name" "$upload_file_name";
                        upload_set_form_field "${upload_field_name}_content_type" "$upload_content_type";
                        upload_set_form_field "${upload_field_name}_path "$upload_tmp_path";

                        # Inform backend about hash and size of a file

                        upload_aggregate_form_field "${upload_field_name}_md5" "$upload_file_md5";
                        upload_aggregate_form_field "${upload_field_name}_size" "$upload_file_size";

                        # pass fields from form
                        # pass all:
                        #upload_pass_form_field "(.*)";
                        # pass only submit and description:
                        upload_pass_form_field "^submit$|^description$";

                        upload_cleanup 400 404 499 500-505;
                    }

                    # Pass altered request body to a fastcgi backend
                    location @new_upload {
                        # user-defined headers
                        add_header x-request-time $request_time;
                        add_header x-request-length $request_length;

                        fastcgi_pass  unix:/var/run/fcgiwrap.socket;
                        # Fastcgi parameters, include the standard ones
                        include /etc/nginx/fastcgi_params;
                        # Adjust non standard parameters (SCRIPT_FILENAME)
                        fastcgi_param SCRIPT_FILENAME /usr/local/bin/new_upload.sh;

                        # user-defined parameters:
                        fastcgi_param REQUEST_LENGTH $request_length;
                        fastcgi_param REQUEST_TIME $request_time;
                    }
          • /usr/local/bin/new_upload.sh
            • #!/bin/bash
              echo -e "Content-type: text/plain\n"

              source cgi_functions.sh

              # register all GET and POST variables
              cgi_getvars BOTH ALL

              echo "New upload ==============================="
              export
        • Other examples
        • Compilation

        • client
          response code
          POST multipart/form-data curl -i -X POST -H 'Content-Type: multipart/form-data' -F toto='@myfile.png' http://<server>/up 200 OK
          POST x-www-form-urlencoded curl -i --data-binary '@myfile.png' http://<server>/up 415 Unsupported Media Type
          PUT
          curl --upload-file myfile.png http://<server>/up 405 Not Allowed
    • Resources (.com)
    • i18n
      • ...
    • 3rd party modules
    • Problemes / Problemes
      • check content of:
        • /var/log/nginx/error.log
        • /usr/local/nginx/logs/error.log
      • ...
      • related to fcgiwrap
        • connect() to unix:/var/run/fcgiwrap.socket failed (11: Resource temporarily unavailable) while connecting to upstream, client: ..., server: ..., request: ..., upstream: "fastcgi://unix:/var/run/fcgiwrap.socket:", host: ...
          • uwsgi with nginx (11: Resource temporarily unavailable) while connecting to upstream)
          • if, for some reason, a script called by fastcgi get stuck, next requests are not being served (netstats reports: CONNECTING), and they are queued.
          • This can be checked with: netstat --unix | grep fcgi | wc
          • is there a limit?
          • workaroud: systemctl restart spawn-fcgi.service
          • to get the process that is blocking fastcgi
          • debug:
            • netstat --unix
            • ss --unix
            • strace
            • lsof | grep fcgi
      • related to uWSGI (WSGI)
        • connect() to unix:///var/lib/uwsgi/....sock failed (11: Resource temporarily unavailable) while connecting to upstream
        • connection to page gives "502 Bad Gateway"
          • systemctl status emperor.uwsgi.service
            • import memcache
              ImportError: No module named memcache
        • connection to page gives "502 Bad Gateway"
          • /var/log/nginx/error.log
            • connect() to unix:///etc/uwsgi/wct_backend.sock failed (13: Permission denied) while connecting to upstream ...
          • If it doesn't work
          • Solució / Solution
            1. regenerate nginx.pp file
            2. sudo systemctl restart emperor.uwsgi.service
            3. sudo systemctl restart nginx.service
        • connection to page gives "502 Bad Gateway"
          • /var/log/nginx/error.log
            • connect() to unix:///etc/uwsgi/wct_backend.sock failed (2: No such file or directory) while connecting to upstream ...
          • Solució / Solution
            • check output from (maybe some misspelling?): 
              • sudo systemctl status emperor.uwsgi.service
            • check that uwsgi is installed in your virtualenv
            • check uwsgi configuration
      • related to Django and Daphne (ASGI)
        • connect() failed (111: Connection refused) while connecting to upstream, client: ..., server: ..., request: ..., upstream: "http://[::1]:9000/ws/...", host: ...
          • IPv6 not activated
          • Solució / Solution
            • conf.d/my_nginx.conf
              • upstream my-upstream {
                    # use 127.0.0.1 instead of localhost to avoid errors related to ipv6
                    server 127.0.0.1:9000;
                }
        • [alert] 2398#0: 1024 worker_connections are not enough
          • Optimal value for Nginx worker_connections
          • monitor the number of used worker_connections:
            • sudo ss -tupn | grep nginx | wc -l
            • (deprecated) sudo netstat -tupn | grep nginx | wc -l
          • Solució / Solution
            • nginx.conf
              • events {
                    # this value must not exceed:
                    # - max number of open files for nginx user and nginx.service:
                    #     sudo su - nginx -s /bin/bash -c "ulimit -Hn"
                    #     grep LimitNOFILE /usr/lib/systemd/system/nginx.service
                    # - total range of sockets to enable per IP: sysctl net.ipv4.ip_local_port_range
                    # default value: worker_connections  1024;
                    worker_connections  16384;
                }
            • set max number of open files
            • set total range of sockets to enable per IP
              • sysctl net.ipv4.ip_local_port_range
              • sysctl -w net.ipv4.ip_local_port_range ...
        • [error] 22590#0: *344 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: ..., server: ..., request: "GET ...", upstream: "http://127.0.0.1:9000/...", host: ...
        • [info] 22590#0: *355 peer closed connection in SSL handshake while SSL handshaking, client: ..., server: 0.0.0.0:443
          • ...
      • accept4() failed (24: Too many open files)
        • Nginx: 24: Too Many Open Files Error And Solution
        • nginx 24: Too many open files
        • see also: worker_connections are not enough
        • Solució / Solution:
          • increase limits for nginx user:
            • present values:
              • sudo su - nginx -s /bin/bash -c "ulimit -Hn"
              • sudo su - nginx -s /bin/bash -c "ulimit -Sn"
            • set new values at os level for user nginx:
              • /etc/security/limits.d/nginx.conf
                • # default: nginx       hard    nofile   4096
                  nginx       hard    nofile   16384
                  # default: nginx       soft    nofile   1024
                  nginx       soft    nofile   4096
            • set new values at service level:
              • /usr/lib/systemd/system/nginx.service
              • systemctl daemon-reload
              • systemctl restart nginx.service
            • check new values in running process:
              • nginx_pid=$(pgrep -f "nginx: master process")
              • cat /proc/${nginx_pid}/limits | grep -E "Limit|Max open files"
                • Limit                     Soft Limit           Hard Limit           Units    
                  Max open files            16384                16384                files  
          • perform stress tests
      • exec: fork failed (12: Cannot allocate memory), client: ..., server: ...
      • nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (13: Permission denied)
        • Solució / Solution
          • sudo setsebool -P httpd_read_user_content 1
      • nginx: [emerg] bind() to 0.0.0.0:8000 failed (13: Permission denied)
      • nginx: [emerg] bind() to 0.0.0.0:444 failed (13: Permission denied)
      • content is not cached
  • Apache HTTP Server (Apache Software Foundation)
  • Fake http server with netcat
  • Cherokee Web Server (wp)
  • Netscape Enterprise Server 
  • NCSA HTTPd
  • CERN httpd
  • Servidors lleugers / Light servers
  • Jetty Java HTTP Servlet Server
  • Web Servers Comparison
  • Creating Net Sites
  • Designing Your Web Site
  • Databases interfaces
  • Utilitats / Utilities
  • Gateway interface
    gateway interface backend server config


    nginx apache
    CGI (Common Gateway Interface) proxying to
    WSGI (Python Web Server Gateway Interface)
    • uwsgi
    • Gunicorn

    ASGI (Asynchronous Server Gateway Interface)
    • CGI (Common Gateway Interface)
      • CGI on Apache
      • FastCGI
      • RFC 3875 "The Common Gateway Interface (CGI) Version 1.1"
      • The Web Developer's Virtual Library course
      • CGIHTML (C routines)
      • CGI Developer's Guide
      • Bash
        • Bash.CGI
          • cgi_functions.sh
            • #!/bin/bash

              # https://oinkzwurgl.org/hacking/bash_cgi/
              # Phillippe Kehi <phkehi@gmx.net> and flipflip industries
              # to test this function locally:
              # source cgi_functions.sh
              # QUERY_STRING="var1=value1" cgi_getvars GET var1; echo $var1
              # QUERY_STRING="var1=value1&var2=value2" cgi_getvars GET ALL; echo $var1; echo $var2
              # CONTENT_TYPE="application/x-www-form-urlencoded" ... cgi_getvars POST ALL; echo $var1; echo $var2

              # to test this function locally:
              # source cgi_functions.sh
              # QUERY_STRING="var1=value1" cgi_getvars GET var1; echo $var1
              # QUERY_STRING="var1=value1&var2=value2" cgi_getvars GET ALL; echo $var1; echo $var2
              # CONTENT_TYPE="application/x-www-form-urlencoded" ... cgi_getvars POST ALL; echo $var1; echo $var2

              # (internal) routine to store POST data
              function cgi_get_POST_vars()
              {
                  # check content type
                  # FIXME: not sure if we could handle uploads with this..
                  if [ "${CONTENT_TYPE}" != "application/x-www-form-urlencoded" ] && ! [[ "${CONTENT_TYPE}" =~ ^"multipart/form-data" ]]
                  then
                      # if, in /etc/sysconfig/spawn-fcgi, OPTIONS contains -f,
                      # this echo can be seen at /var/log/nginx/error.log
                      echo "[`basename $0`] WARNING: received CONTENT_TYPE is ${CONTENT_TYPE}, but implemented content types for POST are only: "\
                           "application/x-www-form-urlencoded, multipart/form-data" 1>&2
                      return
                  fi

                  # save POST variables (only first time this is called)
                  [ -z "$QUERY_STRING_POST" \
                    -a "$REQUEST_METHOD" = "POST" -a ! -z "$CONTENT_LENGTH" ] && \
                      read -N $CONTENT_LENGTH RECEIVED_POST
                  #echo "RECEIVED_POST: $RECEIVED_POST"
                 
                  if [[ "${CONTENT_TYPE}" =~ ^"multipart/form-data" ]]
                  then
                      # export variables from multipart
                      boundary=$(echo $CONTENT_TYPE | awk -F= '{print$2}');
                      #QUERY_STRING_POST=$(echo "$RECEIVED_POST" | awk -v b=$boundary 'BEGIN {RS=b"\r\n";FS="\r\n";ORS="&"} $1 ~ /^Content-Disposition/ {gsub(/Content-Disposition: form-data; name=/,"",$1); gsub("\"","",$1); print $1"="$3}')
                      # variables must be exported here, because they can contain ampersands in value
                      eval $(echo "$RECEIVED_POST" | awk -v b=$boundary 'BEGIN {RS=b"\r\n";FS="\r\n";ORS=" "} $1 ~ /^Content-Disposition/ {gsub(/Content-Disposition: form-data; name=/,"",$1); gsub("\"","",$1); print "export "$1"=\""$3"\""}')
                  else
                      # take input string as is
                      QUERY_STRING_POST="$RECEIVED_POST"
                  fi
                 
                  return
              }


              # (internal) routine to decode urlencoded strings
              function cgi_decodevar()
              {
                  [ $# -ne 1 ] && return
                  local v t h
                  # replace all + with whitespace and append %%
                  t="${1//+/ }%%"
                  while [ ${#t} -gt 0 -a "${t}" != "%" ]; do
                      v="${v}${t%%\%*}" # digest up to the first %
                      t="${t#*%}"       # remove digested part
                      # decode if there is anything to decode and if not at end of string
                      if [ ${#t} -gt 0 -a "${t}" != "%" ]; then
                          h=${t:0:2} # save first two chars
                          t="${t:2}" # remove these
                          v="${v}"`echo -e \\\\x${h}` # convert hex to special char
                      fi
                  done
                  # return decoded string
                  echo "${v}"
                  return
              }

              # routine to get variables from http requests
              # usage: cgi_getvars method varname1 [.. varnameN]
              # method is either GET or POST or BOTH
              # the magic varible name ALL gets everything
              function cgi_getvars()
              {
                  [ $# -lt 2 ] && return
                  local q p k v s
                  # get query
                  case $1 in
                      GET)
                          [ ! -z "${QUERY_STRING}" ] && q="${QUERY_STRING}&"
                          ;;
                      POST)
                          cgi_get_POST_vars
                          [ ! -z "${QUERY_STRING_POST}" ] && q="${QUERY_STRING_POST}&"
                          ;;
                      BOTH)
                          [ ! -z "${QUERY_STRING}" ] && q="${QUERY_STRING}&"
                          cgi_get_POST_vars
                          [ ! -z "${QUERY_STRING_POST}" ] && q="${q}${QUERY_STRING_POST}&"
                          ;;
                  esac
                  shift
                  s=" $* "
                  # parse the query data
                  while [ ! -z "$q" ]; do
                      p="${q%%&*}"  # get first part of query string
                      k="${p%%=*}"  # get the key (variable name) from it
                      v="${p#*=}"   # get the value from it
                      q="${q#"$p"&*}" # strip first part from query string
                      # decode and assign variable if requested
                      [ -n "$p" ] && [ "$1" = "ALL" -o "${s/ $k /}" != "$s" ] && \
                          export "$k"="`cgi_decodevar \"$v\"`"
                  done
                  return
              }
        • CGI shell script using bash
        • Bash
          • Saving a binary file
        • How to parse $QUERY_STRING from a bash CGI script
        • Creating CGI Programs with Bash: Handling POST Data
          • curl -i -X POST --data 'tata=tata_value' http://127.0.0.1/test
        • Examples with POST
          • See also Parse GET and POST parameters
          • simple_cgi.sh
            • #!/bin/bash

              code="200 OK"
              echo -e "Status: ${code}\r"
              echo -e "Content-type: text/plain\r"
              echo -e '\r'

              echo -e "REQUEST_METHOD: ${REQUEST_METHOD}\r"
              echo -e "CONTENT_TYPE: ${CONTENT_TYPE}\r"
              echo -e "CONTENT_LENGTH: ${CONTENT_LENGTH}\r"
              echo -e "QUERY_STRING: ${QUERY_STRING}\r"

              exit 0
            • #!/bin/bash

              code="200 OK"
              echo -e "Status: ${code}\r"
              echo -e "Content-type: text/plain\r"
              echo -e 'Access-Control-Allow-Origin:*\r'
              echo -e 'Access-Control-Allow-Credentials: true\r'
              echo -e 'Access-Control-Allow-Methods: GET, POST, OPTIONS\r'
              echo -e 'Access-Control-Allow-Headers: DNT, X-CustomHeader, Keep-Alive, User-Agent, X-Requested-With, If-Modified-Since, Cache-Control, Content-Type, Authorization \r'
              echo -e '\r'

              echo -e "REQUEST_METHOD: ${REQUEST_METHOD}\r"
              echo -e "QUERY_STRING: ${QUERY_STRING}\r"
              echo -e "CONTENT_TYPE: ${CONTENT_TYPE}\r"
              echo -e "CONTENT_LENGTH: ${CONTENT_LENGTH}\r"

              exit 0
            • #!/bin/bash
              # when called with:
              #  curl -i "http://localhost/?a=b&c=d" --data "toto=tu" --data "toto=ta"
              # result is:
              #  REQUEST_METHOD: POST
              #  QUERY_STRING: a=b&c=d
              #  CONTENT_TYPE: application/x-www-form-urlencoded
              #  CONTENT_LENGTH: 15
              #  CONTENT: toto=tu&toto=ta

              code="200 OK"
              echo -e "Status: ${code}\r"
              echo -e "Content-type: text/plain\r"

              echo -e "REQUEST_METHOD: ${REQUEST_METHOD}\r"
              echo -e "QUERY_STRING: ${QUERY_STRING}\r"

              echo -e "CONTENT_TYPE: ${CONTENT_TYPE}\r"
              echo -e "CONTENT_LENGTH: ${CONTENT_LENGTH}\r"
              read -N ${CONTENT_LENGTH} CONTENT
              echo -e "CONTENT: ${CONTENT}\r"

              exit 0
            • #!/bin/bash -x
              echo -e "Content-type: text/plain\n"
              echo "content type: ${CONTENT_TYPE}"
              echo "QUERY_STRING_POST:"
              cat

            • #!/bin/bash -x
              echo -e "Content-type: text/plain\n"
              echo "content type: ${CONTENT_TYPE}"
              read -N $CONTENT_LENGTH QUERY_STRING_POST
              echo "QUERY_STRING_POST:"
              echo "$QUERY_STRING_POST"
          • parse_multi.sh
            • #!/bin/bash
              echo -e "Content-type: text/plain\n"

              echo "CONTENT_TYPE: $CONTENT_TYPE"
              echo "CONTENT_LENGTH: $CONTENT_LENGTH"

              # get boundary from CONTENT_TYPE
              boundary=$(echo $CONTENT_TYPE | awk -F= '{print$2}')
              echo "== boundary:"
              echo "$boundary"

              # read standard input
              read -N $CONTENT_LENGTH QUERY_STRING_POST
              echo "== received standard input:"
              echo "$QUERY_STRING_POST"

              # parse standard input in multipart/form-data
              post_params=$(echo "$QUERY_STRING_POST" | awk -v b=$boundary 'BEGIN {RS=b"\r\n";FS="\r\n";ORS="&"} $1 ~ /^Content-Disposition/ {gsub(/Content-Disposition: form-data; name=/,"",$1); gsub("\"","",$1); print $1"="$3}')
              echo "== post_params:"
              echo "$post_params"

              exit 0
      • Perl
      • Python
    • WSGI (Python Web Server Gateway Interface)
    • ASGI (Asynchronous Server Gateway Interface)
  • Servlets / JSP servers
  • C Server Pages (CSP)
  • MIME types configuration

Servidors FTP / FTP servers

  • Clients FTP
  • vsftpd
  • ProFTPD
    • Instal·lació / Installation
      • CentOS
        • sudo yum install proftpd
        • systemctl start proftpd
    • Mini-Howto
      • Debugging
        • /usr/lib/systemd/system/proftpd.service
          • ExecStart = /usr/sbin/proftpd -d10 $PROFTPD_OPTIONS
      • Configuring a directory
      • Virtual users
        • Passos / Steps
          • sudo -i
          • cd /etc; mkdir proftpd; cd proftpd
          • wget https://raw.githubusercontent.com/proftpd/proftpd/master/contrib/ftpasswd
          • chmod +x ftpasswd
          • ./ftpasswd --group -gid 9000 --name ftp_users
          • ./ftpasswd --passwd --name user1 --home /home --shell /bin/bash --uid 8000 --gid 9000
          • /etc/proftpd.conf
            • # users
              AuthUserFile /etc/proftpd/ftpd.passwd
              AuthGroupFile /etc/proftpd/ftpd.group
          • systemctl restart proftpd.service
        • Test
          • ftp://user1...
      • AWS and ProFTPD
        • security groups
          • tcp 21
          • tcp 1024 - 1048
        • proftpd.conf
          • # aws
            PassivePorts        1024 1048
            MasqueradeAddress   <your_public_ip_address>
    • /etc/proftp.conf
      • DefaultRoot ~
      • <Directory ~>
            <Limit ALL>
              DenyAll
            </Limit>
            <Limit DIRS READ>
              AllowAll
            </Limit>
        </Directory>
        <Directory ~/upload>
            <Limit WRITE>
              AllowAll
            </Limit>
        </Directory>

Clients FTP / FTP Clients

  • sftp
  • lftp
    • download
      • lftpget ftp://user@remote_host/dir1/dir2/dir3/file1
      • lftp ftp://user@remote_host/dir1 -e "get dir2/dir3/file1; bye"
      • mirror
        • lftp ftp://user@remote_host/dir1 -e "mirror; bye"
    • upload
      • reverse (-R) mirror
        • lftp -d -f commands.lftp 2>> toto.log
          • commands.lftp
            • open -u user,password remote_host
              set ftp:ssl-allow no
              mirror -R local_dir
      • Exclude .git dir:
        • lftp exclude syntax confusion
        • commands.lftp
          • open -u user,password remote_host
            set ftp:ssl-allow no
            mirror --exclude --exclude=^\.git/$ -R local_dir remote_dir
      • How to use rsync over ftp
  • curlftpfs (mount locally)
  • weex (updating web pages)
  • Python

PHP

  • Pàgines web dinàmiques / Dynamic web pages
    • Frameworks
  • PEAR packages
  • PHP CLI (command line)
    • PHP CLI
    • Using PHP from the command line
    • Instal·lació / Installation
      • Mageia
        • urpmi php-cli
    • Exemples / Examples: (-r to avoid <?php...?>)
      • php [-f] toto.php
      • php -r "mail('to_user@example.org','subjecte','cos del missatge','From:from_user@example.org' );"
  • IDE
  • PHP debug
    • php -i
    • /etc/php.ini (Mageia: urpmi php-ini) or /etc/php5/apache2/php.ini (Debian/Ubuntu)
      • display_errors = On
    • service httpd restart (or: sudo service apache2 restart)
    • comprova la sintaxi / check syntax
      • urpmi php-cli
      • php -l toto.php
    • /etc/php.ini or /etc/php5/apache2/php.ini
    • logs
      • error_log
      • Examples
        • error_log("You messed up!", 3, "/var/tmp/my-errors.log");
        • error_log(print_r($variable, TRUE));
        • error_log("You messed up!", 3, "php://stdout");
    • see also Apache logs
    • check the output
      • php -f toto.php
      • if result is empty and "echo $?" returns 255, it could mean that dependencies not satisfied
    • xdebug
      • Instal·lació / Installation
        • Mageia
          • urpmi php-xdebug
        • Ubuntu
          • sudo apt-get install php5-xdebug
      • Eclipse
        • Debugging using XDebug
        • Setup php ini file:
          • Mageia: /etc/php.d/A29_xdebug.ini
          • Ubuntu: /etc/php5/apache2/conf.d/20-xdebug.ini
          • xdebug.remote_enable = 1
            xdebug.remote_handler = 'dbgp'
        • An http server must be configured and running
        • Eclipse: Window / Preferences / PHP / Servers
          • Server
            • Base URL: http://localhost
            • Document Root: /var/www/html
          • Debugger: XDebug
            • Port: 9000
        • Problems
          • Progress tab: "waiting for xdebug session"
            • Solution
              • xdebug.remote_handler = 'dbgp'
      • kdevelop
    • Debugging remote CLI with phpstorm
  • Dependencies
    • inclued
      • Installation
        • Mageia
          • urpmi php-inclued
  • Errors
    • Reference - What does this error mean in PHP?
    • "Warning: cannot modify header information..."
      • Si a php.ini hi ha:
        • error_reporting = E_ALL & ~E_DEPRECATED
        • i es produeix un "Notice:...", es posarà al text de resposta, i el codi serà un 200, i ja no es podrà  canviar per un altre
      • En canvi, si a php.ini hi ha:
        • error_reporting = E_ALL & ~E_DEPRECATED & ~E_NOTICE
        • no s'escriurà cap "Notice:..." a la resposta, i el codi HTTP es podrà canviar mitjançant un "header("HTTP/1.0 204 No Response")"
      • Penseu també  a eliminar les línies en blanc o retorns de carro fora de <php>
  • i18n
  • Smarty
    • i18n
      • smarty-gettext
        • Smarty gettext plugin
        • Installation
          • cd <smarty>/plugins
          • wget https://raw.githubusercontent.com/smarty-gettext/smarty-gettext/master/block.t.php
          • wget https://raw.githubusercontent.com/smarty-gettext/smarty-gettext/master/function.locale.php
        • Setup
          • cd public
          • get script to extract translatable strings
            • wget https://raw.githubusercontent.com/smarty-gettext/smarty-gettext/master/tsmarty2c.php
            • chmod +x tsmarty2c.php
          • edit your master tpl file:
            • {locale path="Locale" domain="messages"}
          • edit your_view.tpl
            • {t}Hello world from Smarty!{/t}
          • extract all translatable strings to a pot file:
            • ./tsmarty2c.php -o smarty.pot <path_to>/views/
          • merge pot files from *.php and *.tpl
            • xgettext --from-code=UTF-8 -o php.pot *.php
            • msgcat -o messages.pot php.pot smarty.pot
          • create po files from pot
            • cp messages.pot Locale/ca/LC_MESSAGES/messages.po
          • cd Locale/ca/LC_MESSAGES
          • edit messages.po to update translations
          • compile po to mo:
            • msgfmt -o messages.mo messages.po
        • Next times:
          • regenerate pot
            • xgettext --from-code=UTF-8 -o php.pot *.php
            • ./tsmarty2c.php -o smarty.pot <path_to>/views/
            • msgcat -o messages.pot php.pot smarty.pot
          • update po
            • msgmerge -U Locale/ca/LC_MESSAGES/messages.po messages.pot
            • msgmerge -U Locale/fr/LC_MESSAGES/messages.po messages.pot
            • ...
          • recompile mo
            • msgfmt -o Locale/ca/LC_MESSAGES/messages.mo Locale/ca/LC_MESSAGES/messages.po
            • msgfmt -o Locale/fr/LC_MESSAGES/messages.mo Locale/fr/LC_MESSAGES/messages.po
            • ...
  • Correu electrònic / E-mail
  • curl-php
    • PHP Curl with port number issue
    • POST of array to djangorestframework, in multipart (PhpMultiPartParser)
      • <?php
        // http://stackoverflow.com/questions/3772096/posting-multidimensional-array-with-php-and-curl
        function http_build_query_for_curl( $arrays, &$new = array(), $prefix = null ) {

            if ( is_object( $arrays ) ) {
                $arrays = get_object_vars( $arrays );
            }

            foreach ( $arrays AS $key => $value ) {
                $k = isset( $prefix ) ? $prefix . '[' . $key . ']' : $key;
                if ( is_array( $value ) OR is_object( $value )  ) {
                    http_build_query_for_curl( $value, $new, $k );
                } else {
                    $new[$k] = $value;
                }
            }
        }

        $ch = curl_init();
        $url = "http://127.0.0.1:8000/mymodels/";
        $myvalues = array("first", "second");
        $post_fields = array("name" => "toto", "myvalues" => $myvalues );

        http_build_query_for_curl( $post_fields, $post );

        print_r($post);

        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: multipart/form-data'));
        curl_setopt($ch, CURLOPT_POST, sizeof($post_fields));
        curl_setopt($ch, CURLOPT_POSTFIELDS, $post );

        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

        $server_output = curl_exec($ch);

        $info    = curl_getinfo($ch);
        if(curl_errno($ch)) {   
                    echo "something was wrong in your request";
                    }
        curl_close($ch);
        print $server_output;

        ?>
  • Certificats / Certificates
  • Cross-Origin Resource Sharing (CORS)
    • Requests with credentials
    • Server-side Acess Control (PHP examples)
    • http://foo.example/ (Javascript)
      • var invocation = new XMLHttpRequest();
        var url = 'https://bar.other/'

        invocation.onreadystatechange = function() {
          if (invocation.readyState!=4) return;

          var is_ok = invocation.responseText=='OK';
          if (is_ok) {
            document.location.href = isok.html
          } else {
           
        document.location.href = isnotok.html
          }
        }

        invocation.open('GET', url, true );
        invocation.withCredentials = "true";
        invocation.send();
    • https://bar.other/ (PHP)
      • <?php
        header('Access-Control-Allow-Origin: http://
        foo.example/');
        header('Access-Control-Allow-Credentials: true');

        if ($_SERVER['SSL_CLIENT_VERIFY']=='SUCCESS') {
          echo 'OK';
        } else {
          $sslcn = $_SERVER['SSL_CLIENT_S_DN_CN'];   $msg = "$sslcn NOT_OK";
          echo
        $msg;
        }
        ?>
    • Nota: si no es fan servir Access-Control-Allow- (o bé "Access-Control-Allow-Origin: *"), no estarà disponible responseText a la URL origen. (Firefox: Cross-domain requests with credentials return empty)
    • "Access-Control-Allow-Origin: *" no pot estar lligat a withCredentials = "true"
  • HTML parsing
  • PHP3 (es) (ch)
  • Databases interaction
  • Emacs modes
    • Php Mode (EmacsWiki)
      1. cd /usr/share/emacs/24.2/lisp/progmodes/
      2. wget http://php-mode.svn.sourceforge.net/svnroot/php-mode/tags/php-mode-1.5.0/php-mode.el
      3. ~/.emacs
        • (autoload 'php-mode "php-mode" "Major mode for editing php code." t)
          (add-to-list 'auto-mode-alist '("\\.php$" . php-mode))
          (add-to-list 'auto-mode-alist '("\\.inc$" . php-mode))
  • LBD/EPFL
  • Manual (local)
  • Tutorial
  • PHP Nuke
  • Suhosin (Hardened-PHP project - PHP security)

Crawling

Recerca / Search

  • Search engine optimization (SEO) (wp)

Robots

Seguretat / Security

Servidors de correu / E-mail servers

Clients de correu electrònic / E-mail clients

  • Obrir un adjunt winmail.dat / Open a winmail.dat attachment:
    • Install tnef:
      • urpmi tnef
      • apt-get install tnef
    • Unpack attachement content:
      • tnef winmail.dat
  • Command line
    • How do I test an imap server?
    • Testing POP3 and IMAP servers from the command line in CMD or bash
    • imap:
      • telnet mail.example.com 143
      • ...
    • telnet/nc to an SMTP server (after EHLO, responses starting with 250 show available commands) (cannot continue a connection when STARTTLS has been invoked; use openssl s_client or mailx instead)
      • port 25 (default)
        • HELO (basic commands)
          • $ telnet mail.example.org 25
            Trying x.x.x.x...
            Connected to mail.example.org.
            Escape character is '^]'.
            220 client.example.org ESMTP Postfix
            HELO mail.example.org
            250
            client.example.org
            MAIL FROM:<localuser@example.org>
            250 2.1.0 Ok
            RCPT TO:<remoteuser@example.org>
            250 2.1.5 Ok
            DATA
            354 End data with <CR><LF>.<CR><LF>
            To:<remoteuser@example.org>
            From:<localuser@example.org>
            Subject:First test
            Hi, you!
            .
            250 2.0.0 Ok: queued as 489C411EE7AC
            QUIT
            221 2.0.0 Bye
            Connection closed by foreign host.
        • EHLO (enhanced commands)
          • $ telnet mail.example.org 25
            Trying x.x.x.x...
            Connected to mail.example.org.
            Escape character is '^]'.
            220 client.example.org ESMTP Postfix
            EHLO mail.example.org 250-client.example.org
            250-PIPELINING
            250-SIZE 10240000
            250-VRFY
            250-ETRN
            250-STARTTLS
            250-ENHANCEDSTATUSCODES
            250-8BITMIME
            250 DSN
            ...
      • port 587 (must be activated; see Postfix on port 587) and STARTTLS:
        • $ telnet mail.example.org 587 Trying x.x.x.x...
          Connected to mail.example.org.
          Escape character is '^]'.
          220 client.example.org ESMTP Postfix
          EHLO mail.example.org 250-client.example.org
          250-PIPELINING
          250-SIZE 10240000
          250-VRFY
          250-ETRN
          250-STARTTLS
          250-ENHANCEDSTATUSCODES
          250-8BITMIME
          250 DSN
          STARTTLS
          ...
    • openssl s_client
      • openssl s_client -connect mail.example.org:587 -starttls smtp
      • openssl s_client -starttls smtp -connect smtp.gmail.com:587 -crlf -ign_eof
      • Problemes / Problems
        • read:errno=0 (and connection is closed):
          • check, on postfix server: /var/log/maillog
    • mail / mailx
      • Instal·lació / Installation
        • CentOS
          • sudo yum install mailx
      • Utilització / Usage
  • Mozilla Thunderbird (correu/mail)
    • Command line options
      • create a new profile
        • thunderbird -CreateProfile my_profile
        • it will be created at:
          • ~/.thunderbird/xxxxxxxx.my_profile/
      • open gui profile manager
        • thunderbird -P
      • open a profile
        • thunderbird -p my_profile
    • Thunderbird 3
    • Redacció d'un missatge en HTML quan per omissió és en text pla / Compose a message in HTML when default is plain text:
      • Majúscula + click a "Redacta" / Shift + click on "Compose"
    • Expanded columns in folder pane
    • Signatures - Thunderbird
    • enllaços cap a firefox i altres aplicacions: ~/.thunderbird/.../prefs.js:
      • Opening hyperlinks from some GTK-based applications doesn't work after updating Firefox to a new version
      • Bug 58784 - Using "Make firefox the default web browser" should use /usr/bin/firefox not /usr/lib(64)/firefox-<version>/firefox
      • $ gconf-editor
        • /desktop/gnome/url-handlers/http/command
          • enabled: no
          • com que el valor per omissió és "enabled=true", això crearà el fitxer .gconf/desktop/gnome/url-handlers/http/%gconf.xml (si no, no existeix el fitxer)
        • llavors el thunderbird ens preguntarà amb quina aplicació volem obrir els http: /usr/bin/firefox
      • gconf: ~/.gconf/desktop/gnome/url-handlers/http/%gconf.xml
      • Al Firefox: Edita / Preferències / Avançat / General / Comprova-ho ara (predeterminat) (*)
      • user_pref("network.protocol-handler.app.http", "/usr/bin/xdg-open");
      • user_pref("network.protocol-handler.app.http", "/usr/bin/mozilla-firefox");
      • user_pref("network.protocol-handler.app.https", "/usr/bin/mozilla-firefox");
      • user_pref("network.protocol-handler.app.ftp", "/usr/bin/mozilla-firefox");
      • user_pref("network.protocol-handler.app.smb", "/usr/bin/konqueror");
    • Locale-Switcher Extension 
    • Tips & tricks
    • Consells i trucs (Softcatalà)
    • Use different Quote Level Colors
      • ~/.thunderbird/xxx.default/chrome/userContent.css
        • /* Quote Levels Colors */
          blockquote[type=cite] {
             color: navy !important; background-color: RGB(245,245,245) !important;
          }
          blockquote[type=cite] blockquote {
             color: maroon !important; background-color: RGB(235,235,235) !important;
          }
          blockquote[type=cite] blockquote blockquote {
             color: green !important; background-color: RGB(225,225,225) !important;
          }
          blockquote[type=cite] blockquote blockquote blockquote {
             color: purple !important; background-color: RGB(215,215,215) !important;
          }
          blockquote[type=cite] blockquote blockquote blockquote blockquote {
             color: teal !important; background-color: RGB(205,205,205) !important;
          }
    • Problem: "This body part will be downloaded on demand"
      • View->Display Attachments Inline
    • how to not include external images on html messages? - Mozilla
      • moz-do-not-send true
      • En recepció, caldrà acceptar "Mostra el contingut remot"
    • Lectura de fitxers mbox / Read mbox files
      • Reading an mbox file with Thunderbird
      • min_thunderbird.sh
        • #!/bin/bash

          EXPECTED_ARGS=1
          if (( $# != $EXPECTED_ARGS ))
          then
              cat <<EOF
          Usage: `basename $0` profile_name

          Create a thunderbird profile with given name, and add a default mail setup, in order to read mbox files.
          EOF
              exit 1
          fi

          # check that thunderbird is installed
          if ! command -v thunderbird2 >/dev/null 2>&1
          then
              echo "ERROR: command thunderbird does not exist. Please install it (https://www.thunderbird.net/) before running this script."
              exit 1
          fi

          profile_name=$1

          root_thunderbird_dir=${HOME}/.thunderbird

          # create profile
          echo "1. creating profile: ${profile_name}"
          thunderbird -CreateProfile ${profile_name}

          # get profile path
          profile_dirname=$(find ${HOME}/.thunderbird -type d -name "*\.${profile_name}")
          echo "  created: ${profile_dirname}"

          # start thunderbird, to create dirs
          echo "2. starting thunderbird to create dirs: please just cancel and close"
          thunderbird -p ${profile_name}


          # backup orginal prefs.js
          prefs_path=${profile_dirname}/prefs.js
          original_prefs_path=${profile_dirname}/prefs.original.js
          echo "3. backing up ${prefs_path} -> ${original_prefs_path}"
          cp ${prefs_path} ${original_prefs_path}

          # modify prefs.js
          echo "4. modifying ${prefs_path} to add a minimal mail config"
          cat >>${prefs_path} <<EOF
          user_pref("mail.root.none", "${profile_dirname}/Mail");
          user_pref("mail.accountmanager.accounts", "account1");
          user_pref("mail.accountmanager.localfoldersserver", "server1");
          user_pref("mail.account.account1.server", "server1");
          user_pref("mail.server.server1.directory", "${profile_dirname}/Mail/Local Folders");
          user_pref("mail.server.server1.directory-rel", "[ProfD]Mail/Local Folders");
          user_pref("mail.server.server1.hostname", "Local Folders");
          user_pref("mail.server.server1.name", "Carpetes locals");
          user_pref("mail.server.server1.nextFilterTime", 10);
          user_pref("mail.server.server1.spamActionTargetAccount", "mailbox://nobody@Local%20Folders");
          user_pref("mail.server.server1.storeContractID", "@mozilla.org/msgstore/berkeleystore;1");
          user_pref("mail.server.server1.type", "none");
          user_pref("mail.server.server1.userName", "nobody");
          EOF

          # start thunderbird
          echo "5. starting thunderbird: just check that default mail folders have been created and close it"
          thunderbird -p ${profile_name}

          # additional mbox files
          echo "6. to read mbox files, put them on (or link them from) ${profile_dirname}/Mail/Local Folders/ and restart thunderbird by using one of the following:"
          echo "  thunderbird -p ${profile_name}"
          echo "  thunderbird -P"
          echo "To delete profile ${profile_name}:"
          echo "  thunderbird -P"
          echo "and select Delete Profile..."
          echo "IMPORTANT: if you made a symbolic link to your mbox file from ${profile_dirname}/Mail/Local Folders/, you can choose Delete Files option. Otherwise, choose Don't Delete Files."

          exit 0
  • Gmail

Navegadors / Browsers

HTML

CSS

XHTML

XML

  • Extensible Markup Language (XML) (W3C)
  • Schemas
  • The XML library for Gnome
  • <recursos XML/>
  • XML Pitstop
  • Search
    • XPath
      • XPath tutorial (w3 schools)
      • Free Online XPath Tester / Evaluator
      • ...

        <root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org">
          <actors>
            <actor id="1">Christian Bale</actor>
            <actor id="2">Liam Neeson</actor>
            <actor id="3">Michael Caine</actor>
          </actors>
          <foo:singers>
            <foo:singer id="4">Tom Waits</foo:singer>
            <foo:singer id="5">B.B. King</foo:singer>
            <foo:singer id="6">Ray Charles</foo:singer>
          </foo:singers>
        </root>
        selecciona el node del document /
        selecciona el node 'root' /root
        selecciona tots els elements 'actor' que són directament fills de l'elelement 'actors' /root/actors/actor
        selecciona tots els elements 'singer', siguin on siguin al document //foo:singer
        selecciona l'atribut 'id' dels elements 'singer', siguin on siguin //foo:singer/@id
        selecciona el valor del text del primer element 'actor' //actor[1]/text()
        selecciona l'últim element 'actor' //actor[last()]
        selecciona el primer i segon elements 'actor' segons la seva posició //actor[position() < 3]
        selecciona tots els elements 'actor' que tinguin atribut 'id' //actor[@id]
        selecciona l'element 'actor' que té l'atribut 'id' amb valor '3' //actor[@id='3']
        selecciona l'element 'actor' que té l'atribut 'id' amb valor més petit o igual que '3' //actor[@id<='3']
        selecciona totsel fills del node 'singers' /root/foo:singers/*
        selecciona tots els elements del document //*
        selecciona els elements 'actor' i els elements 'singer' //actor|//foo:singer
        selecciona el nom del primer element del document name(//**[1])
        selecciona el valor numèric de l'atribut del primer element 'actor' number(//actor[1]/@id)
        selecciona la representació en text del valor de l'atribut 'id' del primer element 'actor' string(//actor[1]/@id)
        selecciona la longitud del valor textual del primer element 'actor' string-length(//actor[1]/text())
        selecciona el nom local del primer element 'singer' (és a dir, sense espais) local-name(//foo:singer[1])
        selecciona el nombre d'elements 'singer' count(//foo:singer)
        selecciona la suma dels atributs 'id' dels elements 'singer' sum(//foo:singer/@id)
  • Validate
  • Style
    • XSL (EXtensible Stylesheet Language)
      • style sheets for XML (CSS = style sheets for HTML)
  • Transform
    • Formats
      • XSLT
        • language desc tutorial reference functions
          XSLT (XSL Transformations) transforming XML documents XSLT Introduction XSLT elements reference
          Functions reference
          XPath navigating in XML documents XPath tutorial

          XQuery querying XML documents


        • XML and XSLT (w3schools)
        • Conversion XML -> HTML
          • browser will take file specified on xml-stylesheet to render XML as HTML; if no xml-stylesheet is specified, it will render raw xml
          • cdcatalog.xml cdcatalog.xsl (xml -> html) cdcatalog_text.xsl (xml -> txt)
            <?xml version="1.0" encoding="UTF-8"?>
            <?xml-stylesheet type="text/xsl" href="cdcatalog.xsl"?>
            <catalog>
              <cd>
                <title>Empire Burlesque</title>
                <artist>Bob Dylan</artist>
                <country>USA</country>
                <company>Columbia</company>
                <price>10.90</price>
                <year>1985</year>
              </cd>
            .
            .
            </catalog>
            <?xml version="1.0" encoding="UTF-8"?>

            <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

            <xsl:output method="html"/>
            <xsl:template match="/">
              <html>
              <body>
              <h2>My CD Collection</h2>
              <table border="1">
                <tr bgcolor="#9acd32">
                  <th>Title</th>
                  <th>Artist</th>
                </tr>
                <xsl:for-each select="catalog/cd">
                 
            <xsl:sort select="artist"/>       <tr>
                    <td><xsl:value-of select="title"/></td>
                    <td><xsl:value-of select="artist"/></td>
                  </tr>
                </xsl:for-each>
              </table>
              </body>
              </html>
            </xsl:template>

            </xsl:stylesheet>
            <?xml version="1.0" encoding="UTF-8"?>

            <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
              <xsl:output method="text" media-type="text/plain"/>

              <xsl:template match="/">
                <xsl:for-each select="catalog/cd">
                  <xsl:value-of select="title"/>
                  <xsl:text>: </xsl:text>
                  <xsl:value-of select="artist"/>
                  <xsl:text>&#xa;</xsl:text>
                </xsl:for-each>
              </xsl:template>
             
            </xsl:stylesheet>
        • Conversion XML -> JSON
    • Tools
    • Exemples / Examples
  • Editors 
  • Biblioteques / Libraries
  • Compressió / Compression
  • Eines / Tools
    • Command-line XML processing
    • How to parse XML using shellscript? [duplicate]
    • xmlstarlet (transform, query, validate and edit)
      • Documentation (single html)
      • Instal·lació / Installation
        • Mageia
          • urpmi xmlstarlet
          • binary is xmlstartlet instead of xml
      • Changing the value of an XML element depending on the value of another element in the tree
      • Searching for XML tag by value between them and inserting a new tag in shell script
      • Estructura / Structure:
        • xmlstarlet el toto.xml
      • Reformata / Reformat (prettify)
        • xmlstarlet format --encode utf-8 input.xml >output.xml
      • Validating XML documents (using XSD, DTD)
        • xml val -s toto.xsd -e toto.xml >validation_output.txt 2>&1
        • xmlstarlet val -s toto.xsd -e toto.xml >validation_output.txt 2>&1
      • Transforming XML documents (using XSLT)
        • xml tr xsl/param1.xsl xml/table.xml
        • xmlstarlet tr xsl/param1.xsl xml/table.xml
        • set parameters defined in param1.xsl as <xsl:param name="my_param">
          • xmlstarlet tr xsl/param1.xsl -s my_param=my_value xml/table.xml
      • Print to plain text
        • How to convert multi level xml to a single line containing all levels using XMLSTARLET
        • XMLStarlet: Printing one line per item, while using datum from parent element
        • Use XMLStarlet to parse XML in the Linux terminal
        • xmlstarlet sel ...
          • -C - show xsl generated code
          • -T - text output
          • -t - next arguments specify a template
            • -m, --match <xpath> - match
            • -v, --value-of <xpath> - print value (between open and closing tags)
            • -o <string> - print literal string
            • -n - print newline
            • ...
        • Exemples / Examples
          • myfile.xml
            • <?xml version="1.0" encoding="UTF-8" standalone="no"?>
              <xml>
                <os>
                 <linux>
                  <distribution when="today">
                    <name>Fedora</name>
                    <release>7</release>
                    <codename>Moonshine</codename>
                    <clau usage="signing">
                      clau per signar
                    </clau>
                    <clau usage="encryption">
                      clau per xifrar
                    </clau>
                    <spins>
                      <name>Live</name>
                      <name>
                        Fedora
                      </name>
                      <name>Everything</name>   
                    </spins>
                  </distribution>

                  <distribution when="tomorrow">
                    <name>Fedora Core</name>
                    <release>6</release>
                    <codename>Zod</codename>
                    <spins></spins>
                  </distribution>
                 </linux>
                </os>   
              </xml>
          • estructura:
            • xmlstarlet el input.xml
            • sense ordre ni repetició:
              • xmlstarlet el -u input.xml
            • amb atributs:
              • xmlstarlet el -a input.xml
            • amb els valors dels atributs:
              • xmlstarlet el -v input.xml
          • parts a partir de distribution, sense les etiquetes, substituïdes per espais buits:
            • xmlstarlet select --template --value-of /xml/os/linux/distribution input.xml
          • si voleu tot l'xml:
            • xmlstarlet select --template --copy-of /xml/os/linux/distribution input.xml
          • només el contingut de distribution/name:
            • xmlstarlet select --template --value-of /xml/os/linux/distribution/name --nl myfile.xml
          • selecció condicional, fent servir XPath:
            • xmlstarlet sel --template --value-of '/xml/os/linux/distribution[name = "Fedora Core"]/release' --nl myfile.xml
            • xmlstarlet sel --template --value-of '/xml/os/linux/distribution[name = "Fedora Core"]/codename' --nl myfile.xml
            • filtre per atribut, amb sortida xml (--copy-of):
              • xmlstarlet sel --template --match '/xml/os/linux/distribution/clau[@usage = "encryption"]' --copy-of '.' --nl myfile.xml
          • match seguit de value-of:
            • xmlstarlet sel --template --match '/xml/os/linux/distribution/spins' --value-of '.' --nl myfile.xml
            • xmlstarlet sel --template --match '/xml/os/linux/distribution' --value-of 'spins' --nl myfile.xml
          • atribut:
            • xmlstarlet sel --template --match '/xml/os/linux/distribution' --value-of '@when' --nl myfile.xml
          • si voleu eliminar els espais i canvis de línia:
            • xmlstarlet sel --template --match '/xml/os/linux/distribution/spins/name' --value-of 'normalize-space()' --nl myfile.xml
            • xmlstarlet sel --noblanks --template --match '/xml/os/linux/distribution/clau[@usage = "encryption"]' --value-of 'normalize-space()' myfile.xml
          • si el fitxer especifica xmlns (namespace):
            • myfile_ns.xml
              • <?xml version="1.0" encoding="UTF-8" standalone="no"?>
                <xml xmlns="...">
                  <os>
                   <l