Thursday, November 24, 2016

Logstash+Elasticsearch: Best way to handle JSON arrays



Before I start with the solution, let's review what's the problem we're trying to solve here. If we have these two JSON documents pushed to ES:-
{
    "test": {
        "steps": [{
            "response_time": "100"
        }, {
            "response_time": "101"
        }]
    }
}
{
    "test": {
        "steps": [{
            "response_time": "101"
        }, {
            "response_time": "100"
        }]
    }
}
And you write a Kibana query like:
test.steps.response_time:101
# Full ES query in the background
{
    "query": {
        "query_string": {
           "query": "test.steps.response_time:101"
        }
    }
}
It'll match both documents. Why? Because Elasticsearch flattens the arrays internally.
More details:- https://www.elastic.co/guide/en/elasticsearch/guide/current/complex-core-fields.html#object-arrays and https://www.elastic.co/guide/en/elasticsearch/guide/current/nested-objects.html

Not just that, if I were to write a query to search all documents with response_time=101 in second element of array, logically, test.steps[1].response_time:101, it's not possible.

To fix this, we can simple create a filter in Logstash which converts these arrays to hashes recursively, ie, all arrays are converted to hashes, even the nested ones. Hence, we want to write a filter which converts arrays like this.

Before:-
{
  "foo": "bar",
  "test": {
    "steps": [
      {
        "response_time": "100"
      },
      {
        "response_time": "101",
        "more_nested": [
          {
            "hello": "world"
          },
          {
            "hello2": "world2"
          }
        ]
      }
    ]
  }
}
After:-
{
  "foo": "bar",
  "test": {
    "steps": {
      "0": {
        "response_time": "100"
      },
      "1": {
        "response_time": "101",
        "more_nested": {
          "0": {
            "hello": "world"
          },
          "1": {
            "hello2": "world2"
          }
        }
      }
    }
  }
}
The filter that can do this is shared below:-
ruby {
    init => "
        def arrays_to_hash(h)
          h.each do |k,v|
            # If v is nil, an array is being iterated and the value is k.
            # If v is not nil, a hash is being iterated and the value is v.
            value = v || k
            if value.is_a?(Array)
                # "value" is replaced with "value_hash" later.
                value_hash = {}
                value.each_with_index do |v, i|
                    value_hash[i.to_s] = v
                end
                h[k] = value_hash
            end

            if value.is_a?(Hash) || value.is_a?(Array)
              arrays_to_hash(value)
            end
          end
        end
      "
      code => "arrays_to_hash(event.to_hash)"
}
Now, to search the document which contains response_time=101 in second element of array, it's simple.
test.steps.1.response_time:101
Happy ELKing!

Tuesday, September 6, 2016

Monitoring Raspberry Pi using my DigitalOcean VPS

So, this is a simple one and just felt like sharing in case someone wants to save time writing those one-liners.

I had a faulty raspberry pi power supply and my Rpi used to hang unexpectedly. Here is what I did to monitor uptime of my Rpi.


  • Host a simple text file via nginx on my DigitalOcean VPS.
➜ $?=0 /home/shadyabhi [ 1:20PM] % cat /var/www/html/rpihealth
GOOD

>>>  0s elasped...
➜ $?=0 /home/shadyabhi [ 1:20PM] %
  • Write a simple systemd service in Rpi to curl this file. 
➜ $?=0 /home/shadyabhi [ 5:21PM] % sudo cat /usr/lib/systemd/system/send_latest_ip.service
[Unit]
Description=Latest IP sender

[Service]
User=shadyabhi
ExecStart=/usr/local/bin/send_uptime.sh

[Install]
WantedBy=multi-user.target

>>>  0s elasped...
➜ $?=0 /home/shadyabhi [ 5:21PM] % cat /usr/local/bin/send_uptime.sh
#!/bin/bash

while [ 1 ]; do
        curl https://abhijeetr.com/rpihealth
        sleep 1
done

>>>  0s elasped...
➜ $?=0 /home/shadyabhi [ 5:21PM] % sudo systemctl enable send_uptime
  • A sample logline for a health checks looks like this.
106.51.128.57 - - [06/Sep/2016:13:37:46 -0400] "GET /rpihealth HTTP/1.1" 200 5 "-" "curl/7.50.0" "-"
  • A cronjob on my VPS to send me notifications via pushover if there are no log lines for this health check in the last 10 seconds. (using https://github.com/jnwatts/pushover.sh)
* * * * * sudo /usr/bin/perl -MDate::Parse -ne 'print if/^.*- - \[(.*?)\] .*?/&&str2time($1)>time-10' /var/log/nginx/access.log | if ! /bin/grep -q "/rpihealth"; then /home/shadyabhi/pushover.sh/pushover.sh -t "RPi Health" "Rpi is down"; fi

Now, whenever my Rpi is down, I get push notifications on my phone.





Saturday, February 27, 2016

Golang: Creating HTTPS connection via proxy

I'm writing this post because I had to read a bit of golang code to figure out how to go about doing this.

User familiar with crypto/tls/ will notice that there's already a function available to establish a TLS connection, tls.Dial. However, it doesn't have any option to specify proxy. Why? Because, TLS connection has nothing to do with proxy, TLS is available as a addon to an already existing TCP connection. It exists one level below HTTP, the application layer protocol. That's exactly the reason tls.Dial function is not present in the net/http package too.

A simple way to do a request is http.Client.Get. (). Let's see what it does.
func (c *Client) Get(url string) (resp *Response, err error) {
    req, err := NewRequest("GET", url, nil)
    if err != nil {
        return nil, err
    }
    return c.doFollowingRedirects(req, shouldRedirectGet)
}

It simply does a request using NewRequest function. All it does is returns a proper Request struct depending on various things like type of request, URL to open, headers etc etc. Depending on various StatusCode, headers etc received, request is handled and ultimately (*Client).send is called. This function handles cookies and then actually calls the send function in the net/http package which is not available to the outside world.

func (c *Client) send(req *Request, deadline time.Time) (*Response, error) {
 if c.Jar != nil {
  for _, cookie := range c.Jar.Cookies(req.URL) {
   req.AddCookie(cookie)
  }
 }
 resp, err := send(req, c.transport(), deadline)
 if err != nil {
  return nil, err
 }
 if c.Jar != nil {
  if rc := resp.Cookies(); len(rc) > 0 {
   c.Jar.SetCookies(req.URL, rc)
  }
 }
 return resp, err
}

Interesting thing to note here is the c.transport() that we're passing. This function returns the transport being used for the current client.
func (c *Client) transport() RoundTripper {
 if c.Transport != nil {
  return c.Transport
 }
 return DefaultTransport
}
From the docs:-
Transport is an implementation of RoundTripper that supports HTTP, HTTPS, and HTTP proxies (for either HTTP or HTTPS with CONNECT).
So, this is where we can set proxies. RoundTripper is an interface and pointer to http.Transport implements that interface.

RoundTripper is an interface representing the ability to execute a single HTTP transaction, obtaining the Response for a given Request. The http module providers an implementation of RoundTripper, http.Transport
type Transport struct {

    // Proxy specifies a function to return a proxy for a given
    // Request. If the function returns a non-nil error, the
    // request is aborted with the provided error.
    // If Proxy is nil or returns a nil *URL, no proxy is used.
    Proxy func(*Request) (*url.URL, error)
We've found our Proxy. SUCCESS!

So, all we need to do is, create a custom http.Transport and then a http.Client that uses this newly created http.Transport.

Now onto the real task at hand.
  • Create a custom transport using proxy
  •         // Create proxy
            proxyURL, _ := url.Parse(*proxy)
    
            transport := http.Transport{
                Proxy:           http.ProxyURL(proxyURL),
                TLSClientConfig: &tls.Config{},
            }
    
  • Create the http.Client object
  •         client = http.Client{
                Transport: &transport,
            }
    
  • Call http.Client.Get
  • resp, _ := client.Get("https://www.google.co.in")
    
Using what we have learnt, here's the code below that connects to a live HTTPS endpoint, fetches its certificate and shows the days to expiry for that cert. 
package main

import (
        "crypto/tls"
        "flag"
        "fmt"
        "net/http"
        "net/url"
        "time"
)

const timeout time.Duration = 10

func main() {
        // Parse cmdline arguments using flag package
        server := flag.String("server", "abhijeetr.com", "Server to ping")
        port := flag.Uint("port", 443, "Port that has TLS")
        proxy := flag.String("proxyURL", "", "Proxy to use for TLS connection")
        flag.Parse()

        // Prepare the client
        var client http.Client
        if *proxy != "" {
                proxyURL, err := url.Parse(*proxy)
                if err != nil {
                        panic("Error parsing proxy URL")
                }
                transport := http.Transport{
                        Proxy:           http.ProxyURL(proxyURL),
                        TLSClientConfig: &tls.Config{},
                }
                client = http.Client{
                        Transport: &transport,
                        Timeout:   time.Duration(time.Millisecond * timeout),
                }

        } else {
                client = http.Client{}
        }
        // Now we've proper client, with or without proxy

        resp, err := client.Get(fmt.Sprintf("https://%v:%v", *server, *port))
        if err != nil {
                panic("failed to connect: " + err.Error())
        }

        fmt.Printf("Time to expiry for the certificate: %v\n", resp.TLS.PeerCertificates[0].NotAfter.Sub(time.Now()))
}


 Cheers !!

Friday, January 15, 2016

Code Snippet: A redirection service in Go

Recently, I've been very fascinated with Go. That language is so easy to write and as it's statically compiled, less changes of bugs too. So, I keep finding excuses to write Go code.

I needed a service to redirect clients which runs inside a docker container. One obvious option is to run apache inside docker with proper configs but as I enjoy writing Go code, I wrote this really simple program that acts like a redirection service. Later, I just copy this compiled binary into docker container.
  • Specify location and port from command-line to set the location to redirect to. 
  • Each client is handled in it's own goroutine, that means, it's scalable. 
  • That few lines of code are so elegant. I love Go.

Code:- ( https://github.com/shadyabhi/redirection_service )

package main

import (
	"flag"
	"log"
	"net/http"
	"strconv"

	"github.com/asaskevich/govalidator"
)

// redirect function does the redirection
func redirect(w http.ResponseWriter, r *http.Request, location string) {
	w.Header().Set("Location", location)
	w.WriteHeader(http.StatusFound)
}

func main() {
	// Handle cmdline arguments
	location := flag.String("location", "localhost:80", "The URI at which the ")
	port := flag.Int("port", 80, "Port at which the http server binds to")
	flag.Parse()

	// Validate
	isURL := govalidator.IsURL(*location)
	if !isURL {
		log.Fatal("The location you provided is not a valid URL")
	}

	log.Printf("Starting web server... Location: %s, Port: %d", *location, *port)

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		redirect(w, r, *location)
	})

	err := http.ListenAndServe(":"+strconv.Itoa(*port), nil)
	if err != nil {
		log.Fatal("Error starting web server. ", err)
	}
}

Usage:-

% ./redirection_service -h
Usage of ./redirection_service:
  -location string
        The URI at which the  (default "localhost:80")
  -port int
        Port at which the http server binds to (default 80)
% sudo ./redirection_service -location=https://abhijeetr.com
2016/01/15 09:33:54 Starting web server... Location: https://abhijeetr.com, Port: 80

Tuesday, January 12, 2016

Code Snippet: Fetch certificate information from a live endpoint in Go

I generally try to post snippets on my blog that I don't find on a Google search. This is like one of those snippets.

This sample code fetches the time to expiry for a certificate that's currently deployed at a live endpoint. 

Code:-

package main

import (
 "crypto/tls"
 "flag"
 "fmt"
 "time"
)

func main() {
 // Parse cmdline arguments using flag package
 server := flag.String("server", "abhijeetr.com", "Server to ping")
 port := flag.Uint("port", 443, "Port that has TLS")
 flag.Parse()

 conn, err := tls.Dial("tcp", fmt.Sprintf("%s:%d", *server, *port), &tls.Config{})
 if err != nil {
  panic("failed to connect: " + err.Error())
 }

 // Get the ConnectionState struct as that's the one which gives us x509.Certificate struct
 connectionState := conn.ConnectionState()

 fmt.Printf("Time to expiry for the certificate: %v\n", connectionState.PeerCertificates[0].NotAfter.Sub(time.Now()))
 conn.Close()
}

Usage:-

% ./ssl_certs -h
Usage of ./ssl_certs:
  -port uint
     Port that has TLS (default 443)
  -server string
     Server to ping (default "abhijeetr.com")
% ./ssl_certs
Time to expiry for the certificate: 1843h21m3.857177591s