Monday, 4 December 2017

Puppet - Roles and Profiles

Forgetting about Puppet for a moment - if we are to conceptually describe how our servers are built as part of a large distributed group then we would describe them initially in terms of business function - e.g. this server stores data of financial transactions, it also needs to allow certain people to login to it, it needs to be monitored etc etc - whilst another server might be described at a high level as running a web server, needs the ability to allow certain people to login etc.

The high level description of what a server does we call a role (e.g. data store role and web server role in the example above).

The technical sub-components of a role we call a profile (e.g. install postgres database module, install ldap client module, install nginx module).

We can further subdivide profiles into roles that are common to many systems, e.g. we can define . common profile that contains:

common profile -> ldap module + packages module + monitoring module + security module

then we could define a web server profile:

webserver profile -> nginx module + haproxy module

we can define multiple profiles that contain the technical building blocks which we can put together for a specific role (business function). Each server is described by one role. Each role is formed of at least one profile - usually more. Profiles can be reused multiple times in different roles.

then our role would look like:

webserver role -> common profile + webserver profile

I have a git repository that shows a simple example of using roles and profiles - puppet rolesandprofiles

Sunday, 1 October 2017

Arista EOS - CLI quickstart

This is just a brain dump for me as I seem to very occassionally have to work on Arista switches - which by the way are pretty nice - mini version of linux on them.

Why do I like Arista - Andy Bechtolsheim (one of the founders of Sun Microsystems who I used to work for) - nice to have a mad scientist doing well ;-) - also good for Cisco to have some competition - monopolies are never a good thing. Also provided first (that I'm aware) of ULL switches. Other reasons I like it - it has linux (and so bash), sysdb - database on the switch that holds important data - used similarly to IPC, MLAG - allows port channels to exist on multiple switches at the same time, VARP - allows multiple switches to respond to arp requests for same IP, ZTP (zero touch provisioning) - loads config from network, LANZ - latency analyser, email, job scheduler (is this just cron?), tcpdump, event handlers, event monitors.

to login:
ssh admin@switch

what version are we running:
show ver

what interfaces are attached and whats their status:
show interfaces
show int status
show interfaces  ethernet 1-5 status
show run? (detail config with defaults)

what have I been doing here ;-) :
show history

lets go crazy and make changes outside of change control and config management:
chicmsw01(config)#interface ethernet 8
Enter TEXT message. Type 'EOF' on its own line to end.
testing rancid

bash - gets us into a bash shell - nice if your from a linux background
bash python - wow!
environment -> can change fan speed etc
show logging ?
show log last 5 min
can configure syslog/email
show reload cause full (see why switch rebooted?)

lets ensure my crazy changes persist after a restart:
copy running-config startup-config

Also worked with implementing rancid - also pretty cool - dumps configs, saves to change control - and alerts me on any diffs.

Friday, 16 June 2017

Google Cloud Platform

So recently I received an email from google inviting me to use gcm free for a year - also came with $300  to pay towards chargeable services - very nice.

Previously I've used aws - pretty familiar - also its almost becoming the de-facto standard in most companies I've contracted for  - I think competition is good - so having an alternative should be welcomed.

so - google has multiple data centres worldwide - similarly to amazon it has 3-4 in each region.
when creating a machine you choose a region and a zone.

google also has its own fiber network - never more than 500 miles from an access point

  • compute engine  - iass
  • app engine - paas
  • managed services - elastic resources, machine learning, big data
  • container engine (docker/kubernetes)
  • flexible machine types - any cpu/memory configuration
  • simpler firewall rules
  • bills by minute, not hour
accessing gcp :

web console:
android or iphone app
can ssh from command line after server built
programmatically - api - rest: and

stackdriver looks interesting - this is fordiagnostics, logging, monitoring - integration into other tools such as elk/splunk/patrol etc/ debug console for java and python apps

equivalent of lamda is cloud functions - serverless/ephemeral function that responds to events - not worked on this but sounds very interesting - replacement for stuff like jenkins/ci?

cloud storage buckets are similar to s3 (which is uber cool)

machine learning - mlaas? - machine learning as a service - this is something that I certainly want to learn more about 

Monday, 5 June 2017

linux changing screen resolution

So recently I installed Linux Mint/sarah - I installed this under vmware.

All latest vmware tools installed - however trying to change screen resolution didn't give me the option corresponding to the actual resolution of my screens (1920x1080).

To fix this:

stuart@stuart-virtual-machine ~ $ cvt 1920 1080
# 1920x1080 59.96 Hz (CVT 2.07M9) hsync: 67.16 kHz; pclk: 173.00 MHz
Modeline "1920x1080_60.00"  173.00  1920 2048 2248 2576  1080 1083 1088 1120 -hsync +vsync

stuart@stuart-virtual-machine ~ $ xrandr --newmode "1920x1080_60.00"  173.00  1920 2048 2248 2576  1080 1083 1088 1120 -hsync +vsync

stuart@stuart-virtual-machine ~ $ xrandr --addmode Virtual1 "1920x1080_60.00"

now the required resolution appears in display settings and I'm happy :-)

Wednesday, 12 April 2017

FPM - easy way to build packages

FPM ('flipping' rpm ;-) ) is a really cool way to build packages without the general grind of creating spec files etc .. heres my quick howto for CentOS:

  • yum -y install rpm-build
  • gem install fpm
  • mkdir -p stupack/files stupack/etc stupack/logs (just making an arbitrary dir tree)
  • touch stupack/files/test stupack/etc/stu.conf stupack/logs/stulog (dumping some dumb files there
  • fpm -s dir -t rpm -n stupack stupack
              Created package {:path=>"stupack-1.0-1.x86_64.rpm"}
  •  rpm -qlp ./stupack-1.0-1.x86_64.rpm 

      yay - we have an rpm in 2 secs of easyness!

      we can add  version number with a -v x.y.z to the fpm command
      x86_64 was chosen for architecture - but could choose this as anything..


      -t (rpm,deb,solaris..)
      -s input type (directories,gem,rpm..)
      -f force - overwrite existing
      -n name to give to package
      --license - name of lic for the package
      --provides - what this pkg provides
      --config-files - marks a file in package as a config file

      overall - very nice, wish I'd had it when first tediously creating rpms!

        Wednesday, 5 April 2017

        How to use strace

        How to use strace

        really cool unix utility, lets us inspect what an executable is doing - dont need:

        • source code
        • knowledge of what  program the executable came from
        • debugger
        one big caveat to note is that strace stops/starts a process so can make it a lot, lot slower - e.g.:

        [root@einstein ~]# time dd if=/dev/zero of=./test bs=1024k count=1000
        1000+0 records in
        1000+0 records out
        1048576000 bytes (1.0 GB) copied, 11.0078 s, 95.3 MB/s
        real    0m11.050s
        user    0m0.001s
        sys     0m0.377s
        [root@einstein ~]#  sync; echo 3 > /proc/sys/vm/drop_caches
        [root@einstein ~]# time strace -f -o out  dd if=/dev/zero of=./test bs=1024k count=1000
        1000+0 records in
        1000+0 records out
        1048576000 bytes (1.0 GB) copied, 14.4345 s, 72.6 MB/s
        real    0m14.491s
        user    0m0.005s
        sys     0m0.406s
        [root@einstein ~]# 
        so approximately 40% degradation here in performance - something you wouldnt want in prod.

        How does a program interact with my computer?

        so when a program runs in user mode it doesnt have direct access to the hardware (unless its something cool like solaflare network card or mellanox..).
        For the program to get access to the hardware it needs to use a system call..(have a look at man 2 syscalls) - systems calls are essentially how a user program enters the kernel to perform a privileged task.

        the categories of system calls are:
        • process control (load/exec/abort/create/terminate/get/set attributes/wait/allocate/free memory)
        • file management (create/delete/open/close/read/write/reposition/get-set attributes)
        • device management (request/release/read/write/reposition/attach/detach)
        • info maintenance (get/set time date/data/process file or dev attributes)
        • communicate (create/del connection/send/rcv message/transfer status info/attach or detach remote dev)


        you dont need to be root to use strace - just have permission to read the process (i.e. generally you're own proc)
        lots of output - try

        # strace ls

        following the output of this we'll see an :
        • execve - (os starts process)
        • brk(0) - kludge to read end of data segment
        • open of filenames and assigning a filedescriptor number
        • read using filedescriptor
        • fstat details perm owner etc
        • close
        to see which files a program is opening we could use:

        strace -e open ls

        as a practical example - lets see what config files a program uses (e.g. if i type bash does it use profile/bashrc or bash_profile - I sometimes forget!):
        strace -e open bash
        /truncated output)
        open("/home/sabramshumphries/.bashrc", O_RDONLY) = 3
        open("/etc/bashrc", O_RDONLY)           = 3

        yay - we can see the files it opens!!

        useful flags:

        -e just list the system calls specified - e.g. strace -e open,close
        -f follow any subprocesses also that are created
        -p - follow a process that started earlier
        -o - write to a file so can look through output more easily
        -s print out lines of length.. e.g. give a larger number so strace doesnt truncate output)
        -c statistical summary of calls - shows output really nicely 
        -t show timestamp (-tt even finer time)
        -r relative time between calls

        syscall                           function
        open                             opens a file,returns a fd
        close                             closes a filedescriptor
        read                              reads bytes from a fd
        write                             writes to a fd
        fork                               creates a new process - cow of parent
        exec                              executes a new program
        brk                                extend heap pointer
        mmap                           map a file to process address space
        stat                                read info about a file
        ioctl                               set io properties

        Monday, 27 March 2017

        What on earth is devops?

        Devops is a pretty contentious word - lots of old school people say its nothing new and its just a label for automating everything - which people have tried (and are still trying!) to do.

        Most people agree devops isn't:-

        • a job title
        • a job description
        • an organisation/team
        I've been guilty of all three of the above misdemeanours.

        From wikipedia theres a pretty good definition of devops:

        In traditional functionally separated organizations there is rarely cross-departmental integration of these functions with IT operations. devops promotes a set of processes and methods for thinking about communication and collaboration between development, QA, and IT operations.

        So - what this is saying is essentially devops is about bringing systems admin/engineers and developers and testers closer together - rather than in silos. Hence why having a devops team is something I consider not a brilliant idea- we shouldnt be creating an extra silo - we should be bringing teams closer.

        Technology isn't so important here  - the main thing is culture/behaviour - we also need to think more in terms of the overall toolchain rather than in terms of a devops tool, such as puppet/ansible..

        • code - version control, frequent merges
        • build - ci tools
        • test - test code works (verify/validate) - performance
        • package - stage software
        • release (tricky  - always seen a lot of human interaction here) - change management/approvals
        • configure - iaas
        • monitor - is it all working!
        so essentially a large and complex set of practises and tools to bring code quicker and more reliably to the customer.

        bash best practices

        Bash best practices A few hints on bash best practice: * use #!/usr/bin/env bash .. this is more portable but you cant rely on a spe...