bash best practices

Bash best practices

A few hints on bash best practice:


* use #!/usr/bin/env bash .. this is more portable but you cant rely on a specific version
* use set, dont use options to bash - if someone runs your script with bash scriptname then it will ignore the options to bash
* use {} to enclose variables - can cause mistakes if you donr - e.g. variable name becomes VAR_ext rather than what you wanted
* to ensure you always have a value, set defaults - e.g. "${MYNAME}" = "Stuart" - MYNAME defaults to Stuart if not already set
* use spaces for tabs, not tabs - tab not portable
* max line length of 80 characters for readability - use \ to split of lines if needed
* dont have whitespace at end of lines as may confuse source code control like git
* use $(command) instead of backticks
* variables and function names lowercase and underscores - use meaninful names
* constants should be in caps, declare them first in file
* use readonly to set a variable readonly
* can use local to make a variable specific to a function
* put functions together below the constants, I order the functions alphabetically as easier to find
* use a main function if using multiple functions
* check return values from functions
* avoid eval - munges input
* [[ .. ]] is better than test or /usr/bin/[ - stops pathname expansion and word splitting
* comment difficult bits of code
* insecure to have suid/sgid
* i prefer to have .sh extension so easy to recognise file
### good set options:

* -e exit script immediately if command fails
* -o pipefail fails if any part of a pipe fails
* -u treat unset variables as an error and exit immediately
* suggest second line of bash script is: set -eou pipefail
* -x prints each command before executing it - expands arguments also
* use -E if script contains traps
* to start with using a program like shellcheck can be useful - gives your scripts a quick checkover
* a useful tool if you use vim is the bash support plugin - see https://www.thegeekstuff.com/2009/02/make-vim-as-your-bash-ide-using-bash-support-plugin/

/var/log/journal

Currently working on creating a large repo - copied a lot of the packages to a VM in my home test lab - on extracting 15Gb of packages I noticed I was running short on space (had about 20Gb free but wanted to ensure extract didnt fail).

running the old du -hs * from / I noticed /var had a lot of storage being used.
(yes I dont have separate partitions on my smaller VMs)

Looking further I could see that it was pretty much being used in /var/log/journal:

# journalctl --disk-usage
Archived and active journals take up 3.9G in the file system.

I'm really not that interested in these logs - and so will add the following modification to my ansible scripts:

modify /etc/systemd/journald.conf => SystemMaxUse=100M

I then run:

systemctl kill --kill-who=main --signal=SIGUSR2 systemd-journald.service

and:

systemctl restart systemd-journald.service

running check again:

# journalctl --disk-usage

Archived and active journals take up 80.0M in the file system.

cool - lots of space cleared on my VM!

(I do recommend on servers having separate filesystems, and bigger ones than I use - however on home system I have 2Tb worth of ssd - this has to be shared amongst numerous smaller VMs)

TCP/IP

What is TCP/IP

tcp/ip is basically a set of rules/standards - see darpa standard

transmission control protocol/internet protocol

based on OSI model - but slightly different - 4 layers instead of 7

tcp is essentially the transport layer  - responsible for splitting up the data and posting it on the physical link  - but like a clerk in an office getting lots of things ready to deliver to a customer - splitting it into manageable parcels - ip is like the postman - he picks up the parcels and routes them to their destination.

A TCP packet runs ontop of an IP packet.

TCP Packet:

IP packet:

Three way handshake:

1. A tcp connection is established via a three way handshake - client sends a SYN (synchronize) packet to server with a random sequence number.

2. server sends back a SYN-ACK - containing another random sequence number and an ACK number to acknowledge clients sequence number

3. client then sends an ACK number to server, which aknowledges the servers sequence number.

now that the sequence numbers are synchronized, both ends can now send and receive data independently

connect ubuntu to wifi from command line

so i have a server install of a beta release of bionic beaver - ubuntu 18.04 - I want to connect this to wifi as I'm not near my cobbler server and cant connect to it via ethernet at the moment.

If we've connected before we can view previous connections:

nmcli c

- havent had any previous connections so no luck there.

to see wifi hotspots near me:

root@bionic-beaver-x8664:~# nmcli d wifi list
IN-USE  SSID                  MODE   CHAN  RATE        SIGNAL  BARS              SECURITY
abramshumps                       Infra  13    270 Mbit/s              100     ▂▄▆█        WPA2
abramshumps_5G                Infra  36    270 Mbit/s                69      ▂▄▆_         WPA1 WPA2
abramshumps                       Infra  13    270 Mbit/s                64      ▂▄▆_         WPA1 WPA2

cool - so wifi working out of the box this time - not going to have to go through pain of building drivers for my laptop (Dell Latitiude 7280)

so next lets see my wifi device:

root@bionic-beaver-x8664:~# ip link show
1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s31f6: mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether a4:4c:c8:21:7c:44 brd ff:ff:ff:ff:ff:ff
3: wlp2s0: mtu 1500 qdisc mq state DOWN mode DORMANT group default qlen 1000
    link/ether cc:2f:71:73:a3:3e brd ff:ff:ff:ff:ff:ff

we can try and connect to it now:

root@bionic-beaver-x8664:~# nmcli d wifi connect abramshumps password supersecretpw
Device 'wlp2s0' successfully activated with 'ead1b11c-3d70-4ba8-9ca9-48389235c7db'.

now lets test connectivity to internet:

root@bionic-beaver-x8664:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=60 time=13.2 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=60 time=13.3 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 13.292/13.344/13.397/0.126 ms
root@bionic-beaver-x8664:~#

yay - I can now add ubuntu-desktop and other goodies I want :-)

Puppet - Roles and Profiles

Forgetting about Puppet for a moment - if we are to conceptually describe how our servers are built as part of a large distributed group then we would describe them initially in terms of business function - e.g. this server stores data of financial transactions, it also needs to allow certain people to login to it, it needs to be monitored etc etc - whilst another server might be described at a high level as running a web server, needs the ability to allow certain people to login etc.

The high level description of what a server does we call a role (e.g. data store role and web server role in the example above).

The technical sub-components of a role we call a profile (e.g. install postgres database module, install ldap client module, install nginx module).

We can further subdivide profiles into roles that are common to many systems, e.g. we can define . common profile that contains:

common profile -> ldap module + packages module + monitoring module + security module

then we could define a web server profile:

webserver profile -> nginx module + haproxy module

we can define multiple profiles that contain the technical building blocks which we can put together for a specific role (business function). Each server is described by one role. Each role is formed of at least one profile - usually more. Profiles can be reused multiple times in different roles.

then our role would look like:

webserver role -> common profile + webserver profile


I have a git repository that shows a simple example of using roles and profiles - puppet rolesandprofiles

Arista EOS - CLI quickstart

This is just a brain dump for me as I seem to very occassionally have to work on Arista switches - which by the way are pretty nice - mini version of linux on them.

Why do I like Arista - Andy Bechtolsheim (one of the founders of Sun Microsystems who I used to work for) - nice to have a mad scientist doing well ;-) - also good for Cisco to have some competition - monopolies are never a good thing. Also provided first (that I'm aware) of ULL switches. Other reasons I like it - it has linux (and so bash), sysdb - database on the switch that holds important data - used similarly to IPC, MLAG - allows port channels to exist on multiple switches at the same time, VARP - allows multiple switches to respond to arp requests for same IP, ZTP (zero touch provisioning) - loads config from network, LANZ - latency analyser, email, job scheduler (is this just cron?), tcpdump, event handlers, event monitors.

to login:

ssh admin@switch

what version are we running:

show ver

what interfaces are attached and whats their status:

show interfaces

show int status

show interfaces  ethernet 1-5 status

show run? (detail config with defaults)

what have I been doing here ;-) :

show history

lets go crazy and make changes outside of change control and config management:

chicmsw01>enable

chicmsw01#configure

chicmsw01(config)#interface ethernet 8

chicmsw01(config-if-Et8)#comment

Enter TEXT message. Type 'EOF' on its own line to end.

testing rancid

EOF

chicmsw01(config-if-Et8)#end

misc:

bash - gets us into a bash shell - nice if your from a linux background

bash python - wow!

environment -> can change fan speed etc

show logging ?

show log last 5 min

can configure syslog/email

show reload cause full (see why switch rebooted?)

lets ensure my crazy changes persist after a restart:

copy running-config startup-config

Also worked with implementing rancid - also pretty cool - dumps configs, saves to change control - and alerts me on any diffs.

Google Cloud Platform

So recently I received an email from google inviting me to use gcm free for a year - also came with $300  to pay towards chargeable services - very nice.

Previously I've used aws - pretty familiar - also its almost becoming the de-facto standard in most companies I've contracted for  - I think competition is good - so having an alternative should be welcomed.

so - google has multiple data centres worldwide - similarly to amazon it has 3-4 in each region.
when creating a machine you choose a region and a zone.

google also has its own fiber network - never more than 500 miles from an access point

• compute engine  - iass
• app engine - paas
• managed services - elastic resources, machine learning, big data
• container engine (docker/kubernetes)
• flexible machine types - any cpu/memory configuration
• simpler firewall rules
• bills by minute, not hour

accessing gcp :

web console: cloud.google.com

android or iphone app

can ssh from command line after server built

programmatically - api - rest: https://developers.google.com/oauthplayground/ and developers.google.com/apis-explorer/#p

stackdriver looks interesting - this is fordiagnostics, logging, monitoring - integration into other tools such as elk/splunk/patrol etc/ debug console for java and python apps

equivalent of lamda is cloud functions - serverless/ephemeral function that responds to events - not worked on this but sounds very interesting - replacement for stuff like jenkins/ci?

cloud storage buckets are similar to s3 (which is uber cool)

machine learning - mlaas? - machine learning as a service - this is something that I certainly want to learn more about 

linux changing screen resolution

So recently I installed Linux Mint/sarah - I installed this under vmware.

All latest vmware tools installed - however trying to change screen resolution didn't give me the option corresponding to the actual resolution of my screens (1920x1080).

To fix this:


stuart@stuart-virtual-machine ~ $ cvt 1920 1080
# 1920x1080 59.96 Hz (CVT 2.07M9) hsync: 67.16 kHz; pclk: 173.00 MHz
Modeline "1920x1080_60.00"  173.00  1920 2048 2248 2576  1080 1083 1088 1120 -hsync +vsync

stuart@stuart-virtual-machine ~ $ xrandr --newmode "1920x1080_60.00"  173.00  1920 2048 2248 2576  1080 1083 1088 1120 -hsync +vsync

stuart@stuart-virtual-machine ~ $ xrandr --addmode Virtual1 "1920x1080_60.00"

now the required resolution appears in display settings and I'm happy :-)

FPM - easy way to build packages

FPM ('flipping' rpm ;-) ) is a really cool way to build packages without the general grind of creating spec files etc .. heres my quick howto for CentOS:

• yum -y install rpm-build
• gem install fpm
• mkdir -p stupack/files stupack/etc stupack/logs (just making an arbitrary dir tree)
• touch stupack/files/test stupack/etc/stu.conf stupack/logs/stulog (dumping some dumb files there
• fpm -s dir -t rpm -n stupack stupack

              Created package {:path=>"stupack-1.0-1.x86_64.rpm"}

•  rpm -qlp ./stupack-1.0-1.x86_64.rpm 

            /stupack/etc/stu.conf

            /stupack/files/test

            /stupack/logs/stulog

yay - we have an rpm in 2 secs of easyness!

we can add  version number with a -v x.y.z to the fpm command

x86_64 was chosen for architecture - but could choose this as anything..

options: 

-t (rpm,deb,solaris..)

-s input type (directories,gem,rpm..)

-f force - overwrite existing

-n name to give to package

--license - name of lic for the package

--provides - what this pkg provides

--config-files - marks a file in package as a config file

overall - very nice, wish I'd had it when first tediously creating rpms!

How to use strace

How to use strace

really cool unix utility, lets us inspect what an executable is doing - dont need:

• source code
• knowledge of what  program the executable came from
• debugger

one big caveat to note is that strace stops/starts a process so can make it a lot, lot slower - e.g.:

[root@einstein ~]# time dd if=/dev/zero of=./test bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 11.0078 s, 95.3 MB/s

real    0m11.050s
user    0m0.001s
sys     0m0.377s
[root@einstein ~]#  sync; echo 3 > /proc/sys/vm/drop_caches
[root@einstein ~]# time strace -f -o out  dd if=/dev/zero of=./test bs=1024k count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 14.4345 s, 72.6 MB/s

real    0m14.491s
user    0m0.005s
sys     0m0.406s
[root@einstein ~]# 

so approximately 40% degradation here in performance - something you wouldnt want in prod.

How does a program interact with my computer?

so when a program runs in user mode it doesnt have direct access to the hardware (unless its something cool like solaflare network card or mellanox..).

For the program to get access to the hardware it needs to use a system call..(have a look at man 2 syscalls) - systems calls are essentially how a user program enters the kernel to perform a privileged task.

the categories of system calls are:

• process control (load/exec/abort/create/terminate/get/set attributes/wait/allocate/free memory)
• file management (create/delete/open/close/read/write/reposition/get-set attributes)
• device management (request/release/read/write/reposition/attach/detach)
• info maintenance (get/set time date/data/process file or dev attributes)
• communicate (create/del connection/send/rcv message/transfer status info/attach or detach remote dev)

examples

you dont need to be root to use strace - just have permission to read the process (i.e. generally you're own proc)

lots of output - try

# strace ls

following the output of this we'll see an :

• execve - (os starts process)
• brk(0) - kludge to read end of data segment
• open of filenames and assigning a filedescriptor number
• read using filedescriptor
• fstat details perm owner etc
• close

to see which files a program is opening we could use:

strace -e open ls

as a practical example - lets see what config files a program uses (e.g. if i type bash does it use profile/bashrc or bash_profile - I sometimes forget!):

strace -e open bash

/truncated output)

open("/home/sabramshumphries/.bashrc", O_RDONLY) = 3

open("/etc/bashrc", O_RDONLY)           = 3

yay - we can see the files it opens!!

useful flags:

-e just list the system calls specified - e.g. strace -e open,close

-f follow any subprocesses also that are created

-p - follow a process that started earlier

-o - write to a file so can look through output more easily

-s print out lines of length.. e.g. give a larger number so strace doesnt truncate output)

-c statistical summary of calls - shows output really nicely 

-t show timestamp (-tt even finer time)

-r relative time between calls

syscall                           function
open                             opens a file,returns a fd
close                             closes a filedescriptor
read                              reads bytes from a fd
write                             writes to a fd
fork                               creates a new process - cow of parent
exec                              executes a new program
brk                                extend heap pointer
mmap                           map a file to process address space
stat                                read info about a file
ioctl                               set io properties

What on earth is devops?

Devops is a pretty contentious word - lots of old school people say its nothing new and its just a label for automating everything - which people have tried (and are still trying!) to do.

Most people agree devops isn't:-

• a job title
• a job description
• an organisation/team

I've been guilty of all three of the above misdemeanours.

From wikipedia theres a pretty good definition of devops:

In traditional functionally separated organizations there is rarely cross-departmental integration of these functions with IT operations. devops promotes a set of processes and methods for thinking about communication and collaboration between development, QA, and IT operations.

So - what this is saying is essentially devops is about bringing systems admin/engineers and developers and testers closer together - rather than in silos. Hence why having a devops team is something I consider not a brilliant idea- we shouldnt be creating an extra silo - we should be bringing teams closer.

Technology isn't so important here  - the main thing is culture/behaviour - we also need to think more in terms of the overall toolchain rather than in terms of a devops tool, such as puppet/ansible..

• code - version control, frequent merges
• build - ci tools
• test - test code works (verify/validate) - performance
• package - stage software
• release (tricky  - always seen a lot of human interaction here) - change management/approvals
• configure - iaas
• monitor - is it all working!

so essentially a large and complex set of practises and tools to bring code quicker and more reliably to the customer.