Troubleshooting Networks – Layers 4-7 in the OSI Model

I was recently asked how to troubleshoot layers 4-7 of the OSI model.

OSI
Lower layers are drawn wider to show protocol encapsulation

Layer 4 (Transport) : Most problems at the transport layer have to do with blocked ports.  Ensure there are no firewalls (ex. iptables) blocking the TCP/UDP ports you’re trying to troubleshoot.  You can also try temporarily disabling quality of service (QoS).

Layer 5 (Session) and Layer 6 (Presentation) : Example protocols in these layers include sockets in the session layer and MIME in the presentation layer.  These two layers play a less active role in the functioning of the network compared to the other layers of the OSI model.  There usually isn’t anything here to troubleshoot.

Layer 7 (Application) : The app layer is where client-server apps are used.  For example, HTTP, HTTPS, SMTP, SSH, DNS.  Regarding DNS, use the dig or nslookup commands as a starting point to figuring out why DNS is failing.  For HTTP, you might use Apache’s or NGINX’s stats pages.  (Be sure to turn these off when you’re done using them though, for security.)  For SSH, SMTP, and all cases: check the logs.  Temporarily enable debug logging if you have to.  You can also use tcpdump to filter TCP/IP packets and analyze the protocols used.
There is certainly more that could be said here, but I just wanted to write down what I’ve learned so far.  Credit for much of the above info goes to:

Sukesh Mudrakola at Techgenix
Brendan Gregg’s book on Systems Performance

Origin of the Glory Be Doxology

One of the prayers I try to say every day is the Glory Be:

Glory be to the Father
and to the Son
and to the Holy Spirit,
as it was in the beginning
is now, and ever shall be
world without end. Amen.

This is a doxology, or a liturgical formula of praise to God.  The Glory Be has been used in a form similar to what we see above since around the year 529, but it actually comes straight from Sacred Scripture.  For example, see 1 Chron 29:11, Phil 4:20, and 2 Cor 13:14.

glory_be_prayer_800.jpg

From Catholic Answers: If at any point in our lives, in joy or in sorrow, in the middle of troubles or struggles, in hope and in fear, when perhaps we cannot find the words, we can always pray perfectly that God may be glorified always, and so we will be praying for all we really want. In a way, we will be praying as God “prays,” since the Savior prayed in the face of his deepest suffering, “Now, Father, glorify your Son with the glory he had before the world began”—that is, “as it was in the beginning, is now, and ever shall be, world without end. Amen.”

 

Troubleshooting Why My Mac Was Slow: photoanalysisd

I woke my Mac after it had been sleeping for a while and I noticed it was slower than usual.  Clicking browser tabs felt like pushing through molasses.

I opened a Terminal window and typed: top, and immediately1 saw photoanalysisd consuming 95% CPU.  It showed no signs of letting up, after several minutes of watching it.

  • What was it doing?
  • Wasn’t photo analysis only supposed to happen when your Mac is idle or in PowerNap/sleep mode?

I opened Photos.  Now two CPU hungry processes were going: Photos itself and photoanalysisd.

After a quick web search, I went straight to the first hit at Stack Exchange, and found this gem:

  1. Start Photos, let it continue past the first dialogue box;
  2. Now Preferences in the app menu is clickable (if it wasn’t before);
  3. Preferences > General , and untick both check boxes in Memories;
  4. Close Photos.
    This stops photoanalysisd cold, no reboot or kill required.
uncheck-photoanalysisd.png
Untick these two boxes in Photos > Preferences > General > Memories

Not sure how this guy figured this out, but this tip worked great.  It did indeed stop photoanalysisd.  Hopefully this trouble won’t crop up again.

Interpreting vmstat (virtual memory and system statistics)

I was asked about vmstat recently and didn’t know as much as I should.  I’m writing a brief post to help myself learn it a little more.

Here’s an excerpt of the manual:

vmstat.png

So what can you do with vmstat?  If you run it without specifying any parameters, it gives you an average view of virtual memory and system usage since your last reboot.

If you run it with the 1 parameter (vmstat 1), it runs (and re-measures) every 1 second, until you break out of it:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0  44712 110052 623096    0    0    30    28  217  888 13  3 83  1  0
 0  0      0  44408 110052 623096    0    0     0     0   88 1446 31  4 65  0  0
 0  0      0  44524 110052 623096    0    0     0     0   84  872 11  2 87  0  0

If you want to see how much memory is swapped in/out from disk, check out the si/so (swapped in/swapped out) columns.

If the amount of memory in the free column is low, you’ll want to check which processes are consuming the most memory.  (with top, for example)

But wait – you can also troubleshoot CPU and disk bottlenecks with vmstat:

For disk bottlenecks, look at the b (blocked) column.  This tells you the number of threads that were blocked/waiting on IO completion.  It should be 0 most of the time, but if it’s not, you can investigate further with iostat.

For CPU bottlenecks, look at the r (run queue) column.  These are the threads that were waiting for a CPU to become available in order to run.  If you see more than 2-5 times the number of CPUs on the system listed in this column, there may be a CPU bottleneck.

For more info, see The Geek Diary’s guide to troubleshooting with vmstat – it’s an excellent resource.  And if you want to know how to run something slightly similar to vmstat on macOS, try vm_stat.

Interpreting iostat Disk I/O Statistics

I was asked a few questions about iostat that I was unable to answer off the top of my head, so I decided to write down a few notes on it to help learn.

From the manual (on macOS) :

  • iostat displays kernel I/O stats on terminal, device, and cpu operations.
  • The first stats you see are averaged over the system uptime.
  • To get info about current activity, a suitable wait time (in seconds) should be specified (with -w), so that subsequent sets of stats will be averaged over that time.

Here is the output from running iostat on my primary disk with a wait time of 5 seconds. This was done while copying a 4GB file over 25 seconds:

%iostat -w 5 disk0
              disk0       cpu    load average
    KB/t  tps  MB/s  us sy id   1m   5m   15m
  261.14  372 94.88  29 14 58  2.53 2.32 2.35
  333.73  452 147.24  29 11 60  2.49 2.31 2.35
  395.37  406 156.82  27  8 65  2.45 2.31 2.35
  254.52  605 150.47  37 11 52  2.41 2.30 2.35
  409.59  391 156.26  25  6 69  2.46 2.31 2.35

KB/t: kilobytes per transfer
tps: transfers per second
MB/s: megabytes per second
us: % of cpu in user mode
sy: % of cpu in system mode
id: % of cpu in idle mode

The tps number is the I/O Operations Per Second, or IOPS. You can compare this to Wikipedia’s list of average IOPS for different storage devices.

My Mac’s SSD hit a high of 605 IOPS during the file copy, which is 3X higher than the fastest mechanical disks, but nowhere near as fast as some of the enterprise SSDs that you can buy.

iostat-examples.png

One note that I found interesting was this, from vaneyckt’s article on iostat at Coderwall:

[On Linux versions of iostat], some people put a lot of faith in the %iowait metric as an indicator for I/O performance. However, %iowait is first and foremost a CPU metric that measures the percentage of time the CPU is idle while waiting for an I/O operation to complete. This metric is heavily influenced by both your CPU speed and CPU load and is therefore easily misinterpreted.

For servers, you should be sending your iostat statistics to an internal data collection and graphing service, so you can get an idea of a baseline over time.  You can then try and correlate spikes in disk I/O with other data, such as slow web site performance, database queries, etc.

Jerky Animation in Safari Reader

I was trying to figure out why Safari’s Reader animation was smooth on my retina MacBook display (“1680×1050”), and super smooth on my old iPad Pro (1668 x 2224), but jerky/stuttering on my USB-C-connected external 4K monitor in any resolution except “1280×720”.  This even happened on simple web sites with very few photos.

safari-reader
Safari Reader Mode is just slow at 4K – if you don’t have a discrete video card. Image credit: Apple

Turns out it’s the VRAM.  From iCruiser7 on Reddit:

Pushing high-res external monitors primarily depends on VRAM size and bandwidth. Integrated graphics have to use system RAM as VRAM which is slower compared to dedicated VRAM. If you only had 8GB system RAM then the shared VRAM would further strain the entire system since less system RAM is available to apps.

Now, the new G7 graphics have more computational performance and are coupled with faster 3733 LPDDR4X system RAM which provides more bandwidth so performance on an external display should be at least somewhat better. However, a discrete graphics card would bring much more improvement. So if pushing high-res external monitors is your goal, I’d recommend you either 1. upgrade to a 16-inch MBP or 2. get a eGPU.

I looked into eGPU prices and they can be $300 for the enclosure alone.  And $700 for an Apple-recommended one with a video card included.  The cheapest MacBook Pro 16″ with an extra video card included is $2100!  I’m going to pass on that and see what external graphics performance is like on Apple Silicon.

Wish there was a way to turn Safari’s reader animation off.  If anyone knows how, please let me know.

Quick Fix for Slow Scrolling in macOS

On long web pages, especially when using a high resolution (4K+) external monitor, scrolling takes a long, long time.  Feels like you’re dragging the page through molasses.

I found a quick fix for this, buried in Accessibility preferences:

  1. Open System Preferences > Accessibility
  2. Scroll down to Pointer Control
  3. Hit Trackpad Options… and move the Scrolling speed slider to Fast.

trackpadoptions.pngWhy this isn’t in Trackpad preferences, I don’t know.  It used to be, back in 2011.  I guess Apple decided the Trackpad preferences were getting too overloaded and hid them deep in Accessibility settings.

 

Learning tcpdump

I was going to write a tcpdump tutorial to help myself learn it again.  Then I found Daniel Miessler already did it, with wonderful style and formatting.  Recommended.

tcpdump.png

What’s missing from Daniel’s tutorial is how to interpret the output depending on your situation.  Kind of hard to write, since there are virtually unlimited network trouble scenarios!  One place you might start is Netgate’s list of practical troubleshooting examples.  This shows how to troubleshoot port forwarding not working, IPsec tunnels not connecting, and outbound NAT configuration.

I also wanted to know how to diagnose dropped packets in tcpdump.  It simply prints a summary of dropped packets at the end.  (if you see this, you can try increasing the packet capture buffer size by passing the -B option to tcpdump)

I found there are many reasons for dropped packets, including but not limited to:

  • packets can go through hardware filtering, and still end up as not intended for the host (multicast)
  • NIC ring buffers can get full and be unable to cope with bursty traffic
  • CPUs receiving NIC interrupts can get too busy to process
  • Cable/hardware/duplex problems
  • NIC driver problems
  • MTU problems, jumbo frames, slightly oversized ethernet frames

Finally, Henry Van Styn at Linux Journal has a good guide on tcpdump-fu.  He writes:

“How much sense the output makes depends on how well you understand the protocols in question. tcpdump tailors its output to match the protocol(s) of the given packet. … There is no better way to learn how networks and protocols work than from watching their actual packets.

The Linux Boot Process, in Short

I need to be able to explain the Linux boot process at work, so I’m going to outline it here.  After the BIOS checks your hardware and lists bootable devices in the system, here’s what’s loaded:

  1. Boot Loader: Presents you with a list of operating systems or kernel configurations to load.  (Example: Master Boot Record + GRUB)
  2. Kernel: The selected kernel, if compressed, decompresses itself.  start_kernel() is called, which sets up interrupts, memory management, and device drivers.  Then the idle process and scheduler are started.
  3. Init: Establishes and operates the user environment.  A series of scripts or config files are executed by systemd or upstart.  User space services are started.  For example, networking, a web server, and database services are started in this phase.
linuxbootprocess
Linux Boot Process (Image Credit: developer.ibm.com)

That’s the core of it.  A few notes:

  • You may not actually need a boot loader if you have a UEFI system where the Linux kernel payload can be executed directly.
  • After init is complete, if you prefer to run in graphical mode, a display and login manager are started, and the session manager starts a session after you login.
  • On shutdown, init is called to terminate user space services.  It then makes a system call to the kernel to shut the system down.

If you want to learn more about how Linux boots, I recommend Yogesh Babar’s book Hands-on Booting, which goes into the process in depth, along with troubleshooting  steps for when it fails to boot.  You can also check out the Wikipedia entries linked above.

Apple Watch Series 3 on Sale for $169 at Amazon

Amazon just listed the Apple Watch Series 3 for $169 – on sale.  This is a great starter watch for anyone interested in dipping their toe in the waters with Apple’s tech.  (They will need an iPhone to use it, though.)  Series 3 runs smoothly with watchOS 6, and will be supported by the upcoming watchOS 7.  (which includes sleep tracking)

apple-watch-series-3.jpg

These watches actually come with blood oxygen detection built in, but it isn’t enabled yet due to FDA restrictions.  Once the FDA eases their rules, this will be great for early detection of COVID-19-like symptoms.