Interpreting iostat Disk I/O Statistics

I was asked a few questions about iostat that I was unable to answer off the top of my head, so I decided to write down a few notes on it to help learn.

From the manual (on macOS) :

  • iostat displays kernel I/O stats on terminal, device, and cpu operations.
  • The first stats you see are averaged over the system uptime.
  • To get info about current activity, a suitable wait time (in seconds) should be specified (with -w), so that subsequent sets of stats will be averaged over that time.

Here is the output from running iostat on my primary disk with a wait time of 5 seconds. This was done while copying a 4GB file over 25 seconds:

%iostat -w 5 disk0
              disk0       cpu    load average
    KB/t  tps  MB/s  us sy id   1m   5m   15m
  261.14  372 94.88  29 14 58  2.53 2.32 2.35
  333.73  452 147.24  29 11 60  2.49 2.31 2.35
  395.37  406 156.82  27  8 65  2.45 2.31 2.35
  254.52  605 150.47  37 11 52  2.41 2.30 2.35
  409.59  391 156.26  25  6 69  2.46 2.31 2.35

KB/t: kilobytes per transfer
tps: transfers per second
MB/s: megabytes per second
us: % of cpu in user mode
sy: % of cpu in system mode
id: % of cpu in idle mode

The tps number is the I/O Operations Per Second, or IOPS. You can compare this to Wikipedia’s list of average IOPS for different storage devices.

My Mac’s SSD hit a high of 605 IOPS during the file copy, which is 3X higher than the fastest mechanical disks, but nowhere near as fast as some of the enterprise SSDs that you can buy.

iostat-examples.png

One note that I found interesting was this, from vaneyckt’s article on iostat at Coderwall:

[On Linux versions of iostat], some people put a lot of faith in the %iowait metric as an indicator for I/O performance. However, %iowait is first and foremost a CPU metric that measures the percentage of time the CPU is idle while waiting for an I/O operation to complete. This metric is heavily influenced by both your CPU speed and CPU load and is therefore easily misinterpreted.

For servers, you should be sending your iostat statistics to an internal data collection and graphing service, so you can get an idea of a baseline over time.  You can then try and correlate spikes in disk I/O with other data, such as slow web site performance, database queries, etc.

Jerky Animation in Safari Reader

I was trying to figure out why Safari’s Reader animation was smooth on my retina MacBook display (“1680×1050”), and super smooth on my old iPad Pro (1668 x 2224), but jerky/stuttering on my USB-C-connected external 4K monitor in any resolution except “1280×720”.  This even happened on simple web sites with very few photos.

safari-reader
Safari Reader Mode is just slow at 4K – if you don’t have a discrete video card. Image credit: Apple

Turns out it’s the VRAM.  From iCruiser7 on Reddit:

Pushing high-res external monitors primarily depends on VRAM size and bandwidth. Integrated graphics have to use system RAM as VRAM which is slower compared to dedicated VRAM. If you only had 8GB system RAM then the shared VRAM would further strain the entire system since less system RAM is available to apps.

Now, the new G7 graphics have more computational performance and are coupled with faster 3733 LPDDR4X system RAM which provides more bandwidth so performance on an external display should be at least somewhat better. However, a discrete graphics card would bring much more improvement. So if pushing high-res external monitors is your goal, I’d recommend you either 1. upgrade to a 16-inch MBP or 2. get a eGPU.

I looked into eGPU prices and they can be $300 for the enclosure alone.  And $700 for an Apple-recommended one with a video card included.  The cheapest MacBook Pro 16″ with an extra video card included is $2100!  I’m going to pass on that and see what external graphics performance is like on Apple Silicon.

Wish there was a way to turn Safari’s reader animation off.  If anyone knows how, please let me know.

Quick Fix for Slow Scrolling in macOS

On long web pages, especially when using a high resolution (4K+) external monitor, scrolling takes a long, long time.  Feels like you’re dragging the page through molasses.

I found a quick fix for this, buried in Accessibility preferences:

  1. Open System Preferences > Accessibility
  2. Scroll down to Pointer Control
  3. Hit Trackpad Options… and move the Scrolling speed slider to Fast.

trackpadoptions.pngWhy this isn’t in Trackpad preferences, I don’t know.  It used to be, back in 2011.  I guess Apple decided the Trackpad preferences were getting too overloaded and hid them deep in Accessibility settings.

 

Learning tcpdump

I was going to write a tcpdump tutorial to help myself learn it again.  Then I found Daniel Miessler already did it, with wonderful style and formatting.  Recommended.

tcpdump.png

What’s missing from Daniel’s tutorial is how to interpret the output depending on your situation.  Kind of hard to write, since there are virtually unlimited network trouble scenarios!  One place you might start is Netgate’s list of practical troubleshooting examples.  This shows how to troubleshoot port forwarding not working, IPsec tunnels not connecting, and outbound NAT configuration.

I also wanted to know how to diagnose dropped packets in tcpdump.  It simply prints a summary of dropped packets at the end.  (if you see this, you can try increasing the packet capture buffer size by passing the -B option to tcpdump)

I found there are many reasons for dropped packets, including but not limited to:

  • packets can go through hardware filtering, and still end up as not intended for the host (multicast)
  • NIC ring buffers can get full and be unable to cope with bursty traffic
  • CPUs receiving NIC interrupts can get too busy to process
  • Cable/hardware/duplex problems
  • NIC driver problems
  • MTU problems, jumbo frames, slightly oversized ethernet frames

Finally, Henry Van Styn at Linux Journal has a good guide on tcpdump-fu.  He writes:

“How much sense the output makes depends on how well you understand the protocols in question. tcpdump tailors its output to match the protocol(s) of the given packet. … There is no better way to learn how networks and protocols work than from watching their actual packets.

The Linux Boot Process, in Short

I need to be able to explain the Linux boot process at work, so I’m going to outline it here.  After the BIOS checks your hardware and lists bootable devices in the system, here’s what’s loaded:

  1. Boot Loader: Presents you with a list of operating systems or kernel configurations to load.  (Example: Master Boot Record + GRUB)
  2. Kernel: The selected kernel, if compressed, decompresses itself.  start_kernel() is called, which sets up interrupts, memory management, and device drivers.  Then the idle process and scheduler are started.
  3. Init: Establishes and operates the user environment.  A series of scripts or config files are executed by systemd or upstart.  User space services are started.  For example, networking, a web server, and database services are started in this phase.
linuxbootprocess
Linux Boot Process (Image Credit: developer.ibm.com)

That’s the core of it.  A few notes:

  • You may not actually need a boot loader if you have a UEFI system where the Linux kernel payload can be executed directly.
  • After init is complete, if you prefer to run in graphical mode, a display and login manager are started, and the session manager starts a session after you login.
  • On shutdown, init is called to terminate user space services.  It then makes a system call to the kernel to shut the system down.

If you want to learn more about how Linux boots, I recommend Yogesh Babar’s book Hands-on Booting, which goes into the process in depth, along with troubleshooting  steps for when it fails to boot.  You can also check out the Wikipedia entries linked above.

Apple Watch Series 3 on Sale for $169 at Amazon

Amazon just listed the Apple Watch Series 3 for $169 – on sale.  This is a great starter watch for anyone interested in dipping their toe in the waters with Apple’s tech.  (They will need an iPhone to use it, though.)  Series 3 runs smoothly with watchOS 6, and will be supported by the upcoming watchOS 7.  (which includes sleep tracking)

apple-watch-series-3.jpg

These watches actually come with blood oxygen detection built in, but it isn’t enabled yet due to FDA restrictions.  Once the FDA eases their rules, this will be great for early detection of COVID-19-like symptoms.

 

MacOS’s Hidden Storage Management App

When you think you need Daisy Disk or Grand Perspective to clean up your disk, you might be surprised Apple has built a pretty good storage management app into the operating system:

  1. Open Spotlight (Cmd-Space)
  2. Type “Storage Management”
  3. On the bottom, in the Reduce Clutter section, hit Review Files, and you’ll see something that looks like this:

macos-catalina-storage-management-documents-large-files-2.jpg

(This is also shown in Apple’s support article on freeing up storage space on your Mac.)

If this isn’t enough for you, or you prefer the command line, you can also run brew install ncdu.  (after installing Homebrew)  This installs the venerable (and still useful) Ncurses Disk Usage utility.

How to Compare Prices Between Doctors (in the United States)

It’s difficult to compare prices between doctors.  Insurance providers have estimator tools, but the prices they give are often for slightly different procedures than the one you’re looking for.  Even if they do have your exact procedure/test listed, the price ranges they list are usually outdated and nowhere near accurate.

One of my colleagues at work had a method he used to get more accurate price estimates.  I’ve slightly modified it to fit my needs, and am listing it here in case it helps someone else:

  1. Call doctor and get diagnosis and/or procedure/CPT codes.
  2. Call facilities, get routed to billing, and ask for estimates.
  3. Call insurance and also ask for estimates.
  4. Pick facility and schedule an appointment.  This may involve calling the doctor back to get a referral or move a referral.
  5. Once scheduled, call again and ask for an estimate (since facilities are sometimes better able to provide estimates once your visit is scheduled.)

This is a lot of work, but may save you hundreds or even thousands of dollars.

cpt-professional-2020
CPT Professional 2020 (CPT / Current Procedural Terminology (Professional Edition)) – You don’t actually need this book to figure out what CPT code you need to reference, but the person in your doctor’s billing office probably will.

Also:

  • Even if you think you might not be eligible, apply for financial aid anyway.  Especially if you have a family.  You might be surprised that you qualify.  And apply before you pay your bill.  It’s much tougher to get retroactive financial aid for a bill you previously struggled to pay.
  • Ask what the uninsured cost is.  It can be cheaper than what would normally billed to your insurance, but then it doesn’t count towards your deductible either.

 

iPad: 90% of a Laptop

I tried the iPad-only life for months.  Here’s what I found:  The iPad gets you about 90% of the way to a laptop.  It’s the last 10% of what you might need that is frustrating.  For example:

  • You can listen to music, but if you want to easily trim that media so you only hear the best parts, the Music app on macOS is what you need.
  • Even if Safari identifies itself as a desktop browser, there are still sites that don’t fully work with it.
  • I had a PDF that was attached to an email that I was unable to rotate on the iPad without a paid 3rd party app such as GoodReader.
  • Another email attachment I wasn’t able to see in iPad Mail until I went to the cloud email portal (which barely worked on the iPad).
  • Tax software on the iPad is even more clunky, expensive, and data-loss prone than on macOS.
  • I was unable to change the text size of data entered in PDF form fields, thus making a printout of the form unreadable. Had to use a Mac.
  • My wife used to love her iPad Pro (it was a lot better than the Windows notebook  that made her break down in tears), but now she’s selling it because she outgrew Swift Playgrounds.  She wants to make a real app in Xcode.

You may say now that you don’t need to do these things, that these are all edge cases, but you will run into your own edge cases where the iPad falls down and get frustrated. Maybe in a few years, the software will be better.  But I suspect the Mac will be best for most use cases for many years to come.

apple-magic-keyboard-for-ipad-pro-1-1280x720
Apple’s Magic Keyboard for iPad Pro (Image: SlashGear)

What I Learned From Craig Federighi’s Talk at Berkeley

Craig Federighi, Apple’s senior vice president of software engineering, recently gave a great talk at Berkeley.

Craig Federighi-967236574
Image: Wired

Here’s what I learned from Craig:

  1. Do what you love!
  2. Work with people whose work you admire.
  3. Pay attention.  (use a notebook and take handwritten notes)
  4. Never stop acting like the new one on the team.  Ask stupid questions.
  5. Team > Self.  Do all that you can do to help the mission of the team.
  6. Commit, Focus, Reassess.  Set yourself a deadline, say 1-4 years from now, to assess where you’re at and figure out what you want to do next.
  7. Follow your heart.  Trust your gut feelings on big and small decisions.

Beyond these points, he also mentioned:

  • Expertise comes with solving real problems.  (my own example: HackerRank vs. c0d3.com)
  • Become good at communicating, both via the written word and verbally.  Craig’s early experience being a DJ at a pirate (?) radio station helped him with this.
  • Don’t read from a script.  If there’s something you’re excited about, just tell that story.  Know what you want to get across.  Be yourself and tell the story.
  • Put yourself in the mindset of the customer.  Translate their needs to the team.
  • Get the right amount of sleep.
  • Avoid burnout by having a diversity of problems that you’re working on.
  • Have a hunger to learn.  Come up with a project and constantly be learning.
  • Focus your main efforts on helping solve the right problems for the team.  If you have a broader concern for the project, you can help manage it.

I loved listening to what Craig had to say and will use many of these tips going forward.