Zac Blanco

Creating a Swap File on a Ramdisk with Ubuntu 20.04

Fri, 13 Aug 2021 00:00:00 +0000

For some recent work I was looking to negate the latency of storage accesses of swapping in order to more easily measure the kernel’s swap system overhead without worrying about the latency of the underlying block device.

Naively, I initially tried with mount -t ramfs /mnt/ramdisk and simply tried creating a swap file on that mount, but after trying something to the tune of swapon /mnt/ramdisk/swap.img I was met with an Operation not supported. Some google-fu led me to the answer. tmpfs and ramfs don’t support the APIs required to act as a device to read and write a swap image.

However, it is possible (at least if you’re on Ubuntu 20.04 Server) to use the brd kernel module to create a block device at /dev/ram0 which can then be formatted with a filesystem swap file that can be used as a ramdisk. Here is the script:

#!/usr/bin/env bash

# create the mount point directory
sudo mkdir /mnt/ramdisk
# insert kernel module for /dev/ram0 and allocate 2GiB for swapfile
sudo modprobe -C brd.conf brd
# Format our ram device
sudo dd if=/dev/zero of=/dev/ram0 bs=1M count=2048
# create ext4 filesystem and mount (2GiB)
sudo mkfs.ext4 -b 4096 /dev/ram0 536870912
# mount the block device
sudo mount /dev/ram0 /mnt/ramdisk
# create swapfile
sudo dd if=/dev/zero of=/mnt/ramdisk/swap.img bs=1M count=2048
# setup permissions and enable the swap file
sudo chown root:root /mnt/ramdisk/swap.img
sudo chmod 600 /mnt/ramdisk/swap.img
sudo mkswap /mnt/ramdisk/swap.img
sudo swapon /mnt/ramdisk/swap.img
swapon -s

And the file for brd.conf for the modprobe command contains parameters to ensure the ramdisk is at least 2GiB (the desired size for my swap file):

options brd rd_size=2147483648
options brd rd_nr=1

Frankly, the latency using a ramdisk directly here was pretty disappointing compared to an NVMe SSD, but I will potentially explore that issue in a later post.

Tracing 1 functions for "read_pages"... Hit Ctrl-C to end.

     usecs               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 11       |                                        |
         8 -> 15         : 61       |                                        |
        16 -> 31         : 1243     |**                                      |
        32 -> 63         : 8108     |***************                         |
        64 -> 127        : 20933    |****************************************|
       128 -> 255        : 17929    |**********************************      |
       256 -> 511        : 11953    |**********************                  |
       512 -> 1023       : 6        |                                        |
      1024 -> 2047       : 1        |                                        |

Migrating Ubuntu 20.04 with LVM Partitioning

Sun, 06 Jun 2021 00:00:00 +0000

We were running some experiments on a local cluster of machines that were originally installed and configured using the Ubuntu 20.04 Server live CD. After brief investigation into some performance issues we found that the operating system (and rest of the system software) was installed on a spinning drive with performance less than 1/5 of SSD on the system.

We didn’t want to re-install the OS and software because there was a lot of configuration we had to do to the operating system due to some specialized hardware we were using. So, we decided to opt for a disk migration instead. I had done some OS migrations years ago, so I figured moving the OS partition for Ubuntu should be straightforward. I’ll preface the rest of the post saying that the experience wasn’t unpleasant but there was little documentation that I could find on the internet related to newer versions of Ubuntu. Everything seemed to be for 12.04, 14.04, etc.

I’m writing this post to hopefully help out some other poor lost souls who might need to migrate their operating system installs from one disk to another (and save some time in the process) that were installed with an LVM partitioning scheme.

Setup

OS initially installed on a 1TiB drive (/dev/sdb)
Unpartitioned/formatted 512GiB SSD (/dev/sda)
Ubuntu 20.04 installer used with all default settings

The Ubuntu OS installer creates 3 partitions when installing the OS to a single disk and when the LVM partition install is used

/dev/sd*1 –> mounts to /boot/efi
/dev/sd*2 –> mounts to /boot
/dev/sd*3 –> mounts to /

The goal is to completely move the boot and data partitions from /dev/sdb to /dev/sda and then finally be able to boot with /dev/sdb completely wiped and formatted.

Instructions

I used the gparted live CD image to boot into the OS. Once booted with gparted:

Make sure all partitions on the target drive (SSD) are removed and the disk is entirely blank
use gparted to clone the partions mapped to /boot and /boot/efi to their respective locations. In other words copy+paste /dev/sdb1 to /dev/sda1 and /dev/sdb2 to /dev/sda2
Shrink the lvm partition so that it can fit on the new disk
- right click the LVM partition -> deactivate. Then, shrink the partition so that it can fit on the new drive, assuming that your target drive has enough free space.

These boot partitions and resizing should be quick. Next, we need to move the main data partition. Unfortunately, Gparted doesn’t support the copy+paste of LVM-based partitions, so we need to use the command-line. I followed some instructions from askubuntu in order to create a new LVM partition and move the data to the SSD.

# We need to create a new partition on the new SSD
# Create the new partition in the shell with "n" using the defaults.
# "w" to write
fdisk /dev/sdNEW

# Once the new partition is created, add it to the logical volume,
# move the data, and remove the old one. "ubuntu-vg" is the volume
# group name that the installer gave to the LVM volume
pvcreate /dev/sdNEW3
vgextend ubuntu-vg /dev/sdNEW3
pvmove /dev/sdOLD3
vgreduce ubuntu-vg /dev/sdOLD3

Finally, use gparted to remove the boot flag from the partition on the old drive and add the boot flag to the copy of that partition on the new drive. After all these steps you can reboot without the gparted live CD.

At this point, the OS should be able to reboot into the OS successfully. However we never deleted or reconfigured the boot process, so if you run lsblk I saw that the HDD /boot and /boot/efi partitions were still mounted.

$ lsblk
NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                         8:0    0 447.1G  0 disk
├─sda1                      8:1    0   512M  0 part
├─sda2                      8:2    0     1G  0 part
└─sda3                      8:3    0 445.6G  0 part
  └─ubuntu--vg-ubuntu--lv 253:0    0   200G  0 lvm  /
sdb                         8:16   0 931.5G  0 disk
├─sdb1                      8:1    0   512M  0 part /boot/efi
├─sdb2                      8:2    0     1G  0 part /boot
└─sdb3                      8:3    0 445.6G  0 part
sdb                         8:16   0 931.5G  0 disk

To fix this, unmount and delete the partitions, then mount the new partitions (they should contain the same data) and then run grub-install and update-grub

sudo umount /boot/efi
sudo umount /boot
# remove the partitions with "d" and finally a "w"
sudo fdisk /dev/sdOLD

sudo mount /dev/sdNEW2 /boot
sudo mount /dev/sdNEW1 /boot/efi
sudo grub-install
sudo update-grub

After running all these steps and a reboot, our lsblk shows the correct output, and a free HDD that we can format again use to store more data.

$ lsblk
NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                         8:0    0 447.1G  0 disk
├─sda1                      8:1    0   512M  0 part /boot/efi
├─sda2                      8:2    0     1G  0 part /boot
└─sda3                      8:3    0 445.6G  0 part
  └─ubuntu--vg-ubuntu--lv 253:0    0   200G  0 lvm  /
sdb                         8:16   0 931.5G  0 disk

Two Years of Reflection

Wed, 30 Sep 2020 00:00:00 +0000

One of the things I learned about during my time as a learning assistant at Rutgers University assisting in classes and teaching in classes is the term metacognition. A page on Vanderbilt University’s website describes metacognition as

the processes used to plan, monitor, and assess one’s understanding and performance¹

This was one of the crucial ideas that we were taught to consider and understand when guiding our peers through the vast web of knowledge in each class. From what I remember being told, those students who succeeded in classes were usually better at thinking about their thinking. Or, in a sense, reflecting on what they’ve learned, and how to integrate it into their current understanding of the their respective subjects. By engaging in metacognition students can help improve recall, fix misconceptions, and improve the speed at which they can solve problems in the specific domain. It also helps identify strengths and weaknesses in the subject matter which can drive the areas a student may study or practice.

Having just recently finished two years in the workforce, and about to enter a PhD program where I will most likely be conducting a whole lot of learning, I think now is a good time to begin to reflect on what has been accomplished over the last few years. It is also where I could look back on where I could have made better judgement and improved on the products that I delivered. Also, I want to reflect on my time in my company as a whole, and where I think the company made missteps and where it earned my respect. Hopefully, trying to acknowledge these parts will enable me to find areas where I can improve as an engineer as well as help me identify organizations and aspects of which that I should look for when seeking work in the future; skills that I’m sure will be integral in helping me reach the end of this degree program and beyond.

For the rest of this post, I’m going to highlight some of the areas I feel that I struggle with, and then research and write about things I can do to improve myself in those areas, along with some plans of action that I can take

Design and Deliverables

One area where I struggled before, and still struggle is clearly communicating technical design verbally to others. I tend to rely heavily on depictions and diagrams to get my points across to others. I find that during explanations, I find myself moving from one point to another somewhat erratically. Unfortunately, I felt that there wasn’t many opportunities to improve on this over the last few years as there were only a few projects that required a thorough design which needed to be communicated and approved before implementing.

Personally, I feel it’s okay to lean on diagrams to help explain technical concepts, but in the future I need to find a better way to make sure my explanations logically flow from one point to the next, using the diagram more as a guide, rather than leaning on it to do the explaining. This is one skill that surely applies across a multitude of areas such as technical interviews, research presentations, and engineer.

Searching around the internet, I found a few places where some say that diagrams (or whiteboards) are fine for communicating technical information². I think whiteboards and meetings are fine for conveying architecture and communication - but I feel they are best used for fleshing out early ideas when collaborating with co-workers. Final designs should be able to be described in technical documentation that can be accessed and read by others with less context, and not require someone else present to explain points in the document. Apparently, there is a standard - IEEE 1471³ which describes how to communicate and document software architecture. Reading further on the IEEE website, this document has actually been superseded by IEEE 42010-2011⁴. Both documents I’m sure provide valuable guidelines for communicating system architectures, so I will be sure to read them to hopefully apply them to my practice.

Project Management

After experiencing the last few years, I think its crucial that companies, especially startups, justify the work they are doing by having a clear GTM (go-to market) strategy⁵ for their products. Having a clear strategy not only helps make it clear to the investors of a company how the product will succeed, but also the employees as well. Small startup companies are inherently risky - every employee of the company is usually putting something down on the line and will want to know how their work contributes to the company’s success.

To some extent, you should always listen your direct manager’s directions. They should be able to justify why or how the work you are doing fits in to the bigger picture. If the company has a clear GTM strategy, then your manager should be able to give you a straightforward answer to what company goals your work is contributing to. If they aren’t able to do that, then there’s probably an issue with your manager, or the company. I think personally, I could have done a better job asking the right question as to where some of my work would fit in and how it would be used by customers.

Now, for small tasks (1-4 days of time) I would argue that it’s probably not as necessary to understand the motivation behind each task. However, if you know you’re going to be involved on a project for an extended period of time (1 -3 months or more), then I think you should be able to hold your manager accountable for being able to explain where the project fits into the bigger picture. If they aren’t able to do so, then maybe it is time to start re-evaluating your current position, team, or even company. Management should at minimum be able to justify their decisions for the direction and assignment of projects.

To be clear, startups pivot for a vast number of reasons, and sometimes the management might not have the clearest picture about what the next steps are for the company. However, if the company tries to hide the fact that it is struggling, or feeds lies about progress, then I think that’s a clear warning sign when things are bad. If your manager(s) are at least honest about what is happening, then you should be able to trust in them to do the right thing. In the future, I’m going to try to do a better job asking the right questions about my assignments, as well as team direction.

Summary

There are surely a multitude of other areas that I could have performed better in. The two that discussed today are things that stuck out starkly to me a couple of times over the last few years. I want to be able to make permanent improvements to my ability to work in an industry setting that will help me improve as an engineer and as an employee. Singling out and writing about them along with giving myself concrete actions to take to improve I hope will improve my prospects for the future.

JVM Pauses - It's more than GC

Thu, 28 May 2020 00:00:00 +0000

Background
Safepoint Operations
Analyzing Safepoint Pauses
Conclusion
Notes

Background

Everyone knows and loves the good-ol’ JVM GC pause. New GC algorithms have been introduced to improve GC latencies or prevent the need to pause altogether. However, that’s not the end of the story for JVM pauses. Unfortunately, there are a variety of other housekeeping tasks within the JVM that can force it to stop executing user code¹.

Safepoint Operations

First, it is important to understand the concept of a safepoint within the JVM. A safepoint is a point in program execution where the state of the program is known and can be examined. Things like registers, memory, etc. For the JVM to completely pause and run tasks (such as GC), all threads must come to a safepoint.

For example, to retrieve a stack trace on a thread we must come to a safepoint. This also means tools like jstack require that all threads of the program be able to reach a safepoint.

To be more precise about what exactly a safepoint is, we refer to the HotSpot Glossary which defines a safepoint as:

A point during program execution at which all GC roots are known and all heap object contents are consistent. From a global point of view, all threads must block at a safepoint before the GC can run. (As a special case, threads running JNI code can continue to run, because they use only handles. During a safepoint they must block instead of loading the contents of the handle.) From a local point of view, a safepoint is a distinguished point in a block of code where the executing thread may block for the GC. Most call sites qualify as safepoints. There are strong invariants which hold true at every safepoint, which may be disregarded at non-safepoints. Both compiled Java code and C/C++ code be optimized between safepoints, but less so across safepoints. The JIT compiler emits a GC map at each safepoint. C/C++ code in the VM uses stylized macro-based conventions (e.g., TRAPS) to mark potential safepoints.

While GC is one of the most common safepoint operations, there are many VM operations² that are run while threads are at safepoints. Some may be invoked externally by connecting to the HotSpot JVM (i.e. jstack, jcmd) while others are internal to the JVM operation (monitor deflation, code deoptimization). A list of common operations is below.

User Invoked:
- Deadlock detection
- JVMTI
- Thread Dumps
Run at regular intervals (see -XX:GuaranteedSafepointInterval³)
- Monitor Deflation
- Inline Cache Cleaning
- Invocation Counter Delay
- Compiled Code Marking
Other:
- Revoking Biased Locks
- Compiled method Deoptimization
- GC

All these operations force the JVM to come to a safepoint in order to run some kind of VM operation. Now we can decompose a safepoint operation times into two categories.

Time taken from initiating the safepoint request until all threads reach a safepoint
Time taken to perform safepoint operation.

You can view this behavior in a JVM by adding the -XX:+PrintGCApplicationStoppedTime and -XX:+PrintGCDetails⁴ flags. This will print messages to the GC logs that look like:

Total time for which application threads were stopped: 0.0572646 seconds, Stopping threads took: 0.0000088 seconds

The 0.0572646 seconds is the sum total time that threads may have been stopped for. This includes the blue and red sections above. The 0.0000088 seconds is the time spent in the blue section, where it took that much time to stop all threads at a safepoint. Even though the JVM flag is named -XX:+PrintGCApplicationStoppedTime⁵ it actually logs all JVM pauses, not just GC.

So, if applications are not responding, it may be because

The JVM is trying to reach a safepoint and most threads have already stopped except maybe one or two, or
The JVM has reached a safepoint and is running some internal operations. May it be GC, lock bias revocation, cache line invalidation, etc.

The issue is that we need to figure out exactly what is triggering the pause in the first place if anything, and then investigate which part of the pause is taking a long time; the time to get to the safepoint (TTSP), or the time spent performing the VM operation.

To do that more logging is required. The flags that need to be added to the JVM are -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1. Adding these two arguments will print to stdout or the configured log file every time a safepoint operation occurs. As of JDK 8 the log output consists of two lines and looks like the following (times are in ms):

  vmop [threads: total initially_running wait_to_block] [time: spin block sync cleanup vmop] page_trap_count
155623.484: RevokeBias [ 1595 0 0 ] [ 0 0 127 7 0 ]  0  

Breaking down the output:

155623.484: The time in seconds since the JVM started
RevokeBias (vmop): The VM operation running during this safepoint.
1595 (total): The total number of threads in the VM when the safepoint was requested
0 (initially running): The number of threads running at the beginning of the safepoint operation
0 (wait to block): The number of threads blocked when the VM began the operation execution
0 (spin): The time spent spinning waiting for threads to reach a safepoint
0 (block): The time spent waiting for threads to block for the safepoint
127 (sync): The total time spent synchronizing the threads for the safepoint. This is a combination of spin + block + “other activity”. This value is what I regard as the “time to safepoint” or TTSP. In the GC logs, the TTSP or “Time stopping threads” typically only includes the spin + block times based on my analysis.
7 (cleanup): Time spent on internal VM cleanup activities
0 (vmop): The time spent on the actual safepoint operation (RevokeBias). This can be any safepoint operation.
0 (page_trap_count): Page trap count

It’s important to reiterate that the spin, block, and sync times represent portions of the TTSP. So, if TTSP is large it can mean that one thread might be attempting to finish its work, while the rest of the JVM threads are paused waiting for it to reach a safepoint. This is why the total pause time of a JVM must be considered TTSP + cleanup + vmop.

With this information we can handily take any JVM logs and figure out which operations were running. It’s critical to consider both the safepoint logs and GC logs. Otherwise it’s possible to miss information about TTSP mentioned above.

Analyzing Safepoint Pauses

Now that we know all about safepoints and how to get their statistics, we need to know what can prevent Java threads from coming to a safepoint. Some of those causes are¹:

Large object initialization
- i.e. initializing a 10GB array. (Single threaded, zeroing the array)
Array copying
JNI Handle Allocation
JNI Critical Regions
Counted Loops
NIO Mapped Byte Buffers
- Memory mapped portion of a file

Usually, if a program is taking a long time to reach a safepoint there is a systemic issue in the code where it performs one or more of the operations above for extended periods of time without allowing the JVM to come to a safepoint.

Fortunately, there are even more options that can be added to the JVM in order to enable logging when it takes a longer than expected time to reach a safepoint⁶.

-XX:+SafepointTimeout -XX:SafepointTimeoutDelay=<timeout in ms>

These two options print to the VM log / stdout all threads which have failed to reach a safepoint after the specified time period. This can help developers troubleshoot which threads might be causing extended pauses of the JVM and whether the root cause is the VM operation or the TTSP.

Next, we’ll do a real analysis on an example of a large TTSP and what the JVM and GC logs will look like. The data below was collected from a long running java server application.

Take the following graph of a GC log from the “Time Application Threads were Stopped” and the “Time Stopping Application Threads”:

The graph above plots the GC log of a worker. It shows a few spikes with a max of ~24 seconds of pause time reported by the JVM. For each spike most of the thread pause time was actually spent stopping the threads, not performing the operations! Notice the correlation of spikes in “Threads waiting to stop” vs “Threads stopped time”. It’s clear something is amiss.

Now, let’s take a look at the safepoint statistics plotted on the same timeline. Each statistic is represented individually on its own graph.

This graph shows the times for each safepoint statistic. Remember the sync time is generally equal to the TTSP (spin + block + “other”), but compare the y-axis scale versus the graph above.

This graph shows spikes correlated at the exact same times as some of the GC pauses on the graph above, and even more frequently than as reported by the GC log output. Additionally, the GC pause times reported peaked at a measly 24s. The values for this graph (under the “sync” operation) show spikes of over 200000ms, or nearly 5 minutes. Notice that the graph for “Op Time” (vmop) has a scale of only 500ms meaning many of the actual VM operations were short-lived.

These graphs show that the “Time Stopping Application Threads” reported in GC logs should not always be trusted. The safepoint statistics are more fine grained, but provide more information about how the application behaves. The reason we didn’t catch issues in the application initially is because the GC log wasn’t providing the full picture on how long the application was actually stopped for. The log only reported ~24 seconds while in reality it was many minutes of time; unacceptable for any application which is expected to serve data in large quantities and low latencies.

Conclusion

Hopefully with this knowledge in-hand it will be easier to detect, identify, and respond promptly when an application begins to fail due to a JVM pause. The most helpful JVM flags to gather info are:

-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDetails
-XX:+PrintSafepointStatistics
-XX:PrintSafepointStatisticsCount=1
-XX:+SafepointTimeout
-XX:SafepointTimeoutDelay=<ms before timeout log is printed>

By analyzing the GC log and safepoint statistics together it is much easier to get a holistic view of how the JVM performs with respect to pause times. Usually, this information is enough to identify a cause or point in the right direction of a culprit.

Notes

There is a highly informative talk given by an Azul (previously Oracle JDK) engineer explaining the reasons, besides GC, that can pause the JVM. Much of the information in this post comes from that video. ↩ ↩²
All Possible VM Operations in JDK 8 ↩
This reference application containing all JVM options is very useful ↩
There is an additional flag which can may be of use: -XX:+PrintGCApplicationConcurrentTime. ↩
There is another flag: -XX:+PrintGCApplicationConcurrentTime that will print the opposite; The time the application spends running between pauses, every time a pause occurs. ↩
These guys really thought ahead didn’t they? There’s also a -XX:+TraceSafepointCleanupTime which gives more detail about time spent on cleanup operations during the safepoint. ↩

Ad-Hoc Networks and Static ARP entries on Raspberry Pi's

Fri, 17 Mar 2017 00:00:00 +0000

Part of our research this year involves creating and using ad-hoc networks on a linux-based operating system (Raspbian based on Debian Jessie).

The hardware we’re working with:

5x Raspberry Pi 2 Model B+
2x Raspberry Pi 3

Unfortunately ad-hoc network support and implementation seems to be lacking. We’ve run across a couple networking issues that we were able to resolve with some simple hacks to the system which allowed our network to function properly.

Ad Hoc Network Support and Drivers

The Raspberry Pi 3 has a wireless chip which is built into the board. The chip uses a broadcom driver called brcmfmac. This driver seems to properly implement support for Ad hoc networks - so our Pi 3’s gave us no trouble.

It was really the Pi 2’s which we had trouble with. We were using some wireless adapters which utilize a realtek chip and driver in linux. Specifically the driver these chips were using was the rtl8192cu. Unfortunately these stock drivers do not properly support ad hoc networks.

The issue seems to lie somewhere with ARP resolution. Basically the driver doesn’t know how to properly go from IP-address to the hardware address on the next layer down. This prevents pretty much all communication on the network. Incoming and outgoing.

The issue is resolved by adding a static ARP entries using the arp command. We can make these entries persist between reboots (but not changes in interface state changes).

Every node in the network has the following interfaces file with a different static IP.

auto eth0
allow-hotplug eth0
iface eth0 inet manual

auto wlan0
allow-hotplug wlan0
iface wlan0 inet static
    address 192.168.2.xxx # xxx is a statically assigned number from 0 to 255 for each node
    netmask 255.255.255.0
    wireless-channel 1
    wireless-essid RPiwireless
    wireless-mode ad-hoc

In our case each node has an IP address ranging from 192.168.2.177 to 192.168.2.190

ARP Table

Every interface has its own hardware address that is needed for the low-level communication protocols. If the devices can’t resolve the hardware address they will not be able to communicate. That’s the issue we’re fix with our ARP entries.

If you run ifconfig or ip addr you can see the HWAddr for each interface.

Run arp -an to see your current arp table.

Turns out you can actually provide the system with some static entries in the ARP table and the network using the following command:

arp -i _ifname_ -s IP_addr HW_addr

Or you can add entries to the /etc/ethers file

IP_addr1 HW_addr1
IP_addr2 HW_addr2

Then run

sudo arp -f

arp -f will populate the table using the /etc/ethers file by default.

Example

Suppose we are on a node A with IP address 192.168.2.190 with HWaddr 3c:ae:3d:cd:02:b4. Node B has IP address 192.168.2.200 with HWaddr 3c:ae:3d:cd:43:8d

Then on Node A we run

arp -i wlan0 -s 192.168.2.200 3c:ae:3d:cd:43:8d

Then if you run arp -an you will see

? (192.168.2.200) at 3c:ae:3d:cd:43:8d [ether] PERM on wlan0

Then on Node B we run

arp -i wlan0 -s 192.168.2.190 3c:ae:3d:cd:02:b4

Then if you run arp -an you will see

? (192.168.2.190) at 3c:ae:3d:cd:02:b4 [ether] PERM on wlan0

Once both nodes have their table populated then try to do ping 192.168.2.190 from node B or ping 192.168.2.200 from node A and the pings should now be returning with reasonable latencies.

(More) Persistent Entries

The issue now become when we had a large network these tables needed to ALWAYS be populated or else communication would fail. The ARP entries do NOT persist after reboot or networking service restarts.

The easiest way to make entries persist (at least through reboots) is to use the /etc/rc.local file which is a script run at every boot. This allows us to add the entries into the table every time we boot. If the entries get erased or we need to change the interface configuration this means we just have to reboot. On a console-only raspberry pi this process only takes about 15 seconds so we settled for it.

We chose to add all of our entries to the /etc/ethers file so we didn’t need to continuously type the commands. Once the ethers file was prepared we then edits rc.local

sudo nano /etc/rc.local

Just before the exit 0 line we added arp -f

arp -f

exit 0

Great! Save the file and reboot (sudo reboot -h now). Then after rebooting and logging in type arp -an and all of the entries should be present.

This allowed us to get the ad hoc networks working using the rtl8192cu driver on linux.

A Beginner's Guide to Git and Version Control

Sun, 16 Oct 2016 00:00:00 +0000

Audience

This guide hopefully will be useful to students and others who have never used or heard of version control and are interested in using it or incorporating it into their own projects. By the end of this guide readers should have a general idea of how to get started using version control systems and begin to effectively use hosted Git services like GitHub.

Topics

What is Version Control?
Why Version Control?
Popular Version Control Programs
A Short History of Git
Introduction to Git
- Installation
- Repositories
- Commits
- Branches
- Checkout
- Merging
- Cloning/Pulling
- Pushing
Typical Workflow

What is Version Control?

From the Google dictionary^[1]:

ver·sion con·trol, noun - computing
the task of keeping a software system consisting of many versions and configurations well organized.

Basically by version control is a way that we can keep track of the changes that occur to a software project. Generally it is used for software and related tech, but you could use version control for anything such as writing a paper, an article, or even for a school project. Basically it just gives users a way to make sure that they don’t lose work on an important project. It also allows them preserve the history of anything that they’ve worked on.

Why Use Version Control?

There are various reasons that version control is useful. But one of the main, almost obvious, reasons is to be able to keep track of the changes that occur to a project in order to identify rogue behavior and prevent a loss of work.

There are several other reasons listed here on this Stack Overflow Article^[2]

Popular Version Control Programs

Below is a list of popular version control systems.

Disclaimer: This is not a full list of all version control systems. There many others but I have chosen to highlight some of the more popular ones.

Git - An open source version control system originally designed and developed by the famous Linux Kernel developer, Linus Torvalds.

Apache Subversion - A version control system originally developed by CollabNet, but donated to the Apache Software Foundation where it has been top level project since 2010^[3]

Bazaar - A version control system which is a part of the GNU project and sponsored by Canonical, the company behind Ubuntu.

Mercurial - A free open source versioning system with a GNU GPLv2 License. It is implemented with Python and has been since around 2005.

A Short History of Git

Git is one of the most widely used, if not the most widely used version control system in existence today. Git is a project which was started by Linus Torvalds, the man behind the Linux Kernel. He originally created the project so he could manage contributions to his wildly popular open source operating system kernel.^[4]

The purpose of git was to be better than the current VCS solutions available at the time. The Linux community had learned its lessons from using other systems while going through development. Torvalds and the community wanted something more reliable and robust than what they had used before. So they based this project “git” around a few main principles^[4]:

Speed
Simplicity (Compared to other systems)
Support for non-linear and branching development.
Completely Distributed
Ability to scale with large projects.

With these principles in mind, git was born into existence. Since then it has become a fierce force in the market for version control systems. Companies have been built around it, services have been created, and hundreds of thousands of projects are now hosted online thanks to Git and companies like GitHub.

Intro To Git

In this section we’re going to take a deep dive into git and its features. We’ll do a quick installation guide and then dive into the basic gits concepts.

From now on, assume that anything like the below block is meant to be run in the command line, such as the bash shell.

echo "I'm a bash command"

Git Installation

Most unix systems will already come with Git installed. However, if not you can run the following commands:

For Ubuntu/Debian based systems:

sudo apt-get install git

CentOS:

sudo yum install git

Mac OS X/MacOS:

First you’ll need to install homebrew

brew install git

Windows:

If you’re on a windows based system you can install git via the git website.

Download Location

After installing git on windows you should have access to a program called “Git Bash” which will give access to a git command line.

Git itself is a program on the command line, but it has what we’ll call subcommands. Each subcommand servers its own purpose. Below is a list of the subcommands we’ll be covering and using.

git init
git status
git add
git commit
git clone
git pull
git push
git diff
git merge
git checkout
git branch

Don’t worry too much about what each one means. We’ll go into detail below.

Repositories

So now that we’ve installed git you should keep your command line open. We’ll work off the command line. I’m going to assume you have basic skills in navigating the Unix command line and in creating directories with commands like mkdir, cd, touch, echo, etc..

Now first let’s go over some of the basic concepts behind git.

Every ‘project’ in git is typically called a repository
A repository is basically a project, or set of code, files, etc. that we want to keep track of and version.
Everything that resides within a repository will be tracked by git.

So basically think of a repository as a kind of “special” directory within your file system. Everything within the directory is what will be versioned.

In order to create a repository we need to run the following command:

git init <REPOSITORY_NAME_HERE>

Alternatively if you want to initialize a repository in the current directory you can simply run

git init

What this does is create a special directory within your repository directory called .git. This directory keeps track of your repository objects and a bunch of other metadata that is useful for kep track of files.

Git Commit

Commits define the basic unit of change in a project.

Every commit defines a set of changes to the repository. You can think of it as a kinda of “delta”. It basically tells us what changes should be played on top of a repository to go from one commit to the next.

So if you haven’t already made a repository, create one by running the following commands:

git init hello-git
cd hello-git

So now that we’ve created that we can run this command called git status. It will tell us which files might have been changed, or if any new files were added to the repository.

git status

You will probably get an output like the following.

On branch master

Initial commit

nothing to commit (create/copy files and use "git add" to track)

So the command tells us exactly what’s going on in our repository and what changes may have been made, Note the last instruction “..use “git add” to track”

So let’s create a new file and add it to our repository.

echo "Hello, World!" > hello.txt
git status

Your output should be something like

On branch master

Initial commit

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        hello.txt

nothing added to commit but untracked files present (use "git add" to track)

Let’s add our file so that it is tracked. Basically a file being tracked means it’s a file within the repository that should have any changes to it tracked by version control.

But why wouldn’t you want to track every file? There’s a couple of reasons why you might not want to.

Tracking uneccesary file can make your repository larger than it needs
- This can lead to longer cloning times and lots of useless information
- Typically you only want to track the files that are 100% necessary to run/compile your code
Sometimes you want to temporarily use data files in the repository which aren’t necessarily part of the code base.
Compiled code should not be tracked.

So now let’s add hello.txt to our tracked files and check the status.

git add hello.txt
git status

We should then have something like

On branch master

Initial commit

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)

        new file:   hello.txt

Great! So now if we make a commit this file will be tracked.

Let’s make our first commit. Remember that a commit defines a basic unit of change in a repository. This means that this commit will define the unit of change going from an empty repository to one with a text file named hello.txt

One important note about commits is adding a useful commit message. Code isn’t always readable. And trying to make sense of code after commits can be difficult. Comments in code help, but typically they’re a little too specific.

A commit message should be short, maybe 30-50 characters which describe the changes that you’re making to a repository. A commit title should be relatively short, but the body of a commit message can be much longer. For now on the command line we’re only going to deal with short commit titles. But one of the most important things about git and tracking changes is having short, descriptive titles for every commit. Ideally these titles should be completely human readable and easily understandable.

The most well-engineered pieces of software will typically have great commit messages. I suggest looking at commit messages from commits on the Linux Kernel or from git development itself

git commit -m 'My first git commit: Add hello.txt'

Output should be similar to

[master (root-commit) f21a1ab] My first git commit: Add hello.txt
warning: LF will be replaced by CRLF in hello.txt.
The file will have its original line endings in your working directory.
1 file changed, 1 insertion(+)
create mode 100644 hello.txt

Awesome! We’ve now created our repository, added a file, and now we’ve even started tracking changes. From now on, any lines added to the file will result in changes when running git status.

Try to add some more files on your own or maybe some code too. Don’t forget to write good commit messages! We’ll be moving on to more advanced topics soon. Do your best to play around and use what you’ve learned so far to get comfortable with git and the command line.

Git Branches

So aside from basic units of change, like commits, one of the other fundamental ideas behind version control (for git at least) is the ability to create what we call branches in a repository.

What is a branch?

One of the easiest ways to think of this is that every project has a main code-line, or master branch (sometimes called the trunk). That is the place where every feature gets added. Then using that main code line, you have the ability to create these branches which stem completely from the main code line. It’s like they’re their own mini-repository. They have all of the code and all of the history from the main code line, but the changes end up being detached from the master branch.

So then why do we have branches?

Basically branches are a way for individuals to break up pieces of work on a project in a way that they don’t affect the main branch, but can still are able to track the work done on a particular set of files or features without affecting the main code line until the work on the branch is completed. Then finally you want the code from the branch to be merged back into the master branch. The reasoning behind this is that you don’t want broken code or bad features to be merged into the main code line for a project. But you still want to use all of the features of the version control as you work on a specific feature for the project. This way when that feature is completed, it can be tested for errors to make sure that the code is stable before being merged to master.

So let’s try to create a branch.

Go into that repository we just made named hello-git and run

git branch my-special-branch

If you run git status you should notice a line

  on branch master

Now run

git branch -v

You should now see the new branch my-special-branch. Next, we’ll show you how to do what is called a checkout of a branch

Git Checkout

So at this point we’ve made a new repository named “hello-git” and we’ve also made a new branch alongside master called “my-special-branch”. The issue now is that we made the branch, but we actually can’t make changes on the branch until we do what is called a “checkout” which makes our repository switch over to the copy of the code which the branch uses.

So let’s run

git checkout my-special-branch
echo "Look a special change!!" >> hello.txt
git status

After running those three commands your git status should be something like

On branch my-special-branch
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   hello.txt

no changes added to commit (use "git add" and/or "git commit -a")

Notice that to commit any changes we still need to add the file to the staging area. But notice that after the checkout we have the line On branch my-special-branch meaning our changes are being made to the “my-special-branch”.

So let’s add the file and make a commit

git add -A
git commit -m "Adding new special line to hello.txt file" # Make sure you have a really clear commit message!

Great! We have our changes committed to the second branch.

Try running

git checkout master
git status
cat hello.txt

If you run these commands you should notice that your changes from the my-special-branch didn’t actually carry over to the master branch yet. That’s due to the fact that the code in a separate branch from master is kept separate until you explicity tell git that you want the changes merged into the other branch. That’s what we’ll be going over next in git merge.

This should at least give you the basics on how to create a branch in git, how to checkout and then make changes, without affecting the rest of the repository. Remember most good project want to keep the commit history on the main code line somewhat clean, but still use the powerful features of version control to save work and keep intermediate versions. That’s what branching is good for. We’ll go over typical git workflows once we learn about merging

Git merge

So at this point our repository has two separate branches, a master and my-special-branch. My special branch has a new change that we want on the master branch. How do we do that? We do it with git merge Git merge does exactly what it sounds like. It merges two branches. But to clarify, git merge is more like a one-way merge than a two-way merge.

So what git doesn’t do:

Take two different branches (one of which was branched from an earlier point in time), merge them into a single one and automatically discard the old ones and just leave the new one.

What git merge actually does:

Looks at the specified branch, and the current branch. It will look at the history of the branch you’re currently on and find how many commits ago the branch was made.
Then it will “replay” the commits (all of the changes) from the other branch onto your current branch and create a new commit on the current branch with a message (if specified).

So in short, it kind of “adds” the history of one branch onto another, thus merging the changes into the new branch with a commit.

Let’s try it now:

git checkout master
git merge my-special-branch

Voila! We’ve now officially merged the branch into master.

You can see the changes by getting the content of the file from cat

cat hello.txt

Cloning and Pulling

So one of the ideas around git is that it is designed to be decentralized. A decentralized system prevents bottlenecks or downtime when cloning code. Not only that but it also means you can grab a copy from the code over the network from anyone’s repository that is available.

There is no “master” that is required to always have the latest version of code, and that’s one of the beauties of git. Any copy of the repository which doesn’t exist on the local machine is called a remote. Remotes can be added or removed to a repository and they allow you to push and pull bits of code too and from your repository to other repositories based on the remote configuration.

So lets jump in to actually grabbing code from another repository:

The clone subcommand will attempt to access a hosted repository based on a specified URL. You can use HTTP, HTTPS, or Git protocols.

Let’s try clone a repository from GitHub.

git clone https://github.com/git/hello-world.git

Another option:

git clone git://github.com/git/hello-world.git

Notice the difference between the protocols https:// and git://

Note however according to the git docs that the git:// protocol is not typically only used for cloning or pulling. It also does not provide any form of authentication.

Either way, you should get something like this:

Cloning into 'hello-world'...
remote: Counting objects: 158, done.
remote: Total 158 (delta 0), reused 0 (delta 0), pack-reused 158
Receiving objects: 100% (158/158), 17.78 KiB | 0 bytes/s, done.
Resolving deltas: 100% (47/47), done.
Checking connectivity... done.

If you do ls you should now see a new directory named hello-world. This is the directory of the freshly cloned repository. Typically a clone is only performed once. It is for the first time you need to get a repository onto your computer.

When you clone the repository you automatically get this thing called a remote added to the git metadata. The remote is the origin location that the repository came from. This will default to whatever url was used to clone. You can add more remotes by using the command git remote add. Use git remote -h to see all of the options. git remote -v will show all of the push and fetch URLs for each remote.

One thing that similar to cloning is called pulling using the command git pull.

Pull is kind of similar to clone, except for a pull, you need to already be within a repository.

If you run git pull -h you’ll get a help text for the commands.

The two most common pulls you’ll use are probably

git pull [remote_name/branchname]
git pull

These two commands basically look for any new commits or updates that were on the remote repository, and attempt to replay them on top of your own repository in order to reflect any changes that have been made to the code, making a commit in the process. It’s similar to git merge, except for remote repositories.

It’s also common to use the --rebase option which runs git rebase instead of git merge. I suggest looking up the documentation for rebase.

Git push

When working with repositories that may not be located on your machine locally, you’ll need a way to be able to get those remote repositories to reflect the changes you’ve made to your local computer. You’re able to do that with git push

git push will basically take the current branch you’re working on and push it to your default (or specified remote) and it’s the basic way to make sure any of your changes are replayed into your remote repository to reflect the work that you’ve done.

It’s relatively simple. Typically you’ll use it if you have remote repositories where you have push access (privileges to change code). Once you get started cloning your own repositories from GitHub you’ll see exactly how to

Typical Git Workflow

Below is a basic workflow that you might find:

git clone https://myremote/myrepository.git
cd myrepository
git checkout -b feature-123 # creates a new branch
# edit/compile/test
git commit -a -s
# edit/compile/test
git commit -a -s
# You're finally happy with your changes, merge into new branch
git checkout master
# Make sure we have any new changes from master 
git pull --rebase
git merge feature-123 -m "[Issue-123] - Add special feature"
# Reflect the update in the remote.
git push 

And that’s it! It’s not too hard to start using. The real challenge is using a tool like git well. Good use of version control should make rollbacks and upgrades easy as well as differentiating the the software at various points in time. Bad version control could be just as bad and more time consuming as no version control at all! Make sure to do some more reading to brush up on the advanced features and some of the options that you can add to commands which can make git even more powerful

I highly highly recommend reading the git documentation to get a better idea of what some of these commands do. This guide is intended for serious beginners who want to “just get started” without learning much of the technical details. But alas, there is a high chance you’ll need to know more in the future. The documentation on git is very thorough and I suggest reading through it to gain a better understanding of the functions of git.

That’s all folks!

References

[1]: Google Definition of Version Control
[2]: Reasons to Use Version Control from StackOverflow
[3]: History of Apache Subversion from Wikipedia
[4]: History of Git

How to Create Your Own Website with GitHub and Jekyll

Fri, 13 May 2016 00:00:00 +0000

This post will serve as a guide on how to create your own website (for free!) using GitHub Pages and Jekyll.

Pre-Requisites

Create a GitHub account if you haven’t already made one.
The GitHub Desktop Client downloaded and installed.
- Make sure that you sign in to your github account with the desktop client.
Install Ruby and Jekyll

Step 1 - The Website Repository

Once you’ve created your GitHub account you’ll need to create a repository which matches the following name:

username.github.io

You should replace username with your own github username.

Example

My GitHub username is “zacblanco”. Thus, my repository name is “zacblanco.github.io”

To create the new repository head to your user account page on GitHub and click the New button under the repositories tab.

Step 2 - Initializing the Jekyll Website

Once you’ve created the GitHub repository, you’re going to need to clone it onto your own computer. Make sure that you know where the GitHub desktop client is going to clone your repository on your computer.

You can find this by going to:

Settings > Options > Clone Path

Make note of this location as you’ll need it soon.

Now open up the GitHub desktop client and clone your newly created repository

After finishing the clone it should now appear in your desktop client.

Once you see it in the sidebar, you’ll need to open up a terminal or command prompt window (depending on whether you’re using OS X/Linux or Windows).

Once you’ve opened you’re terminal window you should cd (change directory) into your clone path that you found earlier using the following command:

cd /path/to/cloned/repos

After executing the command we can then initialize a jekyll site in your repository. Run the following command:

jekyll new username.github.io

Just remember to replace username with your GitHub account name.

Congrats! The site should now be initialized.

Once you’ve intialized the site open up the GitHub desktop client. You should see a list of changes similar to the following:

You should then do the following things:

Write a short, descriptive commit message
Click Commit to master
Click Publish

Each part is circled in red on the image above.

Awesome! Once you’ve clicked Publish the site is actually live!

You should be able to see your site at https://username.github.io

If you aren’t able to access your site and/or receive a 404 error, you should try the following steps to resolve your issues:

Head into your repository directory
- Do this by using cd username.github.io
Run the command jekyll build

If any error messages appear, try to see if you can resolve them using some google-fu (although I would hope no errors arise when simply intializing the repository).

Once you can access your site via https://username.github.io you can head to the next step.

Step 3 - Basic Customization

Now that our Jekyll site is online we’re going to need to do some customization. This will help you run your site more smoothly and test your site before you push changes to the internet.

First, you should find a file in the repository titled .gitignore (Create this file if it doesn’t exist).

Next, add the following line to the .gitignore file:

_site/*

Adding this line will prevent you from pushing any local site builds to your repository. This generally helps to minimize clutter.

Next you should open the _config.yml file. You can open this file with any text editor you like. Change the fields to match what you want your site to be about.

I recommend at minimum changing the following fields to reflect your site:

title
email
description
url
twitter username
github username

Note - If you are missing any of the fields, simply just leave them blank. Once you customize your site layout more you can remove them from the _config.yml file if you wish.

Once you’ve done that we’re going to test how well the site changes with the configuration changes.

Use your terminal and navigate into your username.github.io directory and run the following command:

jekyll serve

Jekyll will now try to build your site and serve it locally. You can access it (only on your computer) with the address http://127.0.0.1:4000

The last thing you should do here is look in the _posts folder and delete the pre-generated posts that are in there. However, do make note of the metadata which is at the top of the file. You’ll need something similar to it later.

I also highly recommend GitHub’s guide to customizing your pages site

At this point it’s imperative that you familiarize yourself with Jekyll’s features and documentation as well as GitHub pages’ features as well. Below is a curated list of links to some of what I believe to be the most important information about Jekyll.

After giving the above links a quick skim you should be up to speed on everything jekyll and can get to customizing your site in the next step!

Styling Your Site (Coming Soon!)

Social Psychology Exam 3 Guide

Sun, 29 Nov 2015 00:00:00 +0000

This guide has moved!

Click here to go to the new page

Computer Science Jokes

Fri, 06 Nov 2015 00:00:00 +0000

Sometimes the stress of school really just gets to you. Don’t forget to laugh every once in a while. Here’s a list of jokes I’ve collected over some time.

Non-Decimal Number Systems

Well, I couldn’t really leave these out…

There are only 10 types of people in the world: those who understand binary, and those who don’t.

There are 10 types of people in the world: those who know binary, those who don’t, and those who didn’t expect this joke to be in base 3.

Unix Scripting

I like to live on the edge:

[ $[ $RANDOM % 6] == 0 ] && rm -rf /* || echo Click
credit: StackOverflow user mxc

Hardware

Q: How many programmers does it take to change a lightbulb
A: None, that’s a hardware problem!
credit: StackOverflow User Steven A. Lowe

Networking

I would tell you a joke about UDP, but you might not get it…
credit: reddit user /u/Portaljacker

Object Oriented Programming

Q: “Whats the object-oriented way to become wealthy?”
A: Inheritance
credit to StackOverflow user spelchec

Disclaimer: These are not mine, and I may have modified them slightly. I have credited jokes from where I have found them on the interwebs :)

Social Psychology Exam 2 Guide

Sun, 01 Nov 2015 00:00:00 +0000

This page has moved to the education section!

Click here to go to the new page